Creating a generic Flutter based mobile app in Python, Groovy, Ruby or JavaScript with Sponge Remote


06 Oct 2020 - Marcin Paś

Is the title of this article a clickbait? Well, yes and no, if you forgive the expression, as Sir Humphrey Appleby would have said. Let’s try to investigate it.

Sponge is an open-source, Java-based action and event processing service.

Sponge Remote is an open-source mobile application built with Flutter, that provides an opinionated, minimalistic, generic user interface to call remote Sponge actions.

If you are not familiar with Sponge and the Sponge Remote mobile app, please read the Getting Started with Sponge and the Sponge Remote Mobile App article first.

The Sponge Remote mobile app is available on Google Play.

Get it on Google Play

Google Play and the Google Play logo are trademarks of Google LLC.

1. Concepts

04 100 mobile architecture

A Sponge action can be implemented in Python, Groovy, Ruby or JavaScript (or Java, Kotlin if you prefer a compiled language) in a Sponge service (i.e. on the server side). An action that is to be visible in the mobile app has to have metadata, like a name, label, arguments, argument data types, etc.

Sponge is written in Java so the Python interpreter used by Sponge is Jython, which is compatible with Python 2.7. You could use CPython (that would run in another process), but it would require you to configute the Py4J plugin. Ruby interpreter is JRuby, JavaScript interpreter is Nashorn.

One of the key concepts here is that every Sponge action, action argument and data type can be configured in order to be rendered by a compatible GUI rendering engine.

Currently, there is only one implementation of such rendering engine. It is the Sponge Flutter API that renders action metadata as a Flutter screen and handles user interactions. The rendered GUI is generic and opinionated and the out of the box options of customizing are intentionally very limited.

In other words, the heavy lifting is done by the Sponge Flutter API, so to get things done you only have to write a Sponge action in the backend Sponge service.

The Sponge Flutter API is used by the Sponge Remote mobile app, that allows you to connect to any Sponge service and run published actions. The mobile app retrieves action metadata from the Sponge service via the Remote API (compatible with JSON-RPC 2.0).

To recap, you don’t write your mobile app code in Python directly. Your action (e.g. Python) code along with its metadata is stored and executed on the server side (in a Sponge service) but can be indirectly (remotely) interacted with on a mobile device. One might see here some distant similarity to HTML and a web browser that renders it. However, generally Sponge Remote is intended for more straightforward or narrow use cases.

If you need to have a more customized GUI, you can write your own Flutter app from scratch using only a subset of the Sponge Flutter API functionalities. An example of such app is Sponge Digits (source codes). The Sponge Flutter API library is currently in alpha phase and supports only a limited set of Sponge features.

2. Why Flutter?

I’ve been using many GUI SDKs and frameworks in the span of over 25 years: XView and Motif for X Windows; MFC library for MS Windows; Oracle Forms; Java AWT, Java Swing, Java SWT; Struts, JSF, Tapestry, Wicket, GWT, Angular for web; Windows Mobile SDK, Android SDK for mobile; and so on. They have features that make them more or less enjoyable to code with. They utilize many common patterns or similar solutions.

After the Sponge engine had been created, I started wondering if there could be an easy way to create a mobile or web GUI to remotely call Sponge actions. Just to use the actions in a user friendly way, for example for connecting to an IoT device that runs Sponge. At that time, as to GUI development, I had the most experience in GWT for web and Android SDK for mobile. So, at the first glance, one of them would have been the first choice for me. However, GWT was losing popularity so the Android SDK seemed a way to go.

Then some day I stumbled upon an article about Flutter. I really liked the way of writing GUI in the code, hot reload and the overall architecture of the framework. Another advantage was that with a single codebase the app could be run both on Android and on iOS. Moreover after some time there was an announcement of the plans to support web and desktop as well. It meant that most of the codebase would be able to run on more platforms without much effort. For me, that sealed the deal.

After about two years of developing in Flutter I must say that it has been the best GUI framework I’ve ever had a pleasure to use. Flutter gives so much fun that it encourages to create more functionalities in the app. I can even say that Flutter is one of the main reasons the Sponge Remote mobile app and the associated underlying Dart libraries have been created as they are.

The Dart language has become one of my favourites, along with Java, Python and Lisp. In a generic code (e.g. that recursively creates a widget tree for a Sponge data type hierarchy) the Dart’s dynamic type is very helpful, because it spares the pain of writing a lot of boilerplate code. It is dangerous of course, but not more than, let’s say, a corresponding Python code.

3. How does it work?

As an example of a Sponge action that is renderable in the Sponge Remote app let’s take the Music Player described in the Using Sponge and the Sponge Remote Mobile App as a Remote Hi-Fi Music Player article.

03 150 action player

I won’t get into much details here in order to focus only on the outline of the main Music Player action. First, let’s see the action code.

class MpdPlayer(Action):
    def onConfigure(self):
        self.withLabel("Player").withDescription("The MPD player.")
        self.withArgs([
            # The song info arguments.
            StringType("song").withLabel("Song").withNullable().withReadOnly()
                .withProvided(ProvidedMeta().withValue()),
            StringType("album").withLabel("Album").withNullable().withReadOnly()
                .withProvided(ProvidedMeta().withValue()),
            StringType("date").withLabel("Date").withNullable().withReadOnly()
                .withProvided(ProvidedMeta().withValue()),

            # The position arguments.
            IntegerType("position").withLabel("Position").withNullable().withAnnotated()
                .withMinValue(0).withMaxValue(100)
                .withFeatures({"widget":"slider", "group":"position"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable()),
            StringType("time").withLabel("Time").withNullable().withReadOnly()
                .withFeatures({"group":"position"}).withProvided(ProvidedMeta().withValue()),

            # The navigation arguments.
            VoidType("prev").withLabel("Previous").withAnnotated()
                .withFeatures({"icon":IconInfo().withName("skip-previous").withSize(30),
                               "group":"navigation", "align":"center"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable()),
            BooleanType("play").withLabel("Play").withAnnotated().withFeatures({"group":"navigation"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable().withLazyUpdate()),
            VoidType("next").withLabel("Next").withAnnotated()
                .withFeatures({"icon":IconInfo().withName("skip-next").withSize(30), "group":"navigation"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable()),

            # The volume argument.
            IntegerType("volume").withLabel("Volume").withAnnotated().withMinValue(0).withMaxValue(100)
                .withFeatures({"widget":"slider"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable().withLazyUpdate()),

            # The mode arguments.
            BooleanType("repeat").withLabel("Repeat").withAnnotated()
                .withFeatures({"group":"mode", "widget":"toggleButton", "icon":"repeat", "align":"right"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable().withLazyUpdate()),
            BooleanType("single").withLabel("Single").withAnnotated()
                .withFeatures({"group":"mode", "widget":"toggleButton", "icon":"numeric-1"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable().withLazyUpdate()),
            BooleanType("random").withLabel("Random").withAnnotated()
                .withFeatures({"group":"mode", "widget":"toggleButton", "icon":"shuffle"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable().withLazyUpdate()),
            BooleanType("consume").withLabel("Consume").withAnnotated()
                .withFeatures({"group":"mode", "widget":"toggleButton", "icon":"pac-man"})
                .withProvided(ProvidedMeta().withValue().withOverwrite().withSubmittable().withLazyUpdate())
        ]).withNonCallable().withActivatable()
        self.withFeatures({"refreshEvents":["statusPolling", "mpdNotification_.*"], "icon":"music", "contextActions":[
            SubAction("MpdPlaylist"),
            SubAction("MpdFindAndAddToPlaylist"),
            SubAction("ViewSongInfo"),
            SubAction("ViewSongLyrics"),
            SubAction("MpdLibrary"),
            SubAction("ViewMpdStatus"),
        ]})

    def onIsActive(self, context):
        return sponge.getVariable("mpc").isConnected()

    def onProvideArgs(self, context):
        """This callback method:
        a) Modifies the MPD state by using the argument values submitted by the user. The names
        are specified in the context.submit set, the values are stored in the context.current map.
        b) Sets the values of arguments that are to be provided to the client. The names are
        specified in the context.provide set, the values are to be stored in the context.provided map.
        """
        MpdPlayerProvideArgsRuntime(context).run()

The MpdPlayer class defines the Music Player action. The action arguments "song", "album", "date", "position", "time", "prev", "play", "next", "volume", "repeat", "single", "random" and "consume" are mapped to the corresponding Flutter widgets. Each argument is defined as a Sponge data type instance.

Actions and arguments can have features, represented as a map of names to values. Features provide additional information and they are flexible in a sense that they are not a part of the static API.

A provided argument (i.e. one that has the withProvided method call) is interactive because it can be refreshed from or submitted to the remote service. This interactiveness is handled by the onProvideArgs callback method of the action.

An annotated argument is wrapped by an instance of the AnnotatedValue class. An annotated argument allows passing a value label, a value description, features, type label and type description along with the argument value. Annotated arguments can be used to alter static, metadata-drived behavior of a client application.

Provided argument metadata:

  • The withOverwrite flag specifies if an argument value modified in the mobile app by a user can be overwritten by a value obtained from the remote service. It is important for arguments which state can be changed on the server side independently from the mobile app.

  • The withSubmittable method specifies if an argument value should be submitted to the service at once after being changed by the user (e.g. "position" or "volume").

  • The withLazyUpdate flag tells if an argument should be updated lazily in the GUI, i.e. its previous value should be shown until a new value is obtained from the server. It is necessary if an argument is annotated and its annotated value changes the label, e.g. "volume". Otherwise the GUI widget would show a stale label for a short amount of time.

The "group" feature allows grouping of the arguments in the GUI, e.g. the navigation arguments "prev", "play" and "next".

Action metadata:

  • The withNonCallable flag makes the action non callable, i.e. it behaves as an interactive form.

  • The withActivatable flag tells that the action has the onIsActive callback method, that will be called remotely before the action is displayed. If the method returns False the action won’t be shown. In this case the method checks if the mpc commandline client can connect to the MPD daemon.

Action features:

  • The "refreshEvents" feature is a list of Sponge events that will cause the GUI to refresh. Events are pushed to the mobile app via gRPC.

  • The "contextActions" feature is a list of Sponge actions that are to be shown in the action menu. You can navigate to any of these actions in the GUI. An action screen can be closed by swiping right.

The mpc variable is an instance of the Mpc class.

The supported icon names (e.g. "pac-man") correspond to the material design icons.

The onProvideArgs callback method essentially modifies the MPD state by using the argument values submitted by the user and then sets the values of arguments that are to be provided, i.e. sent back to the mobile app. The code that implements provisioning of arguments is placed in the MpdPlayerProvideArgsRuntime class.

class MpdPlayerProvideArgsRuntime:
    def __init__(self, context):
        self.context = context

        # An instance of the Mpc class that is a wrapper for the mpc (MPD client)
        # commandline calls.
        self.mpc = sponge.getVariable("mpc")

        # Stores and caches the latest MPD status as returned by the mpc command.
        self.status = None

        # Providers for the submittable arguments.
        self.submitProviders = {
            "position":lambda value: self.mpc.seekByPercentage(value),
            "volume":lambda value: self.mpc.setVolume(value),

            "play":lambda value: self.mpc.togglePlay(value),
            "prev":lambda value: self.mpc.prev(),
            "next":lambda value: self.mpc.next(),

            "repeat":lambda value: self.mpc.setMode("repeat", value),
            "single":lambda value: self.mpc.setMode("single", value),
            "random":lambda value: self.mpc.setMode("random", value),
            "consume":lambda value: self.mpc.setMode("consume", value),
            }

        # The MPD mode types (that correspond to the action arguments).
        self.modeTypes = ["repeat", "single" , "random", "consume"]

    def run(self):
        # Enter the critical section.
        self.mpc.lock.lock()
        try:
            # Pass the context as an argument to the private methods in order
            # to simplify the code.

            # Submit arguments sent from the client.
            self.__submitArgs(self.context)

            # Provide arguments to the client.
            self.__provideInfoArgs(self.context)
            self.__providePositionArgs(self.context)
            self.__provideVolumeArg(self.context)
            self.__provideNavigationArgs(self.context)
            self.__provideModeArgs(self.context)
        finally:
            self.mpc.lock.unlock()

    def __submitArgs(self, context):
        """Submits arguments sent from the client by changing the state of the MPD daemon.
        """
        # Use simple providers to submit values.
        for name in context.submit:
            provider = self.submitProviders.get(name)
            if provider:
                try:
                    # Cache the current MPD status.
                    self.status = provider(
                        context.current[name].value if context.current[name] else None)
                except:
                    sponge.logger.warn("Submit error: {}", sys.exc_info()[1])

    def __getStatus(self):
        # Cache the mpc command status text.
        if not self.mpc.isStatusOk(self.status):
            self.status = self.mpc.getStatus()

        return self.status

    def __provideInfoArgs(self, context):
        """Provides the song info arguments to the client.
        """
        # Read the current song from the MPD daemon only if necessary.
        currentSong = self.mpc.getCurrentSong() if any(
            arg in context.provide for arg in ["song", "album" , "date"]) else None

        if "song" in context.provide:
            context.provided["song"] = ProvidedValue().withValue(
                self.mpc.getSongLabel(currentSong) if currentSong else None)
        if "album" in context.provide:
            context.provided["album"] = ProvidedValue().withValue(
                currentSong["album"] if currentSong else None)
        if "date" in context.provide:
            context.provided["date"] = ProvidedValue().withValue(
                currentSong["date"] if currentSong else None)

    def __providePositionArgs(self, context):
        """Provides the position arguments to the client.
        """
        # If submitted, provide an updated annotated position too.
        if "position" in context.provide or "context" in context.submit:
            context.provided["position"] = ProvidedValue().withValue(
                AnnotatedValue(self.mpc.getPositionByPercentage(self.__getStatus()))
                    .withFeature("enabled",
                                 self.mpc.isStatusPlayingOrPaused(self.__getStatus())))
        if "time" in context.provide:
            context.provided["time"] = ProvidedValue().withValue(
                self.mpc.getTimeStatus(self.__getStatus()))

    def __provideVolumeArg(self, context):
        """Provides the volume argument to the client.
        """
        # If submitted, provide an updated annotated volume too.
        if "volume" in context.provide or "volume" in context.submit:
            volume = self.mpc.getVolume(self.__getStatus())
            context.provided["volume"] = ProvidedValue().withValue(
                AnnotatedValue(volume)
                    .withTypeLabel("Volume" + ((" (" + str(volume) + "%)") if volume else "")))

    def __provideNavigationArgs(self, context):
        """Provides the navigation arguments to the client.
        """
        if "play" in context.provide:
            playing = self.mpc.getPlay(self.__getStatus())
            context.provided["play"] = ProvidedValue().withValue(
                AnnotatedValue(playing).withFeature("icon",
                    IconInfo().withName("pause" if playing else "play").withSize(60)))

        # Read the current playlist position and size from the MPD daemon only if necessary.
        (position, size) = (None, None)
        if "prev" in context.provide or "next" in context.provide:
            (position, size) = self.mpc.getCurrentPlaylistPositionAndSize(self.__getStatus())

        if "prev" in context.provide:
            context.provided["prev"] = ProvidedValue().withValue(
                AnnotatedValue(None).withFeature("enabled", position is not None))
        if "next" in context.provide:
            context.provided["next"] = ProvidedValue().withValue(
                AnnotatedValue(None).withFeature("enabled",
                    position is not None and size is not None))

    def __provideModeArgs(self, context):
        """Provides the mode arguments to the client.
        """
        currentModes = None

        # Provide only required modes (i.e. specified in the context.provide set).
        for arg in [a for a in self.modeTypes if a in context.provide]:
            if currentModes is None:
                # Read the current modes from the MPD daemon only if necessary.
                currentModes = self.mpc.getModes(self.__getStatus())

            context.provided[arg] = ProvidedValue().withValue(
                    AnnotatedValue(currentModes.get(arg, False)))

You may notice that there are a bunch of if statements without else clauses. They are present in order to handle only a subset of supported provided arguments while processing one request.

This is only a simplified description of the Music Player action, but I believe that it gives a feel of how the rendering and handling of an action looks like in the Sponge Remote mobile app.

For more details see the source codes of the sponge-kb-mpd-mpc knowledge base.

4. Summary

So, you can create a generic Flutter based mobile app in Python, Groovy, Ruby or JavaScript using Sponge and Sponge Remote, but you must be aware that it would be suitable only for specific use cases.

You could also start with a generic, opinionated Sponge Remote mobile app and then switch to your own Flutter app based on the Sponge Flutter API. Your Flutter app would connect to the same Sponge service and use the same actions as Sponge Remote but the actions would be handled differently in the GUI.

Link to the Medium article.