Mobile client application


1. Introduction

The Sponge mobile client application is a Flutter application that provides a generic GUI to call remote Sponge actions. It can be run on both Android and iOS. It could be used as the one app to rule them all for Sponge services that publish actions via the Sponge REST API. All business logic has to be placed in Sponge actions, so the only requirement to have the application working is to create your own Sponge service by writing a knowledge base that define actions that will be visible and ready to run in the application or to connect to an existing Sponge service.

The application is especially useful when data and functionality are more important than visual aspects of a GUI, e.g. for prototyping, rapid application development, low cost solutions, etc. The reason is that the application provides only a generic and very opinionated GUI whose customization is limited.

mobile architecture
Figure 1. The mobile application architecture

One of many use cases of the application is to connect to IoT devices that have Sponge installed and execute available actions. Different types of such devices provide different sets of actions that are specific to a device features. For example one device could have a camera and provide an action to take a picture. Another device could have a temperature sensor and provide its readings or have a machine learning model and provide predictions.

The application is currently under development. It will be released as an open source.

2. Features

The following chapters show the key features of the mobile application.

2.1. Connections

actions connection list
Figure 2. Selecting a connection to the Sponge instance in the action list screen

You may configure many connections to Sponge REST API services. The application allows you to connect to a single service at the same time.

connections
Figure 3. List of connections to Sponge instances

You may add, edit and remove connections to Sponge instances as well as activate a connection. To remove a connection swipe the corresponding element.

connections edit
Figure 4. Editing a connection to a Sponge instance

A Sponge address is the URL of the Sponge instance.

2.2. Action list

actions
Figure 5. The action list

The main screen shows the list of actions defined in the connected Sponge engine. Only actions that have argument and result metadata are available. This is because the application uses a generic access to the actions utilizing their data types, labels, descriptions, features and so on. The number in the action icon is the number of action arguments.

To call an action or set action attributes tap the triangular icon on the right side of the action label.

The floating button allows to refresh the action list from the server. The refresh button clears all entered action arguments and received results.

The application currently doesn’t supports all Sponge data types.
drawer
Figure 6. The navigation drawer

The navigation drawer allows switching between the available main views.

2.4. Action call

action call manage lcd
Figure 7. The action call that manages the Raspberry Pi LCD display

Actions may have read only, provided arguments only to show a data from the server (see the Current LCD text attribute). The REFRESH button retrieves the current values of read only, provided arguments from the server.

The definition of the action that manages the Raspberry Pi LCD display
class ManageLcd(Action):
    def onConfigure(self):
        self.withLabel("Manage the LCD text and color")
        self.withDescription("Provides management of the LCD properties (display text and color). A null value doesn't change an LCD property.")
        self.withArgs([
            StringType("currentText").withMaxLength(256).withNullable(True).withFeatures({"maxLines":2})
                .withLabel("Current LCD text").withDescription("The currently displayed LCD text.").withProvided(ProvidedMeta().withValue().withReadOnly()),
            StringType("text").withMaxLength(256).withNullable(True).withFeatures({"maxLines":2})
                .withLabel("Text to display").withDescription("The text that will be displayed in the LCD.").withProvided(ProvidedMeta().withValue()),
            StringType("color").withMaxLength(6).withNullable(True).withFeatures({"characteristic":"color"})
                .withLabel("LCD color").withDescription("The LCD color.").withProvided(ProvidedMeta().withValue().withOverwrite()),
            BooleanType("clearText").withNullable(True).withDefaultValue(False)
                .withLabel("Clear text").withDescription("The text the LCD will be cleared.")
        ]).withNoResult()
    def onCall(self, currentText, text, color, clearText = None):
        sponge.call("SetLcd", [text, color, clearText])
    def onProvideArgs(self, context):
        grovePiDevice = sponge.getVariable("grovePiDevice")
        if "currentText" in context.names:
            context.provided["currentText"] = ProvidedValue().withValue(grovePiDevice.getLcdText())
        if "text" in context.names:
            context.provided["text"] = ProvidedValue().withValue(grovePiDevice.getLcdText())
        if "color" in context.names:
            context.provided["color"] = ProvidedValue().withValue(grovePiDevice.getLcdColor())

class SetLcd(Action):
    def onCall(self, text, color, clearText = None):
        sponge.getVariable("grovePiDevice").setLcd("" if (clearText or text is None) else text, color)
action call manage sensors
Figure 8. The action call that manages the Grove Pi sensors and actuators

The action call screen allows editing the action arguments.

The definition of the action that manages the Grove Pi sensors and actuators
class ManageSensorActuatorValues(Action):
    def onConfigure(self):
        self.withLabel("Manage the sensor and actuator values").withDescription("Provides management of the sensor and actuator values.")
        self.withArgs([
            NumberType("temperatureSensor").withNullable().withLabel(u"Temperature sensor (°C)").withProvided(ProvidedMeta().withValue().withReadOnly()),
            NumberType("humiditySensor").withNullable().withLabel(u"Humidity sensor (%)").withProvided(ProvidedMeta().withValue().withReadOnly()),
            NumberType("lightSensor").withNullable().withLabel(u"Light sensor").withProvided(ProvidedMeta().withValue().withReadOnly()),
            NumberType("rotarySensor").withNullable().withLabel(u"Rotary sensor").withProvided(ProvidedMeta().withValue().withReadOnly()),
            NumberType("soundSensor").withNullable().withLabel(u"Sound sensor").withProvided(ProvidedMeta().withValue().withReadOnly()),
            BooleanType("redLed").withLabel("Red LED").withProvided(ProvidedMeta().withValue().withOverwrite()),
            IntegerType("blueLed").withMinValue(0).withMaxValue(255).withLabel("Blue LED").withProvided(ProvidedMeta().withValue().withOverwrite()),
            BooleanType("buzzer").withLabel("Buzzer").withProvided(ProvidedMeta().withValue().withOverwrite())
        ]).withNoResult()
    def onCall(self, temperatureSensor, humiditySensor, lightSensor, rotarySensor, soundSensor, redLed, blueLed, buzzer):
        grovePiDevice = sponge.getVariable("grovePiDevice")
        grovePiDevice.setRedLed(redLed)
        grovePiDevice.setBlueLed(blueLed)
        grovePiDevice.setBuzzer(buzzer)
    def onProvideArgs(self, context):
        values = sponge.call("GetSensorActuatorValues", [context.names])
        for name, value in values.iteritems():
            context.provided[name] = ProvidedValue().withValue(value)

class GetSensorActuatorValues(Action):
    def onCall(self, names):
        values = {}
        grovePiDevice = sponge.getVariable("grovePiDevice")
        if "temperatureSensor" or "humiditySensor" in names:
            th = grovePiDevice.getTemperatureHumiditySensor()
            if "temperatureSensor" in names:
                values["temperatureSensor"] = th.temperature if th else None
            if "humiditySensor" in names:
                values["humiditySensor"] = th.humidity if th else None
        if "lightSensor" in names:
            values["lightSensor"] = grovePiDevice.getLightSensor()
        if "rotarySensor" in names:
            values["rotarySensor"] = grovePiDevice.getRotarySensor().factor
        if "soundSensor" in names:
            values["soundSensor"] = grovePiDevice.getSoundSensor()
        if "redLed" in names:
            values["redLed"] = grovePiDevice.getRedLed()
        if "blueLed" in names:
            values["blueLed"] = grovePiDevice.getBlueLed()
        if "buzzer" in names:
            values["buzzer"] = grovePiDevice.getBuzzer()
        return values
action call send sms
Figure 9. The action call that sends an SMS from the Raspberry Pi

Actions arguments may be edited in multiline text fields.

The definition of the action that sends an SMS from the Raspberry Pi
class SendSms(Action):
    def onConfigure(self):
        self.withLabel("Send an SMS").withDescription("Sends a new SMS.")
        self.withArgs([
            StringType("recipient").withFormat("phone").withLabel("Recipient").withDescription("The SMS recipient."),
            StringType("message").withMaxLength(160).withFeatures({"maxLines":5}).withLabel("Message").withDescription("The SMS message.")
        ]).withNoResult()
    def onCall(self, recipient, message):
        gsm.sendSms(recipient, message)
action call color
Figure 10. The action call argument editor for a color type

The color picker widget allows a user to choose a color as an argument value.

The definition of the action that takes a color argument
class ChooseColor(Action):
    def onConfigure(self):
        self.withLabel("Choose a color").withDescription("Shows a color argument.")
        self.withArg(
            StringType("color").withMaxLength(6).withNullable(True).withFeatures({"characteristic":"color"})
                .withLabel("Color").withDescription("The color.")
        )
        self.withResult(StringType())
    def onCall(self, color):
        return ("The chosen color is " + color) if color else "No color chosen"
action call digit drawing
Figure 11. The action call argument editor for a digit drawing

The drawing panel allows a user to paint an image that will be set as an argument value in an action call.

The definition of the action that recognizes a handwritten digit
class DigitsPredict(Action):
    def onConfigure(self):
        self.withLabel("Recognize a digit").withDescription("Recognizes a handwritten digit")
        self.withArg(createImageType("image")).withResult(IntegerType().withLabel("Recognized digit"))
    def onCall(self, image):
        predictions = py4j.facade.predict(image)
        prediction = max(predictions, key=predictions.get)
        probability = predictions[prediction]

        # Handle the optional predictionThreshold Sponge variable.
        predictionThreshold = sponge.getVariable("predictionThreshold", None)
        if predictionThreshold and probability < float(predictionThreshold):
            self.logger.debug("The prediction {} probability {} is lower than the threshold {}.", prediction, probability, predictionThreshold)
            return None
        else:
            self.logger.debug("Prediction: {}, probability: {}", prediction, probability)
            return int(prediction)
action call digit
Figure 12. The action call for an attribute of type drawing

The action call screen shows all action arguments.

action call digit result
Figure 13. The action call result for a digit recognition

If the action has been called, the result is shown below the action label. If the result can’t be fully shown in the action list, you may tap the result to see the details.

action call doodle drawing
Figure 14. The action call argument editor for a doodle drawing

Drawing panels can be configured in a corresponding action definition, where a color, a background color etc. could be specified.

The definition of the action that requires drawing a doodle
class DrawAndUploadDoodle(Action):
    def onConfigure(self):
        self.withLabel("Draw and upload a doodle").withDescription("Shows a canvas to draw a doodle and uploads it to the server")
        self.withArg(
            BinaryType("image").withLabel("Doodle").withMimeType("image/png")
                   .withFeatures({"characteristic":"drawing", "width":300, "height":250, "background":"FFFFFF", "color":"000000", "strokeWidth":5})\
        )
        self.withResult(StringType().withLabel("Status"))
    def onCall(self, image):
        fileName = str(System.currentTimeMillis()) + ".png"
        SpongeUtils.writeByteArrayToFile(image, sponge.getProperty("doodlesDir") + "/" + fileName)
        return "Uploaded as " + fileName
action call doodle
Figure 15. The action call for a doodle drawing as an argument

The action call screen shows all action arguments, for example a drawing.

action call arg depends
Figure 16. The action call that shows argument dependencies

Action arguments may depend on each other. Argument dependencies are supported in the action call panel and allow creating simple, interactive forms where some arguments are provided by the server, some entered by the user, some read only and some depend on the values of others. The important thing is that all that configuration is defined in an action in a knowledge base placed on the server side, not in the mobile application.

The definition of the action that provides arguments with dependencies
class DependingArgumentsAction(Action):
    def onConfigure(self):
        self.withLabel("Action with depending arguments")
        self.withArgs([
            StringType("continent").withLabel("Continent").withProvided(ProvidedMeta().withValueSet()),
            StringType("country").withLabel("Country").withProvided(ProvidedMeta().withValueSet().withDependency("continent")),
            StringType("city").withLabel("City").withProvided(ProvidedMeta().withValueSet().withDependency("country")),
            StringType("river").withLabel("River").withProvided(ProvidedMeta().withValueSet().withDependency("continent")),
            StringType("weather").withLabel("Weather").withProvided(ProvidedMeta().withValueSet())
        ])
        self.withResult(StringType().withLabel("Sentences"))
    def onCall(self, continent, country, city, river, weather):
        return "There is a city {} in {} in {}. The river {} flows in {}. It's {}.".format(city, country, continent, river, continent, weather.lower())
    def onProvideArgs(self, context):
        if "continent" in context.names:
            context.provided["continent"] = ProvidedValue().withValueSet(["Africa", "Asia", "Europe"])
        if "country" in context.names:
            continent = context.current["continent"]
            if continent == "Africa":
                countries = ["Nigeria", "Ethiopia", "Egypt"]
            elif continent == "Asia":
                countries = ["China", "India", "Indonesia"]
            elif continent == "Europe":
                countries = ["Russia", "Germany", "Turkey"]
            else:
                countries = []
            context.provided["country"] = ProvidedValue().withValueSet(countries)
        if "city" in context.names:
            country = context.current["country"]
            if country == "Nigeria":
                cities = ["Lagos", "Kano", "Ibadan"]
            elif country == "Ethiopia":
                cities = ["Addis Ababa", "Gondar", "Mek'ele"]
            elif country == "Egypt":
                cities = ["Cairo", "Alexandria", "Giza"]
            elif country == "China":
                cities = ["Guangzhou", "Shanghai", "Chongqing"]
            elif country == "India":
                cities = ["Mumbai", "Delhi", "Bangalore"]
            elif country == "Indonesia":
                cities = ["Jakarta", "Surabaya", "Medan"]
            elif country == "Russia":
                cities = ["Moscow", "Saint Petersburg", "Novosibirsk"]
            elif country == "Germany":
                cities = ["Berlin", "Hamburg", "Munich"]
            elif country == "Turkey":
                cities = ["Istanbul", "Ankara", "Izmir"]
            else:
                cities = []
            context.provided["city"] = ProvidedValue().withValueSet(cities)
        if "river" in context.names:
            continent = context.current["continent"]
            if continent == "Africa":
                rivers = ["Nile", "Chambeshi", "Niger"]
            elif continent == "Asia":
                rivers = ["Yangtze", "Yellow River", "Mekong"]
            elif continent == "Europe":
                rivers = ["Volga", "Danube", "Dnepr"]
            else:
                rivers = []
            context.provided["river"] = ProvidedValue().withValueSet(rivers)
        if "weather" in context.names:
            context.provided["weather"] = ProvidedValue().withValueSet(["Sunny", "Cloudy", "Raining", "Snowing"])
action call arg depends value set
Figure 17. The action call that shows argument dependencies and value sets

Allowed argument values can be defined in an action and provided from the server every time the action call screen is shown or an argument dependency value changes.

2.5. Action result

actions binary result
Figure 18. The action binary result

Actions may return contents that can be viewed for example as a HTML or a PDF file using the mobile OS viewers.

The definitions of the actions that return a HTML and a PDF file respectively
class HtmlFileOutput(Action):
    def onConfigure(self):
        self.withLabel("HTML file output").withDescription("Returns the HTML file.")
        self.withNoArgs().withResult(BinaryType().withMimeType("text/html").withLabel("HTML file"))
    def onCall(self):
        return String("""
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
    <head>
      <title>HTML page</title>
    </head>
    <body>
        <!-- Main content -->
        <h1>Header</h1>
        <p>Some text
    </body>
</html>
""").getBytes("UTF-8")

class PdfFileOutput(Action):
    def onConfigure(self):
        self.withLabel("PDF file output").withDescription("Returns the PDF file.")
        self.withNoArgs().withResult(BinaryType().withMimeType("application/pdf").withLabel("PDF file"))
    def onCall(self):
        return sponge.process(ProcessConfiguration.builder("curl", "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf")
                              .outputAsBinary()).run().outputBinary
actions console result
Figure 19. The action console formatted result

Actions may return a console output, for example the result of running the df -h command on the server.

The definition of the action that returns an OS command output
class OsGetDiskSpaceInfo(Action):
    def onConfigure(self):
        self.withLabel("Get disk space info").withDescription("Returns the disk space info.")
        self.withNoArgs().withResult(StringType().withFormat("console").withLabel("Disk space info"))
    def onCall(self):
        return sponge.process(ProcessConfiguration.builder("df", "-h").outputAsString()).run().outputString
actions markdown result
Figure 20. The action Markdown formatted result

Actions may return a Markdown formatted text.

2.6. User experience

dark theme
Figure 21. The application dart theme

The application may be switched to the dark theme in the settings.

2.7. Included demos

The access to actions in the mobile application is generic. However the application may include demos that use a customized UI.

2.7.1. Handwritten digit recognition

drawer digits
Figure 22. The navigation drawer if connected to a Sponge instance that supports a digit recognition

If the current connection points to a Sponge instance that has the required action that performs a handwritten digit recognition, this demo is enabled in the navigation drawer.

digits info
Figure 23. The digit recognition demo - the information dialog
digits drawing
Figure 24. The digit recognition demo - drawing a digit

The digit recognition demo screen allows drawing a digit that will be recognized by the Sponge action. After each stroke the remote action call is made and the result is shown in the circle.