Building a kiosk for Home Assistant from scrap parts - Part 2

With a project of this type, the hardware preparation is just the tip of the iceberg. It was thoroughly explained in the last post, despite one last change still being pending. What lacks is basically the addition of a resistive touch panel to the front of the screen, in order not to depend on the mouse as a pointer/input device. The panel is still somewhere between China and my location..

The first aspect that I found important to cover, now that I had this Android based kiosk up and running, was the ability to remote control it and launch arbitrary applications and services on startup.

I didn't want to limit it to a single purpose, given the potential to interact and provide multiple types of information to the user. My initial idea was to at least expose a view of Home Assistant, and as such allow some lovelace cards to be used.

Opening Progressive Web Apps programmatically

In order to open certain web sites in the least cluttered way possible (i.e. not having to waste screen real estate with status bars and the browser address bar), I wanted to launch the Home Assistant web UI in the form of a PWA (Progressive Web App). This way Android would be able to launch the Chrome browser maximized.

Given that HA is prepared for installing its page as a PWA, that was not the challenging part of the project. For any compatible site, that is a matter of selecting the corresponding option in the Chrome menu, after having loaded the page:

Once that is done, an icon to the application becomes available in the home screen:

The challenge however was the fact that I wanted the PWA to be loaded on bootup or because of an external trigger. Unlike a regular application which can be easily commanded to start from a shell script (i.e. via the "am start" command with the appropriate intent and/or component being specified), in this case it wasn't so simple.

In practice this application is started in the form of a Chrome instance that is launched with a specific set of settings.

In order to give a little context, in this particular system (the now somewhat old Android Kitkat 4.4), every icon in the home screen is represented as an entry in a SQLite database that belongs to the launcher component.

This database is located in the file:

/data/data/com.android.launcher3/databases/launcher.db

Inside it, there is a table called "favorites":

root@rk3188:/data/data/com.android.launcher3/databases # sqlite3 launcher.db
SQLite version 3.7.11 2012-03-20 11:35:50
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
android_metadata  favorites         workspaceScreens

In this table each shortcut to an application, or group of applications is registered, including the shortcuts to these PWA applications. This includes details for starting the application (e.g. intent, package, extra params, shortcut icon, etc). If we look at the schema of this table, we can have an idea about the meaning of each column (this is just an overview, much of this information is publicly documented as part of the Android development documentation):

sqlite> .schema favorites
CREATE TABLE favorites (
   _id INTEGER PRIMARY KEY,
   title TEXT,
   intent TEXT,
   container INTEGER,
   screen INTEGER,
   cellX INTEGER,
   cellY INTEGER,
   spanX INTEGER,
   spanY INTEGER,
   itemType INTEGER,
   appWidgetId INTEGER NOT NULL DEFAULT -1,
   isShortcut INTEGER,
   iconType INTEGER,
   iconPackage TEXT,
   iconResource TEXT,
   icon BLOB,
   uri TEXT,
   displayMode INTEGER,
   appWidgetProvider TEXT,
   modified INTEGER NOT NULL DEFAULT 0
);

My first attempt at launching the PWA from the command line, was by trying to launch Chrome with some of these extra parameters, but the result would always consist of Chrome starting and loading the correct page, but in the standard mode with the tabs and the address bar visible.

It was only after I passed all the extra arguments present in the database entry corresponding to the PWA, that I was able to launch it as expected, with the command line (via the am start command).

Looking at the contents of the entry:

sqlite> select * from favorites where title = "Assistant";
68|Assistant|#Intent;action=com.google.android.apps.chrome.webapps.WebappManager.ACTION_START_WEBAPP;package=com.android.chrome;B.org.chromium.chrome.browser.is_icon_generated=false;S.org.chromium.chrome.browser.webapp_scope=http%3A%2F%2F192.168.1.20%3A8123%2F;l.org.chromium.chrome.browser.theme_color=-16537100;S.org.chromium.chrome.browser.webapp_short_name=Assistant;i.org.chromium.chrome.browser.webapp_source=7;i.org.chromium.chrome.browser.webapp_shortcut_version=3;S.org.chromium.chrome.browser.webapp_id=21851434-9443-46e2-a6b7-1d96cfadb136;i.org.chromium.chrome.browser.webapp_display_mode=3;l.org.chromium.chrome.browser.background_color=-1;B.org.chromium.chrome.browser.webapp_icon_adaptive=false;i.org.chromium.content_public.common.orientation=0;S.org.chromium.chrome.browser.webapp_icon=iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAABHNCSVQICAgIfAhkiAAADZBJREFU%0AaIHNmWusZldZx3%2F%2FtfZ%2B33Od6Vx62k6nhenM0DKSUmgLxALSIDExEcFYgy1IM0Jrg

...

y1j3r9p%2B7f0y23n5Ojc1ENzuzHfsit%0ALjsOSaFfnTzx6Z%2B%2F7NgPntGK5seybYFta%2B7a4jYEtwEf%2BGHD%2BgGtxfCBLWrrwP8ASqEdtN7pzG8A%0AAAAASUVORK5CYII%3D%0A;S.org.chromium.chrome.browser.webapp_name=Home%20Assistant;S.org.chromium.chrome.browser.webapp_mac=pAlmtbOGf09R0%2B6yGyGXsr5ZcCNyPuG6Eo7k4f%2F4VgY%3D%0A;S.org.chromium.chrome.browser.webapp_url=http%3A%2F%2F192.168.1.20%3A8123%2F%3Fhomescreen%3D1;end|75|0|0|1|1|1|1|-1||1|||▒PNG

||||1587921049390

It was relatively easy to derive what information and how it had to be passed. In order to automate the call, I started by building a shell script.

By looking at how the fields containing URLs and base64 values were represented, I could easily tell that these fields were URL Encoded. E.g.

ldZx3%2F%2FtfZ%2B33O

or

S.org.chromium.chrome.browser.webapp_scope=http%3A%2F%2F192.168.1.20%3A8123%2F

As such, as long as it would work anyway, I preferred to have these values URL decoded in the script. This would make it clearer to visually understand the URLs for example.

The script itself boils down to this (the icon itself is truncated in this example, the complete value is the entire byte array of the png image, encoded in base64):

#!/system/bin/sh

ACTION="com.google.android.apps.chrome.webapps.WebappManager.ACTION_START_WEBAPP"

ICON=iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABXAvmHAAAABHNCSVQICAgIfAhkiAAADZBJREFUaIHNmWusZldZx3//tfZ+33Od6Vx62k6nhenM0DKSUmgLxALSIDEx...

ICON_ADAPTIVE=false

IS_ICON_GENERATED=false

WEBAPP_NAME="Hass.io"

WEBAPP_ID=21851434-9443-46e2-a6b7-1d96cfadb136

WEBAPP_SHORTCUT_VER=3

WEBAPP_SOURCE=7

WEBAPP_URL="http://192.168.1.20:8123/?homescreen=1"

WEBAPP_SCOPE="http://192.168.1.20:8123/"

WEBAPP_MAC=pAlmtbOGf09R0+6yGyGXsr5ZcCNyPuG6Eo7k4f/4VgY=

WEBAPP_ORIENTATION=0

WEBAPP_DISPLAY_MODE=3

BROWSER_BG_COLOR=-1

BROWSER_THEME_COLOR=-16537100

EXTRA_KEYS="--es org.chromium.chrome.browser.webapp_title $WEBAPP_NAME \

           --es org.chromium.chrome.browser.webapp_name $WEBAPP_NAME \

           --es org.chromium.chrome.browser.webapp_short_name $WEBAPP_NAME \

           --ei org.chromium.chrome.browser.webapp_shortcut_version $WEBAPP_SHORTCUT_VER \

           --ei org.chromium.chrome.browser.webapp_source $WEBAPP_SOURCE \

           --ei org.chromium.chrome.browser.webapp_display_mode $WEBAPP_DISPLAY_MODE \

           --es org.chromium.chrome.browser.webapp_icon $ICON \

           --ez org.chromium.chrome.browser.webapp_icon_adaptive $ICON_ADAPTIVE \

           --es org.chromium.chrome.browser.webapp_id $WEBAPP_ID \

           --es org.chromium.chrome.browser.webapp_url $WEBAPP_URL \

           --es org.chromium.chrome.browser.webapp_mac $WEBAPP_MAC \

           --ei org.chromium.content_public.common.orientation $WEBAPP_ORIENTATION \

           --el org.chromium.chrome.browser.background_color $BROWSER_BG_COLOR \

           --el org.chromium.chrome.browser.theme_color $BROWSER_THEME_COLOR \

           --ez org.chromium.chrome.browser.is_icon_generated $IS_ICON_GENERATED \

           --es org.chromium.chrome.browser.webapp_scope $WEBAPP_SCOPE"

echo $EXTRA_KEYS

/system/bin/am start -a $ACTION $EXTRA_KEYS

Basically we split the action and the extra keys that have to be passed to the command, in individual variables. The "am start" command is then called with all these values passed.

By playing with the value of WEBAPP_URL, I learned that Chrome will only open the page as a PWA if the URL is kept intact. Even a small change to the URL will cause Chrome to open in the standard mode. I have then found (by taking a peek at Chromium source code) that this happens because the URL is validated against the WEBAPP_MAC. The later is a digest that is generated based on the original URL and another key in the device. For some reason Google have taken seriously the protection against tampering with this URL. Changing the other extra key, the WEBAPP_SCOPE, doesn't prevent the page from opening as a PWA, but also doesn't seem to have any effect.

This ends up being another caveat, at least in the following two circumstances: 

i) the PWA is not created based on the URL you were originally navigating to before installing the webapp. Instead it is created based on the manifest.json file that is exposed by the site. For example:

{

   "background_color":"#FFFFFF",

   "description":"Home automation platform that puts local control and privacy first.",

   "dir":"ltr",

   "display":"standalone",

   "icons":[

      {

         "purpose":"maskable any",

         "sizes":"192x192",

         "src":"/static/icons/favicon-192x192.png",

         "type":"image/png"

      },

      {

         "purpose":"maskable any",

         "sizes":"384x384",

         "src":"/static/icons/favicon-384x384.png",

         "type":"image/png"

      },

      {

         "purpose":"maskable any",

         "sizes":"512x512",

         "src":"/static/icons/favicon-512x512.png",

         "type":"image/png"

      },

      {

         "purpose":"maskable any",

         "sizes":"1024x1024",

         "src":"/static/icons/favicon-1024x1024.png",

         "type":"image/png"

      }

   ],

   "lang":"en-US",

   "name":"Home Assistant",

   "prefer_related_applications":true,

   "related_applications":[

      {

         "id":"io.homeassistant.companion.android",

         "platform":"play"

      }

   ],

   "short_name":"Assistant",

   "start_url":"/?homescreen=1",

   "theme_color":"#03A9F4"

}

The URL that Chrome will register for the PWA will be the one defined in the start_url element. This prevents from doing useful things such as navigating automatically to the page of interest.

ii) because the URL is restricted to the one originally present in the manifest file, we also cannot add parameters to the request. This could be useful for passing information to an intermediate system such as a load balancer or proxy, or to the web application itself.

I believe the only obvious way of overcoming this problem is to replicate the WEBAPP_MAC generation somewhere else, in order to pass the verification that is performed by Chrome. At the time of this writing I haven't spent time in investigating how to do that. I ended up sticking to the regular Chrome mode where automatic navigation to specific pages was needed.

Launching applications on boot

In spite of a conceptually simple task, launching an Android app or other executable code during boot, may present some challenges.

The first aspect is the level of access to the device. In a unrooted device it may be challenging or impossible to have control of when we want our service or app to launch during bootup. By design, the Android platform defines that for an app to launch on boot, it must register for the appropriate intent, more specifically, the "android.intent.action.BOOT_COMPLETED" intent. As such, the apps that do not declare this, cannot otherwise be configured to launch on boot.

A way of dealing with this limitation, is to create a custom app that launches on boot (it receives the intent shown above) and is capable of loading every other app that we want to be launched during startup.

If we don't want to resort to building or using an app that will do this (e.g. because apps will only be loaded late in the bootup process), we are limited to attempting to change the linux startup configuration, if we are able to.

Even in a rooted device (which is my case) there are some challenges. The main challenge is that Google created their own init system which is very different from the traditional SysV style startup system (created by AT&T for the UNIX OS with that name) that we are used to see in many Linux distributions. You can find detailed information about it here:

https://android.googlesource.com/platform/system/core/+/master/init/README.md

Is not that it is difficult to understand and express our configuration changes, but there is a more physical problem associated to reconfiguring the init.rc files: in many devices these files are kept in a recovery partition. During the initial phase of the bootup process, the init.rc files are unpacked into a temporary file system (ramdisk), and read from there. While the system is running, even though we can modify these files, it is of no use, as the contents is not persisted after a shutdown or a reboot. The only way is to be able to repack and rewrite the recovery partition with our changes. If the systems expects the boot image to be validated against a signature, it can be a challenge to perform valid changes to this partition.

As such I assumed that changing the init.rc files would not be a feasible option. I had to search for a way of circumventing that limitation. By analysing the existing init.rc files, I found that one of the scripts, in particular the /init.rk30board.rc pointed to a script located in another partition:

service runremotectrl /system/bin/run.sh
    enable
    oneshot

In my device, the /system/bin/run.sh script didn't exist, and as this service was set to enabled, this seemed like something that could be exploited. Also this partition was writeable, which means that I could create the missing run.sh script and place commands to be executed during startup in there.

And so it was. Created a simple shell script in that location, with the things I wanted to initialize or start:

#!/system/bin/sh

echo "Starting additional services and apps..."

# crond initialization:

/data/data/berserker.android.apps.sshdroid/home/.bin/crond -b -c /data/crontab

# starts ADB via TCP:

start adbd

# force the sysbar to be hidden:

/system/bin/settings put system sysbar_hide 1

# forces systemui to reload and assume the setting (sysbar hidden):

/data/data/berserker.android.apps.sshdroid/home/.bin/killall com.android.systemui

# open the Hass.io webapp:

# /data/my_scripts/show_hassio_pwa_page.sh

# opens the photoframe app:

/data/my_scripts/show_photoframe.sh

# ensures that the system starts with the screen saver on:

/data/my_scripts/enable_display_power_save.sh

In order to make it work, had to take into account some considerations. Basically, the shebang line had to point to a slightly different interpreter location than is usual:

#!/system/bin/sh

And also, some system commands had to be called with the full pathname, like for example:

/system/bin/settings

otherwise the interpreter cannot locate and execute these commands. Apparently these paths are only setup later during the bootup.

In order to be able to write to the /system partition, it must first be remounted in read/write mode. For that effect the following command has to be executed prior to creating or modifying the files:

# mount -o rw,remount /system

After finishing all the changes, it is a good idea to remount the partition again in read-only mode:

# mount -o ro,remount /system

Remote control of the Kiosk

As a new device in my home automation portfolio, I wouldn't be able to classify it as functionally complete, if it would not be possible to control it from within Home Assistant automation rules or scripts.

At first I considered an interesting possibility to install some kind of app, linux service or script that could enable the kiosk to subscribe or publish messages to MQTT topics, and interpret these as commands. However the relevant apps that I could find, would only provide basic MQTT client type of functionality. These served for things such as allowing the user to control other IoT devices by publishing messages to topics from these other devices. I would be interested in something that could for example receive MQTT payloads containing (adb) shell commands and execute these.

Discarding that approach, I decided to consider the  ADB add on for Home Assistant:

Using this addon I could send arbitrary commands to the Kiosk, which would kind of be enough for what I needed. I only had to make sure that adbd (ADB Daemon) would be running permanently, be launched during boot, and listen on a TCP port, allowing for a remote connection.

For that effect, as I have shown above, this can be done simply by including the following command in the startup script:

# starts ADB via TCP:

start adbd


This will cause the daemon to launch and stay running. By default this daemon is only launched (this is defined in the init.rc scripts) when a USB debug connection is established. For those who are more curious, this is defined in the init.rk30board.usb.rc script:

on property:sys.usb.config=mass_storage,adb
    write /sys/class/android_usb/android0/enable 0
    write /sys/class/android_usb/android0/idVendor 2207
    write /sys/class/android_usb/android0/idProduct 0010
    write /sys/class/android_usb/android0/functions ${sys.usb.config}
    write /sys/class/android_usb/android0/enable 1
    start adbd
    setprop sys.usb.state ${sys.usb.config}

Once having the add-on installed in HA, and the adbd running, it is now a matter of configuring the new integration.

The first step is to add the kiosk to the /config/configuration.yaml file:

media_player:

  - platform: androidtv

    name: HA Kiosk

    host: 192.168.1.40

    adb_server_ip: 127.0.0.1

The adb_server_ip points to the local instance of the adb server, in this case in HA itself. This client side of the adb connection is also considered a server, because it bridges between a local (adb) client and the remote server located in the Android device.

Once this is setup, it becomes a matter of using the new integration. After a reboot, the new entity should appear in the states view:

One simple automation that I have added was to turn on the screen once a person passes close to the entry door, as it has sensors and is where the Kiosk is placed.

The rule boils down to having a trigger based on the state change for the door sensors, and as the action, invoking via the adb integration, a shell script located in the Kiosk. This shell script causes the screen to turn on:

- id: passing_by_the_entry_door

  alias: "User passes by the entry door or enters the house"

  trigger:

    platform: state

    entity_id: 

      - binary_sensor.entry_door_state

      - binary_sensor.entry_door_motion_detector

    to: 'on'

  action: 

    - service: androidtv.adb_command

      data:

        entity_id: media_player.ha_kiosk

        command: /data/my_scripts/turn_on_screen.sh

On a later post I will detail some tweaks I had to do, in order to control a certain aspects, such as powering off the screen when the device is sitting idle without having received user input for a while.

In this particular Android build, the screensaver functions were partially disabled and left misconfigured, because probably as an HDMI dongle they assumed it should not never need put the TV/monitor to sleep when sitting idle (it makes some sense but it would be nice to be left as a configuration).

Future work

Besides the touch panel (which is on its way), another thing that I found that will be important to add is a hardware watchdog timer (WDT). I found that occasionally the device crashes or it takes a long time to recover from misbehaved processes. One that I have found is the adbd process which sometimes hangs, causing the entire system to slow down, by hogging the CPU. During this project:

https://www.creationfactory.co/2016/03/repurposing-old-android-phone.html

I have built a hardware watchdog timer because of the same type of problem. It was integrated with a Samsung Android phone (in that case flashed with CyanogenMod). It was based on a PIC microcontroller, which would cycle the power of the phone in case a signal would not be received from the phone after a certain interval of time. On the phone side, a crontab script would periodically trigger the I/O port that instead of the vibration motor, had an input of the PIC connected to. This would cause the PIC watchdog timer to reset just before overflowing. If this would happen (e.g. due to the Android system becoming slow or unresponsive), the PIC would set an output PIN high, and a transistor would close the circuit to which the phone power button would be connected.

I will probably salvage this WDT and integrate in this project, or do something equivalent with an Arduino or similar device.