Testing application first use experience

Saturday, 16 November 2024 | Volker Krause

When working on an application it’s not uncommon to be testing with your own configuration and data, and often more in a power-user setup of that application. While that has advantages it’s easy to lose sight of how the application looks and behaves when first opened in a clean environment.

Testing in a clean environment

Testing the first use experience is technically easy, you just have to delete the entire application state and configuration, or create a new user account. However that’s very cumbersome and thus wont be done regularly.

Fortunately there are more convenient and less invasive shortcuts.

Isolated XDG environment

For many applications we get very far already by separating the XDG directories. That includes configuration files, application data and state as well as cached data.

This means creating four new directories and pointing the following environment variables to one of those:

  • XDG_CACHE_HOME (cached data)
  • XDG_CONFIG_HOME (configuration files)
  • XDG_DATA_HOME (application data)
  • XDG_STATE_HOME (application state)

Running an application in such an environment will make it not see any of its existing state and configuration (without destroying that). That is, as long as the entire state and configuration is actually stored in those locations.

A somewhat common exception is credential storage in a platform service like Secret Service or KWallet. Those wont be isolated and depending on the application you might not get a clean first use state or you might be risking damaging the existing state.

Multi-instance Akonadi

Other services used by an applications might need special attention as well. A particularly complex one in this context is Akonadi, as it contains a lot of configuration, state and data.

Fortunately Akonadi has built-in support for running multiple isolated instances for exactly that reason. All we need is setting the AKOANDI_INSTANCE environment variable to a unique name and we get our own separated instance.

Automation

Given the above building blocks we can create a little wrapper script that launches a given application in a clean ephemeral environment:

import os
import subprocess
import sys
import tempfile

xdgHome = tempfile.TemporaryDirectory(prefix='testing-')
for d in ['CACHE', 'CONFIG', 'DATA', 'STATE']:
    os.mkdir(os.path.join(xdgHome.name, d))
    os.environ[f"XDG_{d}_HOME"] = os.path.join(xdgHome.name, d)

os.environ['AKONADI_INSTANCE'] = 'testing'

subprocess.call(sys.argv[1:])

subprocess.call(['akonadictl', '--instance', 'testing', 'stop', '--wait'])
xdgHome.cleanup()

I’ve been using this on Itinerary since some time, and it became additionally useful with the introduction of Appium-based UI tests, as those run in a similarly isolated environment.

If you need something slightly longer living, launching a shell with this wrapper is also possible. In that you then can launch your application multiple times, e.g. for testing whether changes are persisted correctly.

Limitations

There’s one unsolved issue with how this isolates applications though: D-Bus. Applications claiming a unique D-Bus service name wont be able to run alongside a second instance this way, so you will have to shut down an already running instance during testing. In most cases that’s not a big deal, but quite inconvenient when working on one of your main communication apps.

I looked at two possible ways to isolate D-Bus (both relatively easy to integrate in a wrapper script):

  • xdg-dbus-proxy: This can limit access to certain host services, but has no way of having a second isolated instance of a service.
  • Running a separate D-Bus session bus: Having a second instance of a service is no problem then, but we have no way to access host services anymore (which means also no credential storage service etc).

Neither of those help with the applications I work on, but they might nevertheless be viable in other scenarios.

Overall, the more entangled an application is in platform state, the harder it becomes to achieve this kind of isolation, and the more you’ll need to customize how to do this. It quickly pays off though, an easy and always available way to quickly test things in a clean state has been super helpful.