Reproducible Builds, Read-only FS and How a Persistent QML Cache then Can Ruin Your Day

Saturday, 5 November 2022 | Andreas Cord-Landwehr

tl;dr : There is nothing wrong with any of the mechanisms in the subject, you just have to be careful when combining them all.

Long title, but the combination is important. Recently, I had an embedded device on my desk, which drove me to claiming quite strongly that “this is not possible what is happening in front of me!!!” If you are familiar with setups of all the points above, this article might be interesting to you.

Starting from the very strange effects I had. The device itself in question has a read-only root file system, as it is common for embedded devices. On this device a QtQuick application is running and, because I have not the most efficient processor, QML cache is enabled. Instead of going the build-time cache generation route, I have a persistent writable partition on the device, on which the QML cache is generated and stored at first start of the QtQuick application after first boot. Note that the cache needs to be persistently stored since otherwise the whole performance improvement on startup is moot.

So far so good, everything works well… until we are starting to update the software or more precisely the root file system. For this example, and this will be important later, the update of the root file system just updates my QtQuick application and its QML file but not the Qt version. What I then see after the update and the following boot is a system where the QML application still looks like before the update. Looking deeper at the file system, everything seems fine, files are updated, even QML files are updated, but the application just ignores them. But even worse, the application now randomly crashes because the executable and the shared libraries apparently do not match to the QML code being executed. — I will shorten this up, because with the intro the problem is quite obvious: the QML cache is not being invalidated even if it should and old versions of the QML files are used to run the applications. But how can this be?!

How a File in QML Cache is Invalided

All the magic that decides if the cache file is up to date or not essentially is located in qv4executablecompilationunit.cpp. Reading that code, we find the following checks:

  1. Check for the right magic key: Ie. here is a basic sanity check that tells if the cache was generated by the right tooling.
  2. Check for the cache file having been created by using the same Qt version as is used to executed the application.
  3. Check if the cache file was being created with the exact same QML_COMPILE_HASH: This value essentially is the git revision of the QtDeclarative module and thus forces a cache invalidation whenever the QtDeclarative module changes (see here for the generation process with Qt 6, in Qt5 it is similar but just with QMake). As I see this, this check is mostly a Qt developer use case with often changing QtDeclarative versions.
  4. Check if the cache file fits to the QML source file: Since all the previous checks are about the Qt versions, there is a final check that checks if the last modified date of the QML file under question is the same or a different one as the one for which the data is in the cache.
  5. Note that there is no further check for e.g. the file’s hash value, which is obviously a performance optimization topic because we are doing the whole QML cache process essentially to speed up startup.

I do not think that much further explanations are required that tell why this can break fundamentally when we are building a root file system image in some environment like Yocto that fixes all timestamps in order to make builds reproducible (Many thanks to the NixOS folks for their initial analysis! Apparently we independently hit this issue at nearly the same time.)

The Origin of the Modified Timestamp

Ok, we now know that the modified timestamp of the QML file (this is what eg. “stat -c%Y MyFile.qml” gives you as result) is the key ingredient for making the cache working correctly in our setting. For this, we have to differentiate between two ways how QML files might land on our root file system:

  1. As ordinary files, which most probably are placed somewhere in /usr/lib/qml/…
  2. As baked-in resource files inside the deployed binaries via the Qt Resource System.

The first case is fairly simple. Here, we have to look into the process on how the root file system image is created (in case of package based distros this is different and you have to check how the packaging system is handling this!). In my case, the root file system is generated by Yocto and there is a global BitBake value called REPRODUCIBLE_TIMESTAMP_ROOTFS, which forces all files inside the root file system to have the same modified time stamp during image creation.

The second case is more interesting though. Here the SOURCE_DATE_EPOCH environment variable is used to set the modified date of the source files to a certain value. Note that one needs such a mechanism in order to make the build really reproducible because one cannot rely on the file date extracted or checkout out sources, which also may further change due to patches being applied during the build process. Rather, we want to use the timestamp of the last commit or a timestamp that is baked into a release tarball.
Yocto, as most modern meta build systems, does exactly this and sets this value for the build from the source data (for details look into Poky). Further, rcc (Qt’s resource compiler) picks this value up and sets the modified timestamps of the QML files correctly while baking the files into the binaries.

Solving the Issue (for Yocto Builds)

Since Yocto already handles SOURCE_DATA_EPOCH correctly, just use a fixed REPRODUCIBLE_TIMESTAMP_ROOTFS and be happy 😉 And here, I urge you to not try workarounds like setting REPRODUCIBLE_TIMESTAMP_ROOTFS=””, because this triggers fallbacks at different places of the code. Eg. you will find /etc/timestamp being the actual time of the build but then you will note later that the modified time stamp of the files in the root file system is now the date of the latest Poky revision. Funny, heh ;D So, never rely of fallbacks, because the tend to be inconsistent.

A much better way is to couple the rootfs timestamp in your project to something that is regularly updated, at least once for every release. I am in the lucky position to usually have a meta-layer/submodule with release notes where I know for sure that there is a commit before every release. Thus, I can simply add the following line in my conf/local.conf:

REPRODUCIBLE_TIMESTAMP_ROOTFS:="${@os.popen('git -C "$BBPATH/../sources/my-layer" log -1 --pretty=%ct 2>/dev/null').read().rstrip()}"

It is not that important what exactly to use to initialize the timestamp, as long as it changes often enough, as long as it still makes your build reproducible.

And again, you only have to really worry about all of the above if you have a QML cache on a persistent partition while updating the root files system.