Saturday, 13 July 2024
See you next week what what will probably be an extra large installment!
See you next week what what will probably be an extra large installment!
Pre-orders are now open for the Akademy 2024 T-shirt, this is only for those who will be attending Akademy, in person, in Würzburg
Pre-orders will close on 31st July
We will be opening in a few weeks separate online orders for t-shirts for those who aren't attending in person
Let’s go for my web review for the week 2024-28.
Tags: tech, politics, law, privacy
It’s time to push European governments to abandon this nonsense.
https://www.patrick-breyer.de/en/take-action-to-stop-chat-control-now/
Tags: tech, ai, machine-learning, gpt, computer-vision
Those brand new models keep failing at surprisingly simple tasks.
https://vlmsareblind.github.io/
Tags: tech, browser, security
This is a concerning finding. One can escape from the browser to the system with such chaining.
https://spaceraccoon.dev/universal-code-execution-browser-extensions/
Tags: tech, security, supply-chain, dependencies
Good tour of all the way dependencies might get compromised in your supply chain. Getting this easy to detect is needed.
https://kerkour.com/supply-chain-attacks-and-backdoored-dependencies
Tags: tech, ubuntu, security
The title is a bit pushing it. Still, I didn’t realize some of the fine prints of the Ubuntu support schemes.
https://gld.mcphail.uk/posts/ubuntu-security-updates-are-a-confusing-mess/
Tags: tech, linux, kernel, rust
It’s nice to have a balanced view on the matter. It’s not just roses and rainbows. This gives a good overview of the current limitations and where Rust can give most benefits in the kernel.
https://www.usenix.org/conference/atc24/presentation/li-hongyu
Tags: tech, linux, system, systemd
Interesting approach to test system changes. Especially welcome on immutable systems.
https://www.codethink.co.uk/articles/2024/A-new-way-to-develop-on-Linux/
Tags: tech, linux, uefi, boot, system
Looks like GRUB days as the standard bootloader are counted. Booting straight using the Linux kernel could bring interesting benefits. Probably not doable on every hardware platform though.
https://fizuxchyk.wordpress.com/2024/06/13/nmbl-we-dont-need-a-bootloader/
Tags: tech, linux, packaging
An interesting puzzle to pursue. Is it possible to rebuild exactly the same binary distribution packages?
https://blog.josefsson.org/2024/07/10/towards-idempotent-rebuilds/
Tags: tech, simd, performance
Another interesting algorithm to handle using SIMD.
https://clement-jean.github.io/simd_binary_search_tree/
Tags: tech, linux, system
It’s really a good reminder of how powerful ptrace is. You can nicely intercept and change the behavior of syscalls with it.
https://healeycodes.com/making-python-less-random
Tags: tech, cloud, storage, cryptography, security, tools
Looks like a nice tool indeed. Might be handy.
https://www.andreagrandi.it/posts/cryptomator-end-to-end-encrypt-files-in-cloud/
Tags: tech, databases
A simple explanation about dirty writes during database transactions.
https://surfingcomplexity.blog/2024/07/05/dirty-writes/
Tags: tech, databases, uuid, performance
Forced to use UUID as primary key in a table? Then make sure to use them properly to not kill the performance more than necessary. Ideally use something else though.
https://maciejwalkowiak.com/blog/postgres-uuid-primary-key/
Tags: tech, programming, python
Ever wondered how attributes work in Python under the hood? Here is how.
https://snarky.ca/unravelling-attribute-access-in-python/
Tags: tech, programming, python
An interesting Python construct to make iterators based on a simple function.
https://mathspp.com/blog/til/making-an-iterator-out-of-a-function
Tags: tech, gui, fonts
Very long read but will be an essential resource to have a fine understanding of text rendering in its current form.
Tags: tech, quality, craftsmanship, engineering, complexity, history
Interesting musing about the “software crisis” which was declared in the late 60s. We’re coping with it by piling levels of abstractions but clearly we’re still not out of it. Our craft still needs to grow.
https://wryl.tech/log/2024/the-software-crisis.html
Tags: tech, team, organization, agile
Good reminder that teams are made out of people. It’s good to look at the daily standups less as a technical management tool and more as a need to get into the work.
https://tidyfirst.substack.com/p/standups-individual-teammate
Bye for now!
Friday, 12 July 2024
KDE today announces the release of KDE Frameworks 6.4.0.
KDE Frameworks are 72 addon libraries to Qt which provide a wide variety of commonly needed functionality in mature, peer reviewed and well tested libraries with friendly licensing terms. For an introduction see the KDE Frameworks release announcement.
This release is part of a series of planned monthly releases making improvements available to developers in a quick and predictable manner.
OPC UA client code that relies on hardcoded NodeIds is brittle and often only works with a specific OPC UA server instance. This article shows the proper way to write robust and portable OPC UA client code.
Continue reading OPC UA: Programming against Type Descriptions at basysKom GmbH.
We are happy to announce the release of Qt Creator 14 RC!
Qt for Android usually supports a wide range of Android versions, some very old. To keep the supported versions to a level that’s maintainable by Qt, especially for LTS releases which are expected to live for three years, Qt for Android is adopting new guidelines for selecting the supported versions for a given Qt release in the hope that this effort would make the selection clear and transparent, and help shape proper expectations of support for each Qt for Android release.
Building system images for embedded devices from the ground up is a very complex process, that involves many different kinds of requirements for the build tooling around it. Traditionally, the most popular build systems used in this context are the Yocto project and buildroot.
These build systems make it easy to set up toolchains for cross-compilation to the embedded target architecture, so that the lengthy compilation process can be offloaded to a more beefy machine, but they also help with all the little details that come up when building an image for what effectively amounts to building an entire custom Linux distribution.
More often than not, an embedded Linux kernel image requires changing the kernel config, as well as setting compiler flags and patching files in the source tree directly. A good build system can take a lot of pain away for these steps, by simplifying this with a declarative interface.
Finally, there is a lot of value in deterministic and reproducible builds, as this allows one to get the very same output image regardless of the context and circumstances where and when the compilation is performed.
Today we take a look at Nix as an alternative to Yocto and buildroot. Nix is a purely functional language that fits all of the above criteria perfectly. The project started as a PhD thesis for purely functional software deployment and has been around for over 20 years already. In the last few years, it has gained a lot of popularity in the Server and Desktop Linux scene, due to its ability to configure an entire system and solve complex packaging-related use cases in a declarative fashion.
In the embedded scene, Nix is not yet as popular, but there have already been success stories of Nix being used as an alternative to Yocto. And with the vast collection of over 80,000 officially maintained packages in the nixpkgs repo (this is more than all official packages of Debian and Arch Linux combined), Nix certainly has an edge over the competition, as most software stacks are already packaged. For most common hardware you will also find an overlay of hardware-specific quirks in the nixos-hardware repository. However, since the user demographic of Nix is slightly different at the moment, for more obscure embedded platforms you are still much more likely to find an OpenEmbedded layer.
For the Nix syntax it is best to refer to the official documentation, but being a declarative language, the following parts should be easy to comprehend even without further introduction. The only uncommon caveat is the syntax for lambdas, which essentially boils down to { x, y }: x + y
being a lambda, that takes a set with two attributes x
and y
and returns their sum.
On Nix there are two possible approaches to do cross-compilation. The first one would be to just pull in the packages for the target architecture (in this case aarch64) and compile it on a x86_64 system by configuring qemu-user as a binfmt_misc handler for user space emulation. While this effectively cheats around the actual cross-compilation by emulating the target instruction set, it has some advantages such as simplifying the build process for all packages, but most importantly it allows to reuse the official package cache, which has already built binaries for most aarch64 packages. While most of the build process can be shortcut with this, packages that need to be actually built will build extremely slow due to the emulation.
For that reason we use the second approach instead, which is actual cross-compilation: Instead of pulling in packages normally, we can use the special pkgsCross.${targetArch}
attribute to cross-compile our packages to whatever we pass as ${targetArch}
. The majority of the packages will just work, but rarely some packages need extra configuration to cross-compile. For example, for Qt we need to set QT_HOST_PATH
to point to a Qt installation on the build host, as it needs to run some tools such as moc
during the actual build on the build host. The disadvantage of this approach is that for most packages the official Nix cache does not provide binaries, so we have to build everything ourselves.
Of course builds are already cached locally by default (because they end up as a derivation in /nix/store
), but it is also possible to set up a custom self-hosted Nix cache server, so that binaries have to be built only once even across multiple machines.
As an example we will be looking into building an entire system image for the Raspberry Pi 3 Model B from the ground up. This includes compiling a custom Linux kernel from source, building the entire graphics stack including Mesa and the whole rest of the software stack. This also means we will build Qt 6 from source.
As example application we will deploy GammaRay, which is already packaged in the official Nix repository. This is to illustrate the advantage of having the large collection of nixpkgs at our disposal. Building a custom Qt application would not be much more involved, for reference take a look at how GammaRay itself is packaged.
Then at the end of the build pipeline, we will have an actual image that can be flashed onto the Raspberry Pi to boot a custom NixOS image with all the settings we have configured.
To build a system image, nixpkgs provides a lot of utility helper functions. For example, to build a normal bootable ISO that can install NixOS like the official installer, the isoImage module can be used. Even more helper functions are available in the nixos-generators repository. However, we do not want to create an “installer” image, instead we are interested in creating an image that can be booted straight away and already has all of the correct software installed. And because the Raspberry Pi uses an SD card, we can make use of the sd-card/sd-image.nix module for that. This module already does a lot of the extra work for us, i.e. it creates a MBR partitioned SD card image, that contains a FAT boot partition and an ext4 partitioned root partition. Of course it is possible to customize all these settings, for example, to add other partitions, we could simply append elements to the config.fileSystems
attribute set.
Leaving out some slight Nix flakes boilerplate (we will get to this at the end), with the following two snippets we would already create a bootable image:
nixosConfigurations.pi = system.nixos {
imports = [
"${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
nixosModules.pi
];
images.pi = nixosConfigurations.pi.config.system.build.sdImage;
};
This first snippet imports the sd-image
module mentioned above and links to a further nixosModules.pi
configuration, that we define in the following snippet and that we can use to configure the entire system to our liking. This includes installed packages, setup of users, boot flags and more.
nixosModules.pi = ({ lib, config, pkgs, ... }: {
environment.systemPackages = with pkgs; [
gammaray
];
users.groups = {
pi = {
gid = 1000;
name = "pi";
};
};
users.users = {
pi = {
uid = 1000;
password = "pi";
isSystemUser = true;
group = "pi";
extraGroups = ["wheel" "video"];
shell = pkgs.bash;
};
};
services = {
getty.autologinUser = "pi";
};
time.timeZone = "Europe/Berlin";
boot = {
kernelPackages = lib.mkForce pkgs.linuxKernel.packages.linux_rpi3;
kernelParams = ["earlyprintk" "loglevel=8" "console=ttyAMA0,115200" "cma=256M"];
};
networking.hostName = "pi";
});
Thus, with this configuration we install GammaRay, set up a user that is logged in by default and uses a custom Raspberry Linux kernel as the default kernel to boot. Most of these configuration options should be self-explanatory and they only show a glimpse of what is possible to configure. In total, there are over 10,000 official NixOS configuration options, that can be searched with the official web interface.
The line where we add GammaRay to the system packages, also automatically adds Qt 6 as a dependency. However, as mentioned previously, this does not work quite well with cross-compilation. In order for it to build we need to patch a dependency of GammaRay to add the QT_HOST_PATH
variable to the cmake flags. What would involve bizarre gymnastics with most other build systems, becomes incredibly simple with Nix: we just add an overlay that overrides the Qt6 package, there is no need to touch the definition of GammaRay at all:
nixpkgs.overlays = [
(final: super: {
qt6 = super.qt6.overrideScope' (qf: qp: {
qtbase = qp.qtbase.overrideAttrs (p: {
cmakeFlags = ["-DQT_HOST_PATH=${nixpkgs.legacyPackages.${hostSystem}.qt6.qtbase}"];
});
});
})
];
And note, how we pass the path to Qt built on the host system to QT_HOST_PATH
. Due to lazy evaluation, this will build Qt (or rather download it from the Nix binary cache) for the host architecture and pass the resulting derivation as string at evaluation time.
In order to quickly test an image, we can write a support script to test the output directly in qemu instead of having to flash it on real hardware:
qemuTest = pkgs.writeScript "qemuTest" ''
zstd --decompress ${images.pi.outPath}/sd-image/*.img.zst -o qemu.img
chmod +w qemu.img
qemu-img resize -f raw qemu.img 4G
qemu-system-aarch64 -machine raspi3b -kernel "${uboot}/u-boot.bin" -cpu cortex-a53 -m 1G -smp 4 -drive file=qemu.img,format=raw -device usb-net,netdev=net0 -netdev type=user,id=net0 -usb -device usb-mouse -device usb-kbd -serial stdio
'';
Here we decompress the output image, resize it so that qemu
can start it and then use the uboot boot loader to finally boot it.
Taking a final look at our config, we now have the following flake.nix
file:
{
description = "Cross compile for Raspberry";
inputs = {
nixpkgs.url = "nixpkgs/nixos-23.11";
};
outputs = { self, nixpkgs, nixos-hardware }: let
hostSystem = "x86_64-linux";
system = nixpkgs.legacyPackages.${hostSystem}.pkgsCross.aarch64-multiplatform;
pkgs = system.pkgs;
uboot = pkgs.ubootRaspberryPi3_64bit;
in rec {
nixosConfigurations.pi = system.nixos {
imports = [
"${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
nixosModules.pi
];
};
images.pi = nixosConfigurations.pi.config.system.build.sdImage;
qemuTest = pkgs.writeScript "qemuTest" ''
zstd --decompress ${images.pi.outPath}/sd-image/*.img.zst -o qemu.img
chmod +w qemu.img
qemu-img resize -f raw qemu.img 4G
qemu-system-aarch64 -machine raspi3b -kernel "${uboot}/u-boot.bin" -cpu cortex-a53 -m 1G -smp 4 -drive file=qemu.img,format=raw -device usb-net,netdev=net0 -netdev type=user,id=net0 -usb -device usb-mouse -device usb-kbd -serial stdio
'';
nixosModules.pi = ({ lib, config, pkgs, ... }: {
environment.systemPackages = with pkgs; [
uboot
gammaray
];
services = {
getty.autologinUser = "pi";
};
users.groups = {
pi = {
gid = 1000;
name = "pi";
};
};
users.users = {
pi = {
uid = 1000;
password = "pi";
isSystemUser = true;
group = "pi";
extraGroups = ["wheel" "video"];
shell = pkgs.bash;
};
};
time.timeZone = "Europe/Berlin";
boot = {
kernelParams = ["earlyprintk" "loglevel=8" "console=ttyAMA0,115200" "cma=256M"];
kernelPackages = lib.mkForce pkgs.linuxKernel.packages.linux_rpi3;
};
networking.hostName = "pi";
nixpkgs.overlays = [
(final: super: {
makeModulesClosure = x: super.makeModulesClosure (x // { allowMissing = true; }); # workaround for https://github.com/NixOS/nixpkgs/issues/154163
qt6 = super.qt6.overrideScope' (qf: qp: {
qtbase = qp.qtbase.overrideAttrs (p: {
cmakeFlags = ["-DQT_HOST_PATH=${nixpkgs.legacyPackages.${hostSystem}.qt6.qtbase}"];
});
});
unixODBCDrivers = super.unixODBCDrivers // {
psql = super.unixODBCDrivers.psql.overrideAttrs (p: {
nativeBuildInputs = with nixpkgs.legacyPackages.${hostSystem}.pkgsCross.aarch64-multiplatform.pkgs; [unixODBC postgresql];
});
}; # workaround for odbc not building
})
];
system.stateVersion = "23.11";
});
};
}
And that’s it, now we can build and test the entire image with:
nix build .#images.pi --print-build-logs
nix build .#qemuTest
./result
Note that this will take quite a while to build, as everything is compiled from source. This will also create a flake.lock
file pinning all the inputs to a specific version, so that subsequent runs will be reproducible.
Nix has been growing a lot in recent years, and not without reason. The Nix language allows to solve some otherwise very complicated packaging tasks in a very concise way. The fully hermetic and reproducible builds are a perfect fit for building embedded Linux images, and the vast collection of packages and configuration options allow to perform most tasks without ever having to leave the declarative world.
However, there are also some downsides when compared to the Yocto project. Due to the less frequent use of Nix on embedded, it is harder to find answers and support for embedded-related questions and you are quickly on your own, especially when using more obscure embedded platforms.
And while the Nix syntax in and of itself is very simple, it should not go unmentioned that there is a lot of complexity around the language constructs such as derivations and how everything interacts with each other. Thus, there is definitely a steep learning curve involved, though most of this comes with the territory and is also true for the Yocto project.
Hence overall, Nix is a suitable alternative for building embedded system images (keeping in mind that some extra work is involved for more obscure embedded platforms), and its purely functional language makes it possible to solve most tasks in a very elegant way.
If you like this article and want to read similar material, consider subscribing via our RSS feed.
Subscribe to KDAB TV for similar informative short video content.
KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.
The post Using Nix as a Yocto Alternative appeared first on KDAB.
Hi there! The past few weeks have been really busy with my final exams, so I had to slow down my work. Here’s a brief status report on my progress over the past 4 weeks:
I created a SubtitleEvent class to help us better manage subtitle event information, which can replace the original SubtitledTime class. To distinguish subtitles from different layers, I also added basic display support for subtitle layers as multiple subtitle tracks.
Currently, I’m focused on refining these features. There are still some minor tasks to complete and bugs to fix. You can find more information at this MR.
Stay tuned!
Every once in a while people ask me about my email routine, so I thought I’d write about it here.
Everything I do starts with the philosophy that work and project email is a task queue. Therefore an email is a to-do list item someone else has assigned to me.
Ugh, how horrible! Better get that stuff done or rejected as soon as possible so I can move on to the stuff I want to do.
This means my target is inbox zero; achieving it means I got all my tasks done. Like everyone, I don’t always achieve it, but zero is the goal. How do I work towards it?
When I’m not in KDE mode, I want to be able to turn that stuff off in my own brain. To accomplish this, I have a home email account and a KDE email account. I adjust all my KDE accounts to only send email to my KDE address.
To manage multiple email accounts without going insane, I avoid webmail. In addition to not supporting multi-account workflows, it’s usually slow, lacking useful features, and has poor keyboard navigation.
I currently use Thunderbird, but I’m investigating moving to KDE’s KMail. Regardless, it has to be a desktop email client that offers mail rules.
I configure my email client with mail rules to automatically tag emails with colored labels according to what they are, and then mark them as read:
This results in almost all newly-arrived emails becoming colored and marked as read:
When I get a new kind of automated email that didn’t automatically receive a color label, I adjust the rules to match that new email so it gets categorized in the future, too.
When I first open my email client in the morning, everything will be categorized except 5-15 emails sent by actual people. To see just these, I’ll filter the inbox by unread status, since all the auto-categorized colored emails got automatically marked as read.
Then it’s time to figure out what to do with them. For anything that needs a response or action today, I mark it as urgent by hitting the “1” key. For anything that needs a personal response in the next few days, I hit “9” to tag it as personal and it becomes green. And so on.
Any emails that don’t need a response get immediately deleted. I never miss them. It’s fast and painless. Put those emails out of their misery.
Urgent means urgent; first I’ll go through these one at a time, and action them somehow. This means one of the following:
The email always ends up deleted! For people like us emails are not historical records, they’re tasks. Do you need to remember what tasks you performed 8 years ago on Tuesday, May 11th? Of course not. Don’t be a digital hoarder; delete your emails. You won’t miss them.
At this point I may realize that I was overzealous in tagging something as urgent. That’s fine; I just re-tag it as something else, and then I’ll get to it later.
Since my day job is “quality assurance manager”, these are important. I’ll go through every automated email from invent.kde.org about merge requests for repos I’m subscribed to and action them somehow:
More deletion! I never keep these emails around; they’re temporal notifications of other people’s work. Nothing worth preserving.
My web browser is now filled up with tabs for merge requests to review. Now it’s time to do that for relevant bug reports. I follow the same process here: open the bug report in my web browser because it needs a comment or other action from me, and then delete the email — or else immediately delete the email because it’s not directly actionable. Delete, delete, delete. It’s the happiest word when it comes to email. Everyone hates emails; delete them! Show them you mean business.
At this point I’ve spent between 15 and 40 minutes just on email, ugh. Time to do some actual work! So now I’ll spend the next several hours going through those tabs in my web browser, from left to right. First reviewing merge requests, then handling the relevant bug reports (closing, re-opening, replying to comments, changing metadata, marking as duplicate, CCing others, etc). During this step, I’ll also triage the day’s new bug reports.
Sometimes I’ll check email again while doing these, since more will be coming in. It’s easy to delete or action them individually.
After all these tabs are closed, hooray! I have some time to be proactive instead of reactive! Usually this amounts to 0-120 minutes a day during working hours. I try to spend this time on fixing small bugs I found throughout the day, opening and participating in discussion topics about important matters, working on the KDE HIG, and sometimes helping people out on http://www.discuss.kde.org or http://www.reddit.com/r/KDE.
Towards the end of the day I’ll look at the emails marked as “Personal” and “KDE e.V./Akademy” and try to knock a few out. It’s okay if I’m too tired; these aren’t urgent and can wait until tomorrow. After a few days of sitting there, I’ll mark them as urgent.
And that’s pretty much it! This is just my workflow; it doesn’t need to be yours. But in case you want to try it, here are answers to some anticipated objections:
It really doesn’t.
On a Monday maybe it takes more like 35 or 40 minutes since there are emails from the weekend to process. But on Tuesday through Friday, it’s closer to 15-20 minutes. Often 10 on Friday. Thanks to the automatic categorization, all of this is much faster than manually looking at every email one by one, and much more effective than getting depressed by hundreds of unread emails in the morning and ignoring them.
You won’t.
But if that’s too scary or painful, set up your email account or client app to archive “deleted” messages in permanent storage rather than truly deleting them. Just keep in mind that you’ll eventually run out of storage space and have to deal with that problem in the future. Once it happens, consider it an opportunity to reconsider, asking yourself how many emails you actually did need to dig out of cold storage. I’m guessing the number will be very low, maybe even 0.
Maybe so, but the general principle of automatically tagging (but not moving) emails applies to anyone. I firmly believe that anyone can benefit from this part. Make the software do the grunt work for you!
If you’re one of those people who has 50,000 emails in your inbox, select all and delete. You won’t miss any of them.
Seriously. All of them. Every single one. Right now. Just do it.
How do I know this is fine?
The good news is that you can get better at this anytime, but it’s almost impossible without making a clean break with a messy past. You’ll be looking at old stuff forever and won’t have time for new stuff.
You need to unsubscribe from some things. Maybe a lot of things. Longtime contributors to any project will have accumulated years worth of subscriptions to sources of emails that are no longer relevant. Prune them!
This may trigger Fear Of Missing Out. Recognize that and fight against it. You can almost always reduce your email load by unsubscribing from this stuff:
Resist the temptation to filter these emails into folders that you tell yourself you’ll remember to look at once in a while. You probably won’t, and by the time you do, everything in them won’t be actionable anymore — if it ever was in the first place. Unsubscribe and delete!