Wednesday, 19 February 2025

All core22 KDE snaps are broken. There is not an easy fix. We have used kde-neon repos since inception and haven’t had issues until now.
libEGL fatal: DRI driver not from this Mesa build (‘23.2.1-1ubuntu3.1~22.04.3’ vs ‘23.2.1-1ubuntu3.1~22.04.2’)
Apparently Jammy had a mesa update?
Option 1: Rebuild our entire stack without neon repos ( fails due to dependencies not in Jammy, would require tracking down all of these and build from source )
Option 2: Finish the transition to core24 ( This is an enormous task and will take some time still )
Either option will take more time and effort than I have. I need to be job hunting as I have run out of resources to pay my bills. My internet/phone will be cut off in days. I am beyond stressed out and getting snippy with folks, for that I apologize. If someone wants to sponsor the above work then please donate to https://gofund.me/fe30793b otherwise I am stepping away to rethink life and my defunct career.
I am truly sorry everyone.
New core24 Snaps:

Arianna – Epub viewer

k3b – Disc burner
Snapcraft:
Fixes for the qt5 kde-neon extension
Monday, 17 February 2025
Fahrenheit, new releases and bugfixes
Welcome to a new issue of "This Week in KDE Apps"! Every week we cover as much as possible of what's happening in the world of KDE apps. This time again a bit delayed due to some personal travel.
Releases
- Kaidan 0.11.0 is out. This new version of KDE's XMPP client brings Qt6 support as well as a few new features.
- Tellico 4.1.1 is out with a few minor fixes.
- Amarok 3.2.2 is out with some minor bugfixes, and improvements for building Amarok on non-UNIX systems and without X11 support.
KDE Itinerary Digital travel assistant
Temperature displayed in Itinerary will now use Fahrenheit units when you set your home country to the USA. (Joshua Goins, 25.04.0. Link)
Kasts Podcast application
Improved the volume button to use an adaptive icon depending on the volume level. (Bart De Vries, 25.04.0. Link)
Kate Advanced text editor
Added a button to clear the debug output in the debug plugin. (Waqar Ahmed, 25.04.0. Link)
Added a button to switch between a normal diff (with only a few lines of context) and a full diff with all the context. (Leo Ruggeri, 25.04.0. Link)
KOrganizer KOrganizer is a calendar and scheduling application
Fixed showing the details of a recurrent event. (Allen Winter, 25.04.0. Link)
Konsole Use the command line interface
Fixed some freezing issues when starting Konsole and any applications using Konsole KPart like Kate. (Waqar Ahmed, 24.12.3. Link)
Merkuro Calendar Manage your tasks and events with speed and ease
Added an option to filter the tasks to only displays the ones due today. (Shubham Shinde, 25.04.0. Link)
SystemDGenie
Ported the editor to KTextEditor. (Thomas Duckworth. Link)

…And Everything Else
This blog only covers the tip of the iceberg! If you’re hungry for more, check out Nate's blog about Plasma and be sure not to miss his This Week in Plasma series, where every Saturday he covers all the work being put into KDE's Plasma desktop environment.
For a complete overview of what's going on, visit KDE's Planet, where you can find all KDE news unfiltered directly from our contributors.
Get Involved
The KDE organization has become important in the world, and your time and contributions have helped us get there. As we grow, we're going to need your support for KDE to become sustainable.
You can help KDE by becoming an active community member and getting involved. Each contributor makes a huge difference in KDE — you are not a number or a cog in a machine! You don’t have to be a programmer either. There are many things you can do: you can help hunt and confirm bugs, even maybe solve them; contribute designs for wallpapers, web pages, icons and app interfaces; translate messages and menu items into your own language; promote KDE in your local community; and a ton more things.
You can also help us by donating. Any monetarnky contribution, however small, will help us cover operational costs, salaries, travel expenses for contributors and in general just keep KDE bringing Free Software to the world.
To get your application mentioned here, please ping us in invent or in Matrix.
Saturday, 15 February 2025
The Amarok Development Squad is happy to announce the immediate availability of Amarok 3.2.2, the second bugfix release for Amarok 3.2 "Punkadiddle"!
3.2.2 features some minor bugfixes, and improvements for building Amarok on non-UNIX systems and without X11 support. Additionally, a 16-year-old feature request has been fulfilled. Concluding years of Qt5 porting and polishing work, Amarok 3.2.2 is likely to be the last version with Qt5/KF5 support, and it should provide a nice and stable music player experience for users on various systems and distributions. The development in git, on the other hand, will soon switch the default configuration to Qt6/KF6, and focus for the next 3.3 series will be to ensure that everything functions nicely with the new Qt version.
Changes since 3.2.1
FEATURES:
- Try to preserve collection browser order when adding tracks to playlist (BR 180404)
CHANGES:
- Allow building without X11 support
- Various build fixes for non-UNIX systems
BUGFIXES:
- Fix DAAP collection connections, browsing and playing (BR 498654)
- Fix first line of lyrics.ovh lyrics missing (BR 493882)
Getting Amarok
In addition to source code, Amarok is available for installation from many distributions' package repositories, which are likely to get updated to 3.2.2 soon, as well as the flatpak available on flathub.
Packager section
You can find the tarball package on download.kde.org and it has been signed with Tuomas Nurmi's GPG key.
Welcome to a new issue of "This Week in Plasma"! Every week we cover as much as possible of what's happening in the world of KDE Plasma and its associated apps like Discover, System Monitor, and more.
Plasma 6.3 is out! So far the response has been very good, but of course a few issues were found once it was in the wild.
Maybe the worst issue is something that KWin devs have actually tracked down to a bug in the GCC compiler, of all things! It only manifests with the kind of release build configurations that many distros use, and also only with GCC 15 and an ICC profile set up. We've informed distros how to work around it until the root cause is understood and GCC gets patched, or KWin devs are able to guard against it internally.
Unfortunately this is a sign that we did not in fact get enough beta testers, since the issue should have been obvious to people in affected environments. Another sign is that most of the regressions are hardware-related. We've got them fixed now, but we need people to be testing the betas with their personal hardware setups! There's simply no way for a small pool of KDE developers to test all of these hardware setups themselves.
Anyway, with those caveats aside, it looks like it's been a pretty smooth release! Building on it, there have been a number of positive changes to the Media Player widget, Weather Report Widget, Info Center Energy page, and touchscreen support.
Notable new Features
Plasma 6.4.0
The Media Player widget now features a playback rate selector when the source media makes this feature available using its MPRIS implementation. (Kai Uwe Broulik, link)

Notable UI Improvements
Plasma 6.3.1
Improved the presentation of search results for the new DWD weather provider in the Weather Report widget. (Ismael Asensio, link 1 and link 2)
The BBC Weather provider has recently improved the quality of their forecast data, so we've changed the weather widget to no longer hide search results from it if there are results from other providers as well. (Ismael Asensio, link)
The updates list in Discover is now sorted case-insensitively. (Aleix Pol Gonzalez, link)
Welcome Center now remembers its window size (and on X11, position) across launches, like most of our other QML app windows these days. (Tracey Clark, link)
Plasma 6.4.0
Improved the graph view on Info Center's Energy page: Now it's in a card, like in System Monitor, and has more normal and visually pleasing margins. (Ismael Asensio, link 1 and link 2)

Spectacle has gained support for pinch-zooming in its screenshot viewer window, which can be especially useful when annotating using a touchscreen. (Noah Davis, link)
You can now actually scroll through the Widget Explorer with a single-finger touchscreen scroll gesture, because dragging widgets using touch now requires a tap-and-hold. (Niccolò Venerandi, link)
Notable Bug Fixes
Plasma 6.3.1
Fixed a regression that would cause KWin to crash in the X11 session when hotplugging or switching between HDMI screens. (Fushan Wen, link 1 and link 2). Consider it a reminder for everyone still on X11 to try the Wayland session again, because the X11 session receives almost no testing from developers anymore!
Fixed a regression that could cause KWin to sometimes crash hours after hotplugging a Thunderbolt dock. (Xaver Hugl, link)
Fixed a regression that would cause KWin to crash when you interact with the Alt+Tab task switcher while using software rendering. (Vlad Zahorodnii, link)
Fixed a regression that could cause certain Qt-based apps to crash on launch when using the Breeze style. (Antonio Rojas, link)
Fixed a case where Plasma might sometimes crash when clicking on the Networks icon in the System Tray, especially when built using GCC 15. (David Edmundson, link)
Fixed a regression that caused the new "Prefer efficiency" ICC color mode setting to not actually improve efficiency on certain hardware. (Xaver Hugl, link)
Panels in auto-hide mode no longer inappropriately hide again when you start dragging Task Manager tasks to re-order them. (Tino Lorenz, link)
The new bar separator between the date and time in the Digital Clock widget no longer appears inappropriately when the date has been intentionally suppressed. (Christoph Wolk, link)
Fixed an issue that broke the layout of the device tiles on Info Center's Energy page when using a larger-than-default font size or loads of devices with batteries. (Ismael Asensio, link)
Fixed two keyboard navigation issues in the Power and Battery widget. (Ismael Asensio, link 1 and link 2)
Fixed an older issue that prevented the keyboard brightness controls on certain laptops from being visible immediately. (Nicolas Fella, link)
Fixed an older issue that caused Info Center's Energy page to vibrate disturbingly at certain window sizes. It was, heh heh heh… very high energy! (Ismael Asensio, link)
Qt 6.8.3
Committed a better Qt fix for the issue whereby the first click after dragging Task Manager tasks got ignored. (David Redondo, link)
Other bug information of note:
- 1 very high priority Plasma bug (same as last week). Current list of bugs
- 27 15-minute Plasma bugs (up from 25 last week). Current list of bugs
- 86 KDE bugs of all kinds fixed over the past week. Full list of bugs
How You Can Help
KDE has become important in the world, and your time and contributions have helped us get there. As we grow, we need your support to keep KDE sustainable.
You can help KDE by becoming an active community member and getting involved somehow. Each contributor makes a huge difference in KDE — you are not a number or a cog in a machine!
You don’t have to be a programmer, either. Many other opportunities exist:
- Triage and confirm bug reports, maybe even identify their root cause
- Contribute designs for wallpapers, icons, and app interfaces
- Design and maintain websites
- Translate user interface text items into your own language
- Promote KDE in your local community
- …And a ton more things!
You can also help us by making a donation! Any monetary contribution — however small — will help us cover operational costs, salaries, travel expenses for contributors, and in general just keep KDE bringing Free Software to the world.
To get a new Plasma feature or a bugfix mentioned here, feel free to push a commit to the relevant merge request on invent.kde.org.
Friday, 14 February 2025
About Me
I’m Rishav Ray Chaudhury, a third year Electrical Engineering undergrad from India. I have participated in Season of KDE this year. I am tasked with implementing a variant of the Mancala board game called Kalah under the guidance of João Gouveia and Benson Muite.
Why I chose this project
As with any aspiring developer, I too started out with game development. A couple of my friends and I started to make games using C# and the Unity framework in my first year of University. The feeling of making games that my friends enjoyed was exhilarating to say the least. This was also around the same time that I learned about open-source software. Apparently, a lot of software that developers frequently use, was developed by a group of passionate programmers. I tried contributing to some repos but was unsuccessful, mainly because I was completely unfamiliar with the projects. That all changed when I started using Arch Linux. For Arch, the desktop environment that I chose was KDE Plasma. After using it for some time, I came to know about Season of KDE and finally took the initiative to start contributing to software that I frequently used. Of the projects, the game development project was the one that caught my eye and now here I am developing a game for it.
The second maintenance release of the 24.12 cycle is out with multiple bug fixes. Notable changes include fixes for crashes, UI resizing issues, effect stack behavior, proxy clip handling, and rendering progress display, along with improvements to Speech-to-text in Flatpak and macOS packages.
- Don’t try to update monitor overlay if effect is disabled. Commit.
- Fix crash setting empty name for folder. Commit. Fixes bug #499070.
- Better fix for expand library clips broken with proxies. Commit. Fixes bug #499171.
- Try to fix Whisper models folder on Flatpak. Commit. See bug #499012.
- Don’t try to delete ui file elements on subtitlemanager close. Commit.
- Fix effect stack widget not properly resizing. Commit.
- Ensure built-in effects reset button is enabled. Commit.
- Ensure vidstab external files are correctly listed and archived. Commit.
- Added 2 decimals for the rotation parameter (addresses bug #498586). Commit.
- Rescale 48-apps-kdenlive.png to 48×48. Commit.
- Fix effects layout broken on resize. Commit. Fixes bug #498749.
- Fix possible crash on exit. Commit.
- Reassemble proxy profile elements in the correct order after validation. Commit. Fixes bug #485356.
- Fix the spinbox range for title position and size. Commit. Fixes bug #487950.
- Fix rendering progress not shown when rendering a zone. Commit.
- Fix FFmpeg for STT not found on Mac. Commit. Fixes bug #498949.
- Backport fix for invalid file names in custom effects. Commit. Fixes bug #498710.
The post Kdenlive 24.12.2 released appeared first on Kdenlive.
Let’s go for my web review for the week 2025-07.
Wikipedia Prepares for ‘Increase in Threats’ to US Editors From Musk and His Allies
Tags: tech, wikipedia, knowledge, politics, privacy
Some powerful bullies want to make the life of editors impossible. Looks like the foundation has the right tools in store to protect those contributors.
The Future Is Too Easy
Tags: tech, business, marketing, criticism
Alright, this piece is full of vitriol… And I like it. The CES has clearly become a mirror of the absurdities our industry is going through. The vision proposed by a good chunk of the companies is not appealing and lazy.
https://defector.com/the-future-is-too-easy?ref=DenseDiscovery-325
Microsoft Study Finds AI Makes Human Cognition “Atrophied and Unprepared”
Tags: tech, ai, machine-learning, gpt, ux, cognition, research
This is clearly pointing in the direction of UX challenges around LLM uses. For some tasks the user’s critical thinking must be fostered otherwise bad decisions will ensue.
https://www.404media.co/microsoft-study-finds-ai-makes-human-cognition-atrophied-and-unprepared-3/
Poisoning for propaganda: rising authoritarianism makes LLMs more dangerous
Tags: tech, ai, machine-learning, gpt, politics, criticism
Of course it would be less of a problem if explainability was better with such models. It’s not the case though, so it means they can spew very subtle propaganda. This is bound to become even more of a political power tool.
https://www.baldurbjarnason.com/2025/poisoning-for-propaganda/
The skill of the future is not ‘AI’, but ‘Focus’
Tags: tech, ai, machine-learning, copilot, programming, focus, learning, criticism
This is an interesting way to frame the problem. We can’t rely too much on LLMs for computer science problems without loosing important skills and hindering learning. This is to be kept in mind.
https://www.carette.xyz/posts/focus_will_be_the_skill_of_the_future/
How does a Linux machine connect to the internet, really?
Tags: tech, linux, networking
Due to NetworkManager you forgot how to setup an interface and the networking stack manually? Here is a good summary of the important steps.
https://pjg1.site/linux-internet-from-scratch
You Should Use /tmp/ More
Tags: tech, habits
Surprising idea… I guess I’ll mull this one over and maybe try. This is not a small change of habit though.
https://atthis.link/blog/2025/58671.html
Thread-safe memory copy – Daniel Lemire’s blog
Tags: tech, memory, multithreading
What do you want? Speed or safety? Ultimately you’ll have to choose one.
https://lemire.me/blog/2025/02/07/thread-safe-memory-copy/
Refined: simple refinement types for Rust
Tags: tech, type-systems, rust
A good example of what can be done when you have a rich type system.
https://jordankaye.dev/posts/refined/
How Does Ada’s Memory Safety Compare Against Rust?
Tags: tech, system, rust, ada, memory
Interesting comparison. Ada doesn’t fare as good as I’d have expected as soon as pointers are in the mix… but there is a twist, you can go a very long way without pointers in Ada.
https://ajxs.me/blog/How_Does_Adas_Memory_Safety_Compare_Against_Rust.html
About GPU Conditionals
Tags: tech, gpu, 3d, shader, optimization
This is indeed something easy to get wrong. Also this misconception is very widespread, so it’s good to debunk it.
https://iquilezles.org/articles/gpuconditionals/
Reassessing Wayland
Tags: tech, linux, wayland
This is obviously all good news on the Wayland front. Took time to get there, got lots of justified (and even more unjustified) complaints, but now things are looking bright.
https://dudemanguy.github.io/blog/posts/2025-02-03-wayland-xorg-2/wayland-xorg-2.html
Operational and Denotational Strategies for Understanding Code
Tags: tech, programming, teaching
A good reminder that you should always bring several perspectives when teaching something. This a a simple framework which can be used widely in our field.
https://noelwelsh.com/posts/operational-denotational-understanding/
Less-Than Estimation
Tags: tech, estimates
Nice and tiny estimation approach. I can see projects where this could work.
https://chrisdone.com/posts/less-than-estimation/
Managing Your Time as a Middle Manager
Tags: management, time
A couple of interesting ideas. This fluid focus concept definitely require communication around it when applied.
https://newsletter.canopy.is/p/managing-your-time-as-a-middle-manager
Bye for now!
Modern distributed systems need to process massive amounts of data efficiently while maintaining strict ordering guarantees. This is especially challenging when scaling horizontally across multiple nodes. How do we ensure messages from specific sources are processed in order while still taking advantage of parallelism and fault tolerance?
Elixir, with its robust concurrency model and distributed computing capabilities, is well-suited for solving this problem. In this article, we’ll build a scalable, distributed message pipeline that:
- Bridges RabbitMQ and Google Cloud PubSub, delivering messages from RabbitMQ queues to a PubSub topic.
- Ensures message ordering for each RabbitMQ queue.
- Scales horizontally across multiple nodes.
- Distribute the message pipelines evenly across the Elixir cluster.
- Gracefully handles failures and network partitions.
Many modern applications require processing large volumes of data while preserving message order from individual sources. Consider, for example, IoT systems where sensor readings must be processed in sequence, or multi-tenant applications where each tenant’s data requires sequential processing.
The solution we’ll build addresses these requirements by treating each RabbitMQ queue as an ordered data source.
Let’s explore how to design this system using Elixir’s distributed computing libraries: Broadway, Horde, and libcluster.
Architecture overview
The system consists of multiple Elixir nodes forming a distributed cluster. Each node runs one or more Broadway pipelines to process messages from RabbitMQ queues and forward them to Google Cloud PubSub. To maintain message ordering, each queue has exactly one pipeline instance running across the cluster at any time. If a node fails the system must redistribute its pipelines to other nodes automatically, and if a new node joins the cluster then the existing pipelines should be redistributed to ensure a balanced load.
Elixir natively supports the ability to cluster multiple nodes together so that processes and distributed components within the cluster can communicate seamlessly. We will employ the libcluster library since it provides several strategies to automatize cluster formation and healing.
For the data pipelines, the Broadway library provides a great framework to support multi-stage data processing while handling back-pressure, batching, fault tolerance and other good features.
To correctly maintain the distribution of data pipelines across the Elixir nodes, the Horde library comes to the rescue by providing the building blocks we need: a distributed supervisor that we can use to distribute and maintain healthy pipelines on the nodes, and a distributed registry that we use directly to track which pipelines exist and on which nodes they are.
Finally, a PipelineManager component will take care of monitoring RabbitMQ for new queues and starting/stopping corresponding pipelines dynamically across the cluster.

Technical implementation
Let’s initiate a new Elixir app with a supervision tree.
mix new message_pipeline --sup
First, we’ll need to add our library dependencies in mix.exs and run mix deps.get:
defmodule MessagePipeline.MixProject do
use Mix.Project
def project do
[
app: :message_pipeline,
version: "0.1.0",
elixir: "~> 1.17",
start_permanent: Mix.env() == :prod,
deps: deps()
]
end
def application do
[
extra_applications: [:logger],
mod: {MessagePipeline.Application, []}
]
end
defp deps do
[
{:libcluster, "~> 3.3"},
{:broadway, "~> 1.0"},
{:broadway_rabbitmq, "~> 0.7"},
{:google_api_pub_sub, "~> 0.34"},
{:goth, "~> 1.4"},
{:tesla, "~> 1.12"},
{:jason, "~> 1.4"},
{:amqp, "~> 4.0"}
]
end
end
Rolling our own PubSub client
Since we just need to publish messages to Google Cloud PubSub and the API is straightforward, we don’t need fancy libraries here.
Let’s start by adding Goth to the supervision tree to manage Google’s service account credentials.
defmodule MessagePipeline.Application do
use Application
def start(_type, _args) do
credentials =
"GOOGLE_APPLICATION_CREDENTIALS_PATH"
|> System.fetch_env!()
|> File.read!()
|> Jason.decode!()
children = [
{Goth, name: MessagePipeline.Goth, source: {:service_account, credentials}},
# Other children...
]
Supervisor.start_link(children, strategy: :one_for_one)
end
end
And here’s our HTTP client to publish messages to Google Cloud PubSub
defmodule MessagePipeline.GooglePubsub do
@google_pubsub_base_url "https://pubsub.googleapis.com"
def publish_messages(messages) when is_list(messages) do
project_id = System.fetch_env!("GOOGLE_PUBSUB_PROJECT_ID")
topic_name = System.fetch_env!("GOOGLE_PUBSUB_TOPIC")
topic = "projects/#{project_id}/topics/#{topic_name}"
request = %{
messages: Enum.map(messages, &%{data: Base.encode64(&1)})
}
with {:ok, auth_token} <- generate_auth_token(),
client = client(auth_token),
{:ok, response} <- Tesla.post(client, "/v1/#{topic}:publish", request) do
%{body: %{"messageIds" => message_ids}} = response
{:ok, message_ids}
end
end
defp client(auth_token) do
middleware = [
{Tesla.Middleware.BaseUrl, @google_pubsub_base_url},
Tesla.Middleware.JSON,
{Tesla.Middleware.Headers, [{"Authorization", "Bearer " <> auth_token}]}
]
Tesla.client(middleware)
end
defp generate_auth_token do
with {:ok, %{token: token}} <- Goth.fetch(MessagePipeline.Goth) do
{:ok, token}
end
end
end
Clustering with libcluster
We’ll use libcluster to establish communication between our Elixir nodes. Here’s an example configuration that uses the Gossip strategy to form a cluster between nodes:
defmodule MessagePipeline.Application do
use Application
def start(_type, _args) do
topologies = [
gossip_example: [
strategy: Elixir.Cluster.Strategy.Gossip
]
]
children = [
{Cluster.Supervisor, [topologies, [name: MessagePipeline.ClusterSupervisor]]},
# Other children...
]
Supervisor.start_link(children, strategy: :one_for_one)
end
end
Distributed process management with Horde
We’ll use Horde to manage our Broadway pipelines across the cluster. Horde ensures that each pipeline runs on exactly one node and handles redistribution when nodes fail.
Let’s add Horde’s supervisor and registry to the application’s supervision tree.
The UniformQuorumDistribution distribution strategy distributes processes using a hash mechanism among all reachable nodes. In the event of a network partition, it enforces a quorum and will shut down all processes on a node if it is split from the rest of the cluster: the unreachable node is drained and the pipelines can be resumed on the other cluster nodes.
defmodule MessagePipeline.Application do
use Application
def start(_type, _args) do
children = [
{Horde.Registry, [
name: MessagePipeline.PipelineRegistry,
members: :auto,
keys: :unique
]},
{Horde.DynamicSupervisor, [
name: MessagePipeline.PipelineSupervisor,
members: :auto,
strategy: :one_for_one,
distribution_strategy: Horde.UniformQuorumDistribution
]}
# Other children...
]
Supervisor.start_link(children, strategy: :one_for_one)
end
end
Broadway pipeline implementation
Each pipeline uses Broadway to consume messages from RabbitMQ and publish them to Google PubSub.
A strict, per-queue ordering is guaranteed by setting a concurrency of 1.
defmodule MessagePipeline.Pipeline do
use Broadway
alias Broadway.Message
def child_spec(opts) do
queue_name = Keyword.fetch!(opts, :queue_name)
pipeline_name = pipeline_name(queue_name)
%{
id: pipeline_name,
start: {__MODULE__, :start_link, opts}
}
end
def start_link(opts) do
queue_name = Keyword.fetch!(opts, :queue_name)
pipeline_name = pipeline_name(queue_name)
pipeline_opts = [
name: {:via, Horde.Registry, {MessagePipeline.PipelineRegistry, pipeline_name}},
producer: [
module: {
BroadwayRabbitMQ.Producer,
queue: queue_name,
connection: [
host: System.fetch_env!("RABBITMQ_HOST"),
port: String.to_integer(System.fetch_env!("RABBITMQ_PORT")),
username: System.fetch_env!("RABBITMQ_USER"),
password: System.fetch_env!("RABBITMQ_PASSWORD")
]
},
concurrency: 1
],
processors: [
default: [
concurrency: 1
]
],
batchers: [
default: [
batch_size: 100,
batch_timeout: 200,
concurrency: 1
]
]
]
case Broadway.start_link(__MODULE__, pipeline_opts) do
{:ok, pid} ->
{:ok, pid}
{:error, {:already_started, _pid}} ->
:ignore
end
end
def pipeline_name(queue_name) do
String.to_atom("pipeline_#{queue_name}")
end
@impl true
def handle_message(_, message, _) do
message
|> Message.update_data(&process_data/1)
end
@impl true
def handle_batch(_, messages, _, _) do
case publish_to_pubsub(messages) do
{:ok, _message_ids} -> messages
{:error, reason} ->
# Mark messages as failed
Enum.map(messages, &Message.failed(&1, reason))
end
end
defp process_data(data) do
# Transform message data as needed
data
end
defp publish_to_pubsub(messages) do
MessagePipeline.GooglePubsub.publish_messages(messages)
end
end
Queue discovery and pipeline management
Finally, we need a process to monitor RabbitMQ queues and ensure pipelines are running for each one.
The Pipeline Manager periodically queries RabbitMQ for existing queues. If a new queue appears, it starts a Broadway pipeline only if one does not already exist in the cluster. If a queue is removed, the corresponding pipeline is shut down.
defmodule MessagePipeline.PipelineManager do
use GenServer
@timeout :timer.minutes(1)
def start_link(opts) do
GenServer.start_link(__MODULE__, opts, name: __MODULE__)
end
def init(_opts) do
state = %{managed_queues: MapSet.new()}
{:ok, state, {:continue, :start}}
end
def handle_continue(:start, state) do
state = manage_queues(state)
{:noreply, state, @timeout}
end
def handle_info(:timeout, state) do
state = manage_queues(state)
{:noreply, state, @timeout}
end
def manage_queues(state) do
{:ok, new_queues} = discover_queues()
current_queues = state.managed_queues
queues_to_add = MapSet.difference(new_queues, current_queues)
queues_to_remove = MapSet.difference(current_queues, new_queues)
Enum.each(queues_to_add, &start_pipeline/1)
Enum.each(queues_to_remove, &stop_pipeline/1)
%{state | managed_queues: new_queues}
end
defp discover_queues do
{:ok, conn} =
AMQP.Connection.open(
host: System.fetch_env!("RABBITMQ_HOST"),
port: String.to_integer(System.fetch_env!("RABBITMQ_PORT")),
username: System.fetch_env!("RABBITMQ_USER"),
password: System.fetch_env!("RABBITMQ_PASSWORD")
)
{:ok, chan} = AMQP.Channel.open(conn)
{:ok, queues} = AMQP.Queue.list(chan)
# Filter out system queues
queues
|> Enum.reject(fn %{name: name} ->
String.starts_with?(name, "amq.") or
String.starts_with?(name, "rabbit")
end)
|> Enum.map(& &1.name)
|> MapSet.new()
end
defp start_pipeline(queue_name) do
pipeline_name = MessagePipeline.Pipeline.pipeline_name(queue_name)
case Horde.Registry.lookup(MessagePipeline.PipelineRegistry, pipeline_name) do
[{pid, _}] ->
{:error, :already_started}
[] ->
Horde.DynamicSupervisor.start_child(
MessagePipeline.PipelineSupervisor,
{MessagePipeline.Pipeline, queue_name: queue_name}
)
end
end
defp stop_pipeline(queue_name) do
pipeline_name = MessagePipeline.Pipeline.pipeline_name(queue_name)
case Horde.Registry.lookup(MessagePipeline.PipelineRegistry, pipeline_name) do
[{pid, _}] ->
Horde.DynamicSupervisor.terminate_child(MessagePipeline.PipelineSupervisor, pid)
[] ->
{:error, :not_found}
end
end
end
Let’s not forget to also add the pipeline manager to the application’s supervision tree.
defmodule MessagePipeline.Application do
use Application
def start(_type, _args) do
children = [
{MessagePipeline.PipelineManager, []}
# Other children...
]
Supervisor.start_link(children, strategy: :one_for_one)
end
end
Test the system
We should now have a working and reliable system. To quickly test it out, we can configure a local RabbitMQ broker, a Google Cloud PubSub topic, and finally a couple of Elixir nodes to verify that distributed pipelines are effectively run to forward messages between RabbitMQ queues and PubSub.
Let’s start by running RabbitMQ with the management plugin. RabbitMQ will listen for connections on the 5672 port, while also exposing the management interface at http://localhost:15672. The default credentials are guest/guest.
docker run -d --name rabbitmq \
-p 5672:5672 \
-p 15672:15672 \
rabbitmq:3-management
Next, install and use the gcloud CLI to create a Google Cloud project, a PubSub topic, and a a service account to access PubSub programmatically.
# Login to Google Cloud
gcloud auth login
# Create a new project (or use an existing one)
gcloud projects create message-pipeline-test
gcloud config set project message-pipeline-test
# Enable PubSub API
gcloud services enable pubsub.googleapis.com
# Create a topic
gcloud pubsub topics create test-topic
# Create service account for local testing
gcloud iam service-accounts create local-test-sa
# Generate and download credentials
gcloud iam service-accounts keys create ./google-credentials.json \
--iam-account local-test-sa@message-pipeline-test.iam.gserviceaccount.com
# Grant publish permissions
gcloud pubsub topics add-iam-policy-binding test-topic \
--member="serviceAccount:local-test-sa@message-pipeline-test.iam.gserviceaccount.com" \
--role="roles/pubsub.publisher"
It’s now time to start two terminal sessions where we can export the needed environment variables before running two Elixir nodes.
# In terminal 1
export RABBITMQ_HOST=localhost
export RABBITMQ_PORT=5672
export RABBITMQ_USER=guest
export RABBITMQ_PASSWORD=guest
export GOOGLE_PUBSUB_PROJECT_ID=message-pipeline-test
export GOOGLE_PUBSUB_TOPIC=test-topic
export GOOGLE_APPLICATION_CREDENTIALS_PATH=$(pwd)/google-credentials.json
export RELEASE_COOKIE=my-secret-cookie
iex --sname node1 -S mix
# In terminal 2 (same variables)
export RABBITMQ_HOST=localhost
export RABBITMQ_PORT=5672
export RABBITMQ_USER=guest
export RABBITMQ_PASSWORD=guest
export GOOGLE_PUBSUB_PROJECT_ID=message-pipeline-test
export GOOGLE_PUBSUB_TOPIC=test-topic
export GOOGLE_APPLICATION_CREDENTIALS_PATH=$(pwd)/google-credentials.json
export RELEASE_COOKIE=my-secret-cookie
iex --sname node2 -S mix
To verify cluster formation, from one of the nodes:
# List all nodes in the cluster, should show the other node
Node.list()
# Check Horde supervisor distribution
:sys.get_state(MessagePipeline.PipelineSupervisor)
Now let’s create some test queues on RabbitMQ and start publishing some messages.
# Download rabbitmqadmin if not already available
wget http://localhost:15672/cli/rabbitmqadmin
chmod +x rabbitmqadmin
# Create queues
./rabbitmqadmin declare queue name=test-queue-1
./rabbitmqadmin declare queue name=test-queue-2
# Publish test messages
./rabbitmqadmin publish routing_key=test-queue-1 payload="Message 1 for queue 1"
./rabbitmqadmin publish routing_key=test-queue-1 payload="Message 2 for queue 1"
./rabbitmqadmin publish routing_key=test-queue-2 payload="Message 1 for queue 2"
# List queues and their message counts
./rabbitmqadmin list queues name messages_ready messages_unacknowledged
# Get messages (without consuming them)
./rabbitmqadmin get queue=test-queue-1 count=5 ackmode=reject_requeue_true
One can also use the RabbitMQ management interface at http://localhost:15672, authenticate with the guest/guest default credentials, go to the “Queues” tab, click “Add a new queue”, and create “test-queue-1” and “test-queue-2”.
After a minute, the Elixir nodes should automatically start some pipelines corresponding to the RabbitMQ queues.
# List all registered pipelines
Horde.Registry.select(MessagePipeline.PipelineRegistry, [{{:"$1", :"$2", :"$3"}, [], [:"$2"]}])
# Check specific pipeline
pipeline_name = :"pipeline_test-queue-1"
Horde.Registry.lookup(MessagePipeline.PipelineRegistry, pipeline_name)
Now, if we publish messages on the RabbitMQ queues, we should see them appear on the PubSub topic.
We can verify it from Google Cloud Console, or by creating a subscription, publishing some messages on RabbitMQ, and then pulling messages from the PubSub subscription.
gcloud pubsub subscriptions create test-sub --topic test-topic
# ...Publish messages on RabbitMQ queues...
gcloud pubsub subscriptions pull test-sub --auto-ack
If we stop one of the Elixir nodes (Ctrl+C twice in its IEx session) to simulate a failure, the pipelines should be redistributed in the remaining node:
# Check updated node list
Node.list()
# Check pipeline distribution
Horde.Registry.select(MessagePipeline.PipelineRegistry, [{{:"$1", :"$2", :"$3"}, [], [:"$2"]}])
Rebalancing pipelines on new nodes
With our current implementation, pipelines are automatically redistributed when a node fail but they are not redistributed when a new node joins the cluster.
Fortunately, Horde supports precisely this functionality from v0.8+, and we don’t have to manually stop and re-start our pipelines to have them landing on other nodes.
All we need to do is enable the option process_distribution: :active on Horde’s supervisor to automatically rebalance processes on node joining / leaving. The option runs each child spec through the choose_node/2 function of the preferred distribution strategy, detects which processes should be running on other nodes considering the new cluster configuration, and specifically restarts those particular processes such that they run on the correct node.
defmodule MessagePipeline.Application do
use Application
def start(_type, _args) do
children = [
{Horde.Registry, [
name: MessagePipeline.PipelineRegistry,
members: :auto,
keys: :unique
]},
{Horde.DynamicSupervisor, [
name: MessagePipeline.PipelineSupervisor,
members: :auto,
strategy: :one_for_one,
distribution_strategy: Horde.UniformQuorumDistribution,
process_redistribution: :active
]}
# Other children...
]
Supervisor.start_link(children, strategy: :one_for_one)
end
end
Conclusion
This architecture provides a robust solution for processing ordered message streams at scale. The combination of Elixir’s distributed capabilities, Broadway’s message processing features, and careful coordination across nodes enables us to build a system that can handle high throughput while maintaining message ordering guarantees.
To extend this solution for your specific needs, consider these enhancements:
- Adopt a libcluster strategy suitable for a production environment, such as Kubernetes.
- Tune queue discovery latency, configuring the polling interval based on how frequently new queues are created. Better yet, instead of polling RabbitMQ, consider setting up RabbitMQ event notifications to detect queue changes in real-time.
- Declare AMQP queues as durable and make sure that publishers mark published messages as persisted, in order to survive broker restarts and improve delivery guarantees. Use publisher confirms to ensure messages are safely received by the broker. Deploy RabbitMQ in a cluster with queue mirroring or quorum queues for additional reliability.
- Add monitoring, instrumenting Broadway and Horde with Telemetry metrics.
- Enhance error handling and retry mechanisms. For example, retry message publication to PubSub N times before failing the messages, thus invalidating the (possibly costly) processing operation.
- Unit & e2e testing. Consider that the gcloud CLI (gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators) contains a PubSub emulator that may come in handy: e.g. gcloud beta emulators pubsub start — project=test-project — host-port=0.0.0.0:8085
- Leverage an HorizontalPodAutoscaler for automated scaling on Kubernetes environments based on resource demand.
- Evaluate the use of Workload Identities if possible. For instance, you can provide your workloads with access to Google Cloud resources by using federated identities instead of a service account key. This approach frees you from the security concerns of manually managing service account credentials.
Thursday, 13 February 2025
Kaidan supports Qt 6 now! In addition, this release improves the user interface and fixes some bugs. Have a look at the changelog for more details.
Changelog
Features:
- Highlight public XMPP provider titles while card is expanded (melvo)
- Round corners of cards and buttons (melvo)
- Add fading in/out hover effect to map previews (melvo)
- Collapse contact profiles by default if they have more than 3 entries (melvo)
- Show colored check mark for delivered messages instead of none to avoid message bubble resizing (melvo)
Bugfixes:
- Fix opening public MUC-based group chats via another XMPP client (melvo)
- Fix playing voice messages and changing playing position (melvo)
- Fix updating message reactions that could not be sent instead of adding them a second time (melvo)
- Fix updating group chat users in user interface (melvo)
- Fix displaying message reaction details (melvo)
- Update filtering contacts by labels even if label list is not open anymore (melvo)
- Fix scrolling media overview (melvo)
- Fix updating draft messages (melvo)
Notes:
- Kaidan requires Qt 6.6 now (mlaurent, melvo, fazevedo, plata)
Download
Or install Kaidan for your distribution: