Skip to content

Monday, 26 August 2024

Every two years, the KDE community selects three goals that serve as focal points for the entire community's efforts in the coming years. This cyclical process of goal-setting and community-wide focus is a great example of KDE's Cumulative Culture in action.

This concept, typically observed in human societies, refers to the ability to build upon previous knowledge and innovations to create increasingly complex and effective solutions. In KDE's case, each cycle of goals represents a new layer of accumulated wisdom, i.e. new features and more stability.

The First Cycle (2018-2020)

The first cycle of goals laid the groundwork with its focus on community growth, privacy, and usability.

  • Streamlined Onboarding: Focused on attracting and retaining new contributors by making the onboarding process smoother and more engaging.
  • Privacy Software: Prioritized user privacy and security, ensuring KDE software respects user data and complies with security standards.
  • Usability & Productivity: Aimed to enhance the usability and productivity of KDE software, making it powerful yet easy to use.

The Second Cycle (2020-2022)

The second cycle tackled more complex challenges. Goals like Wayland implementation improvements (which layed the foundation for the Plasma 6 release), improving the app ecosystem, and ensuring consistency in design and functionality.

  • Wayland: This task aimed at stabilizing Wayland support accross KDE apps.
  • All About the Apps: Improved KDE's app infrastructure, enabling more efficient app delivery and better support services.
  • Improve Consistency across the Board: Ensured uniformity in design and functionality across KDE software, improving usability and reducing redundancy.

The Third Cycle (2022-2024)

The third cycle, which is currently coming to an end, was about progress and adaptation. A focus to include environmental responsibility, operational efficiency, and inclusive design.

  • Sustainable Software: Focused on making KDE software more energy-efficient and environmentally friendly by implementing practices that reduce resource consumption and ensure long-term sustainability.
  • Automate and Systematize Internal Processes: Aimed to streamline KDE’s internal workflows by automating repetitive tasks, adding code tests across projects and creating a Quality Assurance team to name a few.
  • KDE For All: Seeked to make KDE software accessible and inclusive for all users.

A New Cycle A Comin' (2024-2026)

Now, as we enter the fourth cycle of the KDE Goals, we see the full power of this cumulative process. Each goal, whether fully achieved or not, contributes to the collective knowledge and capability of the KDE community. Ideas and partial solutions from past cycles become a solid foundation of knowledge and experience that support future efforts.

The commmunity is currently voting on the following proposals for the next KDE Goals cycle that will guide our efforts and shape our focus for the coming years:

KDE Goals at Akademy

The three most voted goals will be announced at Akademy, where there will also be a wrap-up talk about the achievements of the current goals. Also, there will be Birds-of-a-feather (BoF) sessions with the new goal champions.

Join the Matrix room and keep an eye on the website for the latest KDE Goals updates.

KDE's Okular is the first software which got awarded with the Blue Angel label for resource and energy-efficient software products. The certification was based on the first version of the criteria for this product criteria which were introduced in 2020. Now the criteria have been updated. What has changed and what does that mean for KDE?

The revised criteria are available as version 4 on the Blue Angel web site. Only the German version is currently available; the English version will follow shortly.

New software categories

The biggest change is the scope of the label. In the past it was limited to desktop software. With the updated version, the criteria also include software on mobile devices and server software or a combination of these categories, such as a web service with mobile and desktop clients.

The biggest challenge is the measurement of the energy and resource efficiency for these new categories, which requires a more flexible approach and must accommodate scenarios where the measurement cannot be done by inserting a meter in front of the power supply of a single device. The new criteria address this by defining applicable methods for the measurement of mobile and server applications.

The extended scope covers a much broader range of software. For KDE the desktop category is most relevant, but of course a lot of software also interacts with a server component, for example an email client like KMail, which could now be treated and assessed as a combined client-server system to give more realistic and relevant results.

More flexible measurement procedure

The expansion in scope requires an expanded view on the measurement of energy and resource efficiency as well. The first version of the criteria was quite strict and prescribed a very specific measurement procedure on specified reference systems. It was based on a comparison of measurements in a representative usage scenario and in idle mode. This gave a realistic impression of what the usage of a computer program meant in terms of energy consumption.

The new criteria allow for more variation in how the measurements are carried out. The original method is still there, but variations which lead to comparable results are possible as well. This change means that a new criterion was introduced to document the way measurements are done.

In addition to the measurement of the usage scenario, a new type of measurement was introduced. This measures total energy consumption of a production system over a longer period of time. This is particular useful for server applications, where this method can lead to more realistic numbers by averaging resource consumption over real-world usage of multiple users.

For mobile applications, the measurement also has to include the data volume transmitted during a standard usage scenario and the list of URLs it has accessed. This is based on the assumption that large volumes of data transfer imply a higher energy usage. It can also be used to assess if the application is using advertisements or is collecting tracking information. Both are forbidden under the revised Blue Angel criteria.

Ongoing assessment of energy and resource efficiency

The original criteria demanded that updates of the software still run on old reference systems and that the energy consumption does not increase more than 10%. They were not very clear in how exactly this should be proven and documented. Especially for software which is released very often, testing every individual update is impractical. For mobile and even more for server software, update cycles can be very short, up to multiple updates a day.

In the updated criteria there is a more precise way of handling updates. The general idea is still there that updated software run on old hardware and energy consumption not increase too much. But it's not tied to individual updates anymore. The required procedure is to do a measurement at least once a year and publish the results as part of the documentation of the software product. This includes documentation of the measurement setups and any changes to it as well as preserving the history of measurements, so that users can judge for themselves how much energy and resource usage is increasing over time.

This procedure clarifies the requirments and opens a pragmatic way of measuring updates. It implies a certain burden on updating documentation.

Consequences for KDE and Okular

KDE holds the Blue Angel label for its PDF viewer Okular. This is desktop software and the standard usage scenario doesn't include any network access. That means that the expanded scope does not change anything for the existing certification. The revised criteria open up the opportunity to apply for the Blue Angel label for mobile software, such as KDE Connect, and mixed scenarios which also include server components, but the eco-certification for Okular is covered as it was before.

The more flexible measurement criteria give us more leeway in how we are doing the measurements. We have set up KEcoLab for being able to regularly do measurements. This setup follows the procedure prescribed in the original criteria. As this is still valid, it also means no change for us, and our measurements still fulfill the criteria. However, it gives us more opportunities to improve the lab and doesn't strictly tie us to the original list of reference systems anymore. We might want to take advantage of that.

The documentation of the measurement system is something we have always done in a transparent way, so this also doesn't require any big changes on our side. We have to consider how to best convey this in the documentation of Okular, but this is mostly a question on how we communicate the existing content.

The ongoing assessment of energy and resource efficiency ties very well into how we handle software updates. We have a continuous release stream with frequent updates and incremental changes. This fits the model of the new criteria. We have to review how we include regular updates of the documentation and measurement data in releases, but this again is mostly a question of how we communicate the existing content.

Conclusion

The revised criteria provide a welcome expansion of the Blue Angel to more categories of software and a more flexible way to do energy and resource efficiency measurements. They continue to align well with how KDE develops software in general and Okular in particular, so we do not see any issues with continuing the Blue Angel certification for Okular.

We would be happy if the new version of the criteria would increase adoption of the Blue Angel ecolabel for resource and energy efficient software. Sustainable software is an important topic and the Blue Angel can be one way of making progress in this area more visible to a broad audience.

Sunday, 25 August 2024

I finally took an evening to get NextCloudPi installed on a Raspberry Pi 5 with a large-ish NVMe drive. This was not a smooth ride. For your pleasure, this is how I got it working.

First, use Jeff Geerling’s guide to get the Pi booting from the NVMe drive.

Second, use this guide to move from Debian networking to systemd-networkd, but do not hold the avahi-daemon package.

Third, run the NextCloudPi curl install script.

Next up – the migration from my old instance. I have 1.5TB of files on a spin disk connected via USB that I need to move to the new NVMe storage – but that is for another night.

For the record – I do love NextCloud and NextCloudPi, so no finger pointing here, just sharing some frustration and how I got around the issue.

Friday, 23 August 2024

Let’s go for my web review for the week 2024-34.


Unbundling Profile: MIT Libraries - SPARC

Tags: research, copyright, open-access

It’s good to see major institutions like this get out of contracts with scientific publishing companies. Those unfortunately became mostly parasitic. Open access should be the norm for research.

https://sparcopen.org/our-work/big-deal-knowledge-base/unbundling-profiles/mit-libraries/


Make Firefox Private Again

Tags: tech, mozilla, privacy

Since they unfortunately turned on private attribution by default (why? Mozilla, why?). Here is an easy automated way to turn it off.

https://make-firefox-private-again.com/


Being on The Semantic Web is easy, and, frankly, well worth the bother

Tags: tech, web, semantic

With all those bots and scripts crawling the Web, some of the semantic web vision got silently implemented.

https://csvbase.com/blog/13


Markov chains are funnier than LLMs

Tags: tech, ai, machine-learning, gpt, markov-chains, funny

Interesting musing. The predictability in tone doesn’t make for very funny content indeed. Also as a side-effect this might help people remember that Markov chain are a thing and much less expensive.

https://emnudge.dev/blog/markov-chains-are-funny/


ugh. I picked up a shitty NUC from ewaste…

Tags: tech, security

Scary thread… developers should know better than do this and ship it on devices around the world. Their data is now anyone for the taking and users’ privacy can’t be ensured.

https://digipres.club/@foone/112990331505043510


NetAlert X

Tags: tech, networking, security, tools

Looks like a nice tool to monitor your network.

https://netalertx.com/


Free, OpenSource IPv6 Textbook

Tags: tech, book, ip, networking

Looks like an interesting resource to learn about IPv6.

https://ipv6textbook.com/


Andries Brouwer on the OOM killer

Tags: tech, linux, kernel, memory

Funny musing about the OOM killer. With nice pointers if you want to dive further into the topic.

https://quuxplusone.github.io/blog/2024/08/22/overcommit/


The Closed-Loop Benchmark Trap

Tags: tech, benchmarking

Be sure to pick the right behavior model when you make a benchmark. Otherwise you might just measure the wrong thing.

https://buttondown.com/jaffray/archive/the-closed-loop-benchmark-trap/


What is std::ref?

Tags: tech, c++

A little refresher about std::ref and std::cref. They come in handy sometimes, but also if you don’t realize you need them you’ll generate more copies than necessary.

https://www.sandordargo.com/blog/2024/08/21/std-ref


SIMD Matters :: Box2D

Tags: tech, cpu, simd, performance, physics, simulation

SIMD is hard to use, not all problems can apply to it. But when they can, the performance gain can be great.

https://box2d.org/posts/2024/08/simd-matters/


uv: Unified Python packaging

Tags: tech, python, tools

Looks like there’s another contender for package management for Python. This is sooo fragmented now… this one is compelling though.

https://astral.sh/blog/uv-unified-python-packaging


Common Causes of Memory Leaks in JavaScript

Tags: tech, javascript, memory, leak

There are many ways to create a memory leak in Javascript. Here is a good list of the things to pay attention to.

https://www.trevorlasn.com/blog/common-causes-of-memory-leaks-in-javascript


Toasts are Bad UX – Max Schmitt

Tags: tech, web, frontend, ux

It’s better than no feedback. It’s a bit lazy and far from perfect though.

https://maxschmitt.me/posts/toasts-bad-ux


Reckoning

Tags: tech, web, frontend, ux, criticism

Interesting series about the rise of the javascript frontend framework, the bad practices which came with them and the very real impacts on the users. There are indeed better ways.

https://infrequently.org/series/reckoning/


Don’t Repeat Yourself and the Strong Law of Small Numbers - iRi

Tags: tech, design, programming

This is a good point. The DRY principle has value but the trick is finding the right time to apply it.

https://jerf.org/iri/post/2024/dry_strong/


Code review antipatterns

Tags: tech, codereview

Starts like a satire, but there’s a serious conclusion in the end. Indeed, mind the power dynamics in code reviews. Be nice, steer away from those antipatterns, especially since you might be on the receiving end the next time.

https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/code-review-antipatterns/


Interview with Ron Jeffries

Tags: tech, agile, history, criticism

Very nice interview. This is an interesting reflection on the past 20+ years of Agile Software Development.

https://ronjeffries.com/articles/-x024/-v04/8/


Decision Logs

Tags: tech, product-management

Nice way to keep in check how and why behavior changes as the requests from various stakeholders come in.

https://buttondown.com/j2kun/archive/decision-logs/


Bug squash: An underrated interview question

Tags: tech, hr, interviews, debugging

This is indeed a nice way to approach technical interviews. Unfortunately it requires quite some effort to setup and maintain. You also have to find the right bugs to put in the interview and this is a rarity.

https://blog.jez.io/bugsquash/



Bye for now!

Since Plasma 5.18, nearly five years ago, Plasma has shipped with a "telemetry" system. It’s opt-in, allowing users to send a small amount of data back to us.

Was it useful or worth it? It's a question that comes up occasionally, and the answer is mixed. I believe it showed real potential, though the reality of our implementation was somewhat underwhelming and didn't really deliver. There are many lessons learned that are worth sharing with other projects that might face similar endeavours.

The good bits

Where we had data available for topics being discussed it worked. To give two concrete examples from memory:

  • A developer claimed, "No one is using a screen smaller than 1024x768," while bumping the minimum size of a window. This was proved wrong; the number of users at 800x600 or even 640x480 is surprisingly high. Still low as an overall percentage, but higher than you would ever intuitively think. Presumably, it's the default for a lot of virtual machines.

  • Four years ago, a developer claimed, "No one still uses only OpenGL2; we can change the code to do XYZ." A check of our user base showed it would have affected nearly 5% of our users, so the change was abandoned.

Interestingly, this last topic came up again very recently, as it held back colour management improvements, but in a narrower Wayland-only path and with a fallback. After checking metrics again, the usage was below 1%, so we went ahead with that merge request.

So, are metrics worth it just to stop developers and designers from making nonsense claims out of thin air? Absolutely! 90% of stats are just made up on the spot. Metrics are just as much about preventing changes as it is about sparking changes.

Indirect impact

The other important part is having a more general sense of the landscape. Currently, we have a lot of hard conversations about how quickly we push the move to Wayland. We have voices wanting to maintain support, and we have voices wanting to push quicker. These decisions shouldn't be made just by who can be the loudest. For every individual topic that came up in those discussions, I would always have in mind our current adoption value at the time.

Should we care about Nvidia? Knowing they make up about 25% of our user base makes the decision for us. I ran with an Nvidia card in one machine because of this, implementing Nvidia context loss handling and doing what we could during the Wayland transition
We don't test BSD while developing Plasma, but we also let it hold us back. Should we care about it more or less? My opinion matches exactly what the metrics say.

Some stats and graphs:

Another role of metrics is being a conversation starter—people will fawn over a graph. More topics on Reddit will be about our Wayland usage rather than the topic I'm trying to discuss. I'll focus on Wayland examples beacuse that's a topic close to me.

Wayland adoption over time

I keep tabs on what our metrics show here. We can see the slow increase from under 20% to around 45% over time, showing the progress as both we and the Wayland ecosystem evolved. At Plasma 6, we switched the defaults a small bump in the graph can be seen. but 45% still seemed rather disappointing.

Filtering on just Plasma 6 reveals the true story:

There's still 20% of users switching away, or using a distro with a different default, or having carried over presets, but it's more promising. Interestingly, we can see that the GPU vendor distribution differs between X11 and Wayland.

Problems and lessons learned

Ultimately, despite the positive parts it would be hard to call our telemetry a staggering success. For the handful of examples above, there are a hundreds of cases where we had no data to back anything up. The range of data points was pitiful and it wasn't often used

The viewing tool is really, really important!

Data collection without viewing it is meaningless. As shown above, we often need to drill down and cross-reference filters to extract conclusions.

The original plan was to use the existing UI provided by kuserfeedback, which did not scale at all and quickly fell over. It was designed for high-fidelity data for a small number of users, not what we had.

In a rush, we pivoted to using Grafana because there was already a setup hosted.

It worked—ish, but it’s not designed for this, especially combined with our data structure, which was a manual NoSQL in normal SQL. Every graph needed to be written by hand, and it felt very much like fighting the system rather than working with it. Combined with the limited access permissions granted, it wasn't used by many people.

It being used is the number one indicator of its usefulness!

We need to find a tool specifically designed for visualization and querying datasets (maybe Apache Superset?).

Time-based data just makes noise

Our system sent updates every N days with basically the same data every time. This made writing queries way messier than it should have been. It never added any value; I would always be interested in what the current stats are. As described in the Wayland usage graphs above, if I'm making a Wayland decision, it doesn't matter what most people are using; it matters what people on the latest release are using. We always ended up having to add filters to focus on just the latest version.

The upgrade story needs planning in advance

The amount of data we collected was tiny—some GPU information, screen information, language, and a few other fragments. The plan was to slowly add more and more stats over time, but we hit a wall. Our UX involved the user selecting to enable metrics and it being a fire-and-forget operation.

What do we do when we want to add more data? For example, whether you use an analog or digital clock. We would need to prompt the user and reset their settings in the meantime, which is at odds with it being a setting. The whole thing became such an ordeal that made it not worthwhile.

Wrap up

The project didn't fail, when we had data and it was used it worked, but overall our implementation falls short. I would like to open a discussion at Akademy on how we move forward with our current system potentially starting from scratch treating it more like a survey that we prompt to auto populate and submit each year.

Thursday, 22 August 2024

Hello everyone! Time flies and now we’re already in the final week of GSoC. In this blog post I’ll be sharing the progress I’ve made since my last update, focusing primarily on subtitle styling.

Subtitle Editor

The first thing I did was to enhance the existing subtitle editor. The updated editor now serves as an interface for editing ASS events, which include various components. With the new subtitle editor, we can easily modify elements such as the event’s layer, style, margins, and more. I’ve also simplified the effects section, allowing us to control subtitle scrolling by simply adjusting checkboxes and combo boxes for speed, direction, and range.

However, the most significant change is the text field and the buttons above it. To better understand these changes, it’s important to first introduce the relationship between ASS styles and events. In ASS files, each event must be assigned a valid style that applies to the entire event text. Additionally, ASS override tags are special text blocks within events that allow precise control over the styles of different parts of the text, rather than the entire text. (There are some exceptions, like “Set Position.”)

The text field has been enhanced to assist users in inputting ASS override tags using the provided buttons. For instance, when a user clicks the “Toggle Bold” button, tags are automatically inserted or adjusted to toggle the bold style for either the selected text or the text following the cursor if nothing is selected. Additionally, the text field features a highlighter that renders different parts of the tags in distinct styles, making them more distinguishable, and an auto-completer that lists all valid presets as we start typing a tag name.

For those who prefer the previous subtitle editor, which only displays the rendered text, a “Simple Editor” is also available. This editor syncs with the normal editor but displays only the text without tags while rendering some basic tag effects. However, due to the complexities of ASS tag rules, style editing in the Simple Editor can sometimes behave unpredictably. So it’s best suited for simpler use cases before or after editing styles.

Subtitle Manager

Continuing from the previous improvements, the Subtitle Manager is now integrated with style management and has been divided into four sections: File, Event, Style, and Info, which correspond to the four main components of ASS subtitles. Each section, except for the File section, includes a sidebar for switching between different subtitle files. Additionally, when in the Style section, we can drag and drop a style item onto a subtitle file name in the sidebar to efficiently move or copy styles between files. The same functionality is available in the Event section, where we can move or copy an entire layer to another file.

Misc

Style Editor

A new widget, the Style Editor, was created to edit styles and provide a preview.

Convert Old Global Style

Old styles will now be automatically converted to the “Default” style in the new project. Font size, outline, and shadow will be scaled to maintain the original effects.

Different Default Styles for Layers

Now, we can assign different default styles to each layer, which will automatically be applied to a subtitle event when it’s created on the corresponding layer. This feature is especially useful for quickly building a subtitle file with multiple speakers, allowing each speaker to have a distinct style.

Summary

It has been a wonderful summer getting involved in the KDE community and contributing to Kdenlive! I may not be the best at coding, but I’ve learned a lot throughout this journey. Thanks for everyone who has gave me guidence — Eugen Mohr, Farid Abdelnour, and especially my mentor, Jean-Baptiste Mardelle. While GSoC is coming to an end, my journey with KDE is just beginning. After these updates, I plan to continue improving subtitle functions, including making it easier for users to input more ASS override tags and refining the UI and user experience. See you in my next blog :)

Implementing an Audio Mixer, Part 1

Motivation

When using Qt Multimedia to play audio files, it’s common to use QMediaPlayer, as it supports a larger variety of formats than QSound and QSoundEffect. Consider a Qt application with several audio sources; for example, different notification sounds that may play simultaneously. We want to avoid cutting notification sounds off when a new one is triggered, and we don’t want to construct a queue for notification sounds, as sounds will play at the incorrect time. We instead want these sounds to overlap and play simultaneously.

Ideally, an application with audio has one output stream to the system mixer. This way in the mixer control, different applications can be set to different volume levels. However, a QMediaPlayer instance can only play one audio source at a time, so each notification would have to construct a new QMediaPlayer. Each player in turn opens its own stream to the system.

The result is a huge number of streams to the system mixer being opened and closed all the time, as well as QMediaPlayers constantly being constructed and destructed.

To resolve this, the application needs a mixer of its own. It will open a single stream to the system and combine all the audio into the one stream.

Before we can implement this, we first need to understand how PCM audio works.

PCM

As defined by Wikipedia:

Pulse-code modulation (PCM) is a method used to digitally represent sampled analog signals. It is the standard form of digital audio in computers, compact discs, digital telephony and other digital audio applications. In a PCM stream, the amplitude of the analog signal is sampled at uniform intervals, and each sample is quantized to the nearest value within a range of digital steps.

Here you can see how points are sampled in uniform intervals and quantized to the closest number that can be represented.

Pcm.svg

Image Source: Wikipedia

Description from Wikipedia: Sampling and quantization of a signal (red) for 4-bit LPCM over a time domain at specific frequency.

Think of a PCM stream as a humongous array of bytes. More specifically, it’s an array of samples, which are either integer or float values and a certain number of bytes in size. The samples are these discrete amplitude values from a waveform, organized contiguously. Think of the each element as being a y-value of a point along the wave, with the index representing an offset from

at a uniform time interval.

Here is a graph of discretely sampled points along a sinusoidal waveform similar to the one above:

Discrete_cosine

Image Source: Wikimedia Commons

Description from Wikimedia Commons: Image of a discrete time sinusoid

Let’s say we have an audio waveform that is a simple sine wave, like the above examples. Each point taken at discrete intervals along the curve here is a sample, and together they approximate a continuous waveform. The distance between the samples along the x-axis is a time delta; this is the sample period. The sample rate is the inverse of this, the number of samples that are played in one second. The typical standard sample rate for audio on CDs is 44100 Hz - we can’t really hear that this data is discrete (plus, the resultant sound wave from air movement is in fact a continuous waveform).

We also have to consider the y-axis here, which represents the amplitude of the waveform at each sampled point. In the image above, amplitude

is normalized such that

. In digital audio, there are a few different ways to represent amplitude. We can’t represent all real numbers on a computer, so the representation of the range of values varies in precision.

For example, let’s say we have two different representations of the wave above: 8-bit signed integer and 16-bit signed integer. The normalized value

from the image above maps to

with 8-bit representation and

with 16-bit. Therefore, with 16-bit representation, we have 128 times as many possible values to represent the same range; it is more precise, but the required size to store each 16-bit sample is double that of 8-bit samples.

We call the chosen representation, and thus the size of each sample, the bitdepth. Some common bitdepths are 16-bit int, 24-bit int, and 32-bit float, but there are many others in existence.

Let’s consider a huge stream of 16-bit samples and a sample rate of 44100 Hz. We write samples to the audio device periodically with a fixed-size buffer; let’s say it is 4096 bytes. The device will play each sample in the buffer at the aforementioned rate. Since each sample is a contiguous 2-byte short, we can fit 2048 samples into the buffer at once. We need to write 44100 samples in one second, so the whole buffer will be written around 21.5 times per second.

What if we have two different waveforms though, and what if one starts halfway through the other one? How do we mix them so that this buffer contains the data from both sources?

Waveform Superimposition

In the study of waves, you can superimpose two waves by adding them together. Let’s say we have two different discrete wave approximations, each represented by 20 signed 8-bit integer values. To superimpose them, for each index, add the values at that index. Some of these sums will exceed the limits of 8-bit representation, so we clamp them at the end to avoid signed integer overflow. This is known as hard clipping and is the phenomenon responsible for digital overdrive distortion.

xWave 1 (y1​)Wave 2 (y2​)Sum (y1+y2​)Clamped Sum
0+60−100−40-40
1−120+80−40−40
2+40+70+110+110
3−110−100−210−128
4+50−110−60−60
5−100+60−40−40
6+70+50+120+120
7−120−120−240−128
8+80−100−20−20
9−80+40−40−40
10+90+80+170+127
11−100−90−190−128
12+60−120−60−60
13−120+70−50−50
14+80−120−40−40
15−110+80−30−30
16+90−100−10−10
17−110+90−20−20
18+100−110−10−10
19−120−120−240−128

Now let’s implement this in C++. We’ll start small, and just combine two samples.

Note: we will use qint types here, but qint16 will be the same as int16_t and short on most systems, and similarly qint32 will correspond to int32_t and int.

qint16 combineSamples(qint32 samp1, qint32 samp2)
{
    const auto sum = samp1 + samp2;

    if (std::numeric_limits<qint16>::max() < sum)
        return std::numeric_limits<qint16>::max();

    if (std::numeric_limits<qint16>::min() > sum)
        return std::numeric_limits<qint16>::min();

    return sum;
}

This is quite a simple implementation. We use a function combineSamples and pass in two 16-bit values, but they will be converted to 32-bit as arguments and summed. This sum is clamped to the limits of 16-bit integer representation using std::numeric_limits in the <limits> header of the standard library. We then return the sum, at which point it is re-converted to a 16-bit value.

Combining Samples for an Arbitrary Number of Audio Streams

Now consider an arbitrary number of audio streams n. For each sample position, we must sum the samples of all n streams.

Let’s assume we have some sort of audio stream type (we’ll implement it later), and a list called mStreams containing pointers to instances of this stream type. We need to implement a function that loops through mStreams and makes calls to our combineSamples function, accumulating a sum into a new buffer.

Assume each stream in mStreams has a member function read(char *, qint64). We can copy one sample into a char * by passing it to read, along with a qint64 representing the size of a sample (bitdepth). Remember that our bitdepth is 16-bit integer, so this size is just sizeof(qint16).

Using read on all the streams in mStreams and calling combineSamples to accumulate a sum might look something like this:

qint16 accumulatedSum = 0;

for (auto *stream : mStreams)
{
    // call stream->read(char *, qint64)
    // to read a sample from the stream into streamSample
    qint16 streamSample;
    stream->read(reinterpret_cast<char *>(&streamSample), sizeof(qint16)));

    // accumulate
    accumulatedSum = combineSamples(sample, accumulatedSum);
}

The first pass will add samples from the first stream to zero, effectively copying it to accumulatedSum. When we move to another stream, the samples from the second stream will be added to those copied values from the first stream. This continues, so the call to combineSamples for a third stream would combine the third stream’s sample with the sum of the first two. We continue to add directly into the buffer until we have combined all the streams.

Combining All Samples for a Buffer

Now let’s use this concept to add all the samples for a buffer. We’ll make a function that takes a buffer char *data and its size qint64 maxSize. We’ll write our accumulated samples into this buffer, reading all samples from the streams and adding them using the method above.

The function signature looks like this:

void readData(char *data, qint64 maxSize);

Let’s achieve more efficiency by using a constexpr variable for the bitdepth:

constexpr qint16 bitDepth = sizeof(qint16);

There’s no reason to call sizeof multiple times, especially considering sizeof(qint16) can be evaluated as a literal at compile-time.

With the size of each sample and the size of the buffer, we can get the total number of samples to write:

const qint16 numSamples = maxSize / bitDepth;

For each stream in mStreams we need to read each sample up to numSamples. As the sample index increments, a pointer to the buffer data needs to also be incremented, so we can write our results at the correct location in the buffer.

That looks like this:

void readData(char *data, qint64 maxSize)
{
    // start with 0 in the buffer
    memset(data, 0, maxSize);

    constexpr qint16 bitDepth = sizeof(qint16);
    const qint16 numSamples = maxSize / bitDepth;

    for (auto *stream : mStreams)
    {
        // this pointer will be incremented across the buffer
        auto *cursor = reinterpret_cast<qint16 *>(data);
        qint16 sample;

        for (int i = 0; i < numSamples; ++i, ++cursor)
            if (stream->read(reinterpret_cast<char *>(&sample), bitDepth))
                *cursor = combineSamples(sample, *cursor);
    }
}

The idea here is that we can start playing new audio sources by adding new streams to mStreams. If we add a second stream halfway through a first stream playing, the next buffer for the first stream will be combined with the first buffer of this new stream. When we’re done playing a stream, we just drop it from the list.

Next Steps

In Part 2, we'll use Qt Multimedia to fully implement our mixer, connect to our audio device, and test it on some audio files.

The post Implementing an Audio Mixer, Part 1 appeared first on KDAB.

Tuesday, 20 August 2024

A while ago a colleague of mine asked about our crash infrastructure in Plasma and whether I could give some overview on it. This seems very useful to others as well, I thought. Here I am, telling you all about it!

Our crash infrastructure is comprised of a number of different components.

  • KCrash: a KDE Framework performing crash interception and prepartion for handover to…
  • coredumpd: a systemd component performing process core collection and handover to…
  • DrKonqi: a GUI for crashes sending data to…
  • Sentry: a web service and UI for tracing and presenting crashes for developers

We will look at them in turn. This post introduces KCrash.

KCrash

KCrash, as the name suggests, is our KDE framework for crash handling. While it is a mid-tier framework and could be used by outside projects, it mostly doesn’t make sense to, because some behavior is very KDE-specific.

It installs POSIX signal handlers to intercept crash signals and then prepares the crashed process for handover to coredumpd and DrKonqi. More on these two in another post. Once prepared it sends the crash signal into the next higher level crash handler until the signal eventually reaches the default handler and cause the kernel to invoke the core pattern.

Before that can happen, a bunch of work needs doing inside KCrash. Most of it quite boring, but also somewhat challenging.

You see, when handling a signal you need to only use signal-safe functions. The manpage explains very well why. This proves quite challenging at the level we usually are at (i.e. Qt) because it is entirely unclear what is and isn’t ultimately signal-safe under the hood. Additionally, since we are dealing with crash scenarios, we must not trigger new memory allocation, because the heap management may have had an accident.

To that end, KCrash has to use fairy low-level API. To make that easier to work with, there are actually two parts to KCrash:

  • The Initialization Stage
  • The Crash Stage

The Initialization Stage

Initialization is generally triggered by calling KCrash::initialize. You may already wonder what kind of initialization KCrash could possibly need. Well, the obvious one is setting up the signal handling. But beyond that the init stage is also used to prepare us for the crash stage. I’ve already mentioned the serious constraints we will encounter once the signal hits, so we had best be prepared for that. In particular we’ll do as much of the work as possible during initialization. This most important includes copying QString content into pre-allocated char * instances such that we later only need to read existing memory. The second most important aspect is the metadata file preparation for use in…

The Crash Stage

Once initialization has happened, we are ready for crashes. Ideally the application doesn’t crash, of course. 😉

But if it does the biggest task is rescuing our data!

Metadata

Inside KCrash we have the concept of Metadata: everything we know about the crashed application: the signal, process ID, executable, used graphics device… and so on and so forth. All this data is collected into a metadata file on-disk in ~/.cache/kcrash-metadata at the time of crash.

Here’s an example file:

[KCrash]
exe=/usr/bin/kwin_wayland
glrenderer=
platform=wayland
appname=kwin_wayland
apppath=/usr/bin
signal=11
pid=1353
appversion=6.1.80
programname=KWin
bugaddress=submit@bugs.kde.org

The actual fields vary depending on what is available for any given application, but it’s generally more or less what is shown in the example.

This metadata file will later be consumed by DrKonqi in an effort to obtain information that only existed at runtime inside the application - such as the version that was running, or whether it was running in legacy X11 mode.

Handoff

Once the metadata is safely saved to disk, KCrash simply calls raise(). This re-raises the signal into the default handler, and through that causes a core dump.

What happens next is up to the system configuration as per the core manpage.

The recommended setup for distributions is that a crash handler be configured as core_pattern and that this handler consumes the crash. We recommend an implementation of the coredumpd and journald interfaces as that will then allow our crash handler to come in and log the crash with KDE.

So that was KCrash, the first in our four-step crash-handling pipeline. In the next blog post I’ll tell you all about the next one: coredumpd.

Konqi chasing after bugs

Sunday, 18 August 2024

A new Craft cache has just been published. The update is already available for KDE's CD, CI will follow in the next hours or days.

Please note that this only applies to the Qt6 cache. The Qt5 cache is in LTS mode since April 2024 and does not recieve major updates anymore.

Changes (highlights)

  • Qt 6.7.2
  • FFmpeg 7.0.1
  • llvm 18.1.8
  • boost 1.86.0
  • OpenSSL 3.3.1 (for Android too, which was still on 1.1.1v until recently)
  • CMake 3.30.0
  • Ninja 1.12.1
  • Removed qt-installer-framework (Windows)

About KDE Craft

KDE Craft is an open source meta-build system and package manager. It manages dependencies and builds libraries and applications from source on Windows, macOS, Linux, FreeBSD and Android.

Learn more on https://community.kde.org/Craft or join the Matrix room #kde-craft:kde.org

Saturday, 17 August 2024

It's been more than three weeks since the midterm summary, and the project is now nearing completion.

Currently, all the original features of Blinken have been fully implemented in the QML version. The remaining tasks involve UI adjustments, testing, and fixing potential bugs.

Over the past few weeks, I’ve been working on the following:

Integrating Blinken's Logic

The game logic of Blinken is handled by the BlinkenGame class from the original Blinken. The original code design is quite good, with most of the game logic encapsulated in this one class. The separation between the logic and the UI rendering is well done, so all I needed to do was connect the signals from this class in QML.

As for the audio playback in Blinken, the original code used the Phonon library, which is also open-source but does not support Android. Therefore, I replaced it with the QtMultimedia library, which provides cross-platform audio playback functionality.

Android Build for KF6 Applications

Some features of Blinken rely on libraries provided by the KF6 framework, such as KF6I18n and KF6Config. When cross-compiling to the Android platform, it's necessary to use the aarch64-linux versions of these libraries. If these libraries are not available on your system, you will encounter the following errors during compilation:

ld.lld: error: /usr/lib64/libKF6XmlGui.so.6.4.0 is incompatible with aarch64linux ld.lld: error: /usr/lib64/libKF6ConfigWidgets.so.6.4.0 is incompatible with aarch64linux 
……

However, the package manager on my Fedora distribution does not provide these versions. Compiling and installing them one by one from source is too cumbersome. On the advice of the community, I used Craft to handle cross-compilation.

For reference, here is the Craft tutorial: Craft - KDE Community Wiki

Important note: If you encounter installation failures, make sure to clear all contents under craft-kde-android before trying again, as leftover files may cause the installation to fail.

When installing, choose the Arm64 target architecture. If there are remnants from a previous failed installation, it may prevent the option to select the ARM64 architecture.

If you encounter issues like "Permission denied," you’ll need to disable SELinux:

sudo apt-get install selinux-utils 
sudo setenforce 0

Additionally, note that in the virtual machine invent-registry.kde.org/sysadmin/ci-images/android-qt67 provided by the community, the Java version is outdated, preventing the use of Gradle 8.6. You can either manually update the Java version in the docker or use an older version of Gradle.

To use Craft for building applications, you need to write a script called a Blueprint, which describes the libraries your application depends on. These scripts are relatively easy to write, and you can quickly get started by following the community documentation: Craft/Blueprints - KDE Community Wiki.

Using KF6 Framework Libraries

Some of the libraries originally used by Blinken are compatible with the Android platform, while others are not. By referring to the API Documentation, you can check which libraries are supported on Android. In Blinken, the following libraries are Android-compatible:

  • CoreAddons
  • GuiAddons
  • I18n
  • XmlGui

I needed to use these libraries in the QML version of Blinken.

The KF6 framework provides a convenient internationalization API, and the usage in QML is almost the same as in QWidget, which allowed me to directly reuse Blinken’s original multi-language support, saving a lot of time.

KConfig is used in Blinken to store high score information and settings. For the high scores, I needed to extract the HighScoreManager class from the original HighScoreDialog file, make some modifications, and then create a new high score interface in QML that connects the signals and slots of HighScoreManager. For the settings functionality, it’s as simple as registering a KConfig singleton in QML :

`qmlRegisterSingletonInstance<blinkenSettings>("org.kde.blinken", 1, 0, "BlinkenSettings", blinkenSettings::self());

KF6XmlGui was used in the original Blinken to create the About Blinken Page, About KDE Page, and Handbook Page. Although this library is Android-compatible, it is based on QWidget, while the main interface of Blinken is built with QML. Bringing in QWidget just for these pages didn't seem like a good idea. Luckily, for the Android platform, kirigami-addons provides this functionality. By incorporating it, I also brought in Kirigami, which helps optimize the UI.

After adding new dependencies, it’s important to modify the .kde-ci.yml file to support CI/CD. For more information: Infrastructure/Continuous Integration System - KDE Community Wiki.