Skip to content

Welcome to Planet KDE

This is a feed aggregator that collects what the contributors to the KDE community are writing on their respective blogs, in different languages

Monday, 22 July 2024

Photo by Aron Visuals on Unsplash

What time is it? It’s 9:30 in the morning.

But what does that mean? This isn’t true everywhere in the world, and who decided a day has 24 hours? What is a second? What is time?

What is time, really?

Time, as we understand it, is the ongoing sequence of existence and events that occur in a seemingly irreversible flow from the past, through the present, and into the future.

General relativity reveals that the perception of time can vary for different observers: the concept of what time it is now holds significance only relative to a specific observer. On your wristwatch, time advances at a consistent rate of one second per second. However, if you observe a distant clock, its rate will be influenced by both velocity and gravity: a clock moving faster than you will appear to tick more slowly.

Ignoring relativistic effects for a moment, physics provides a practical definition of time: time is “what a clock reads”. Observing a certain number of repetitions of a standard cyclical event constitutes one standard unit, such as the second.

Time units

Even in ancient times, two notable physical phenomena could be observed cyclically and helped quantify the passage of time:

  • The Earth’s orbit around the Sun results in evident, cyclical events, and defines the “year” as a unit of time.
  • The Sun crossing the local meridian, i.e. the time it takes for the Sun to reach the same angle in the sky. The interval between two successive instances of this event identify the solar “day”.

Note that the solar day, as a time unit, is affected by both the Earth’s own rotation and its revolution around the Sun: since the Earth moves by roughly 1° around the Sun each day, the Earth has to rotate by roughly 361° for the Sun to cross the local meridian.

This is different from the “sidereal day”, which is the amount of time it takes for the Earth to rotate by 360°, since it is computed from the apparent motion of distant astronomical objects instead of the Sun.

However, the solar day is typically the unit of interest for us humans and our daily lives.

How many days in a year?

Although the Earth doesn’t rotate a discrete number of times in its year, and hence the latter cannot be subdivided into a discrete number of days, humans have devised numerous calendar systems to organize days for social, religious, and administrative purposes.

Pope Gregory XIII implemented the Gregorian calendar in 1582, which has become the predominant calendar system in use today. This calendar supplanted the Julian calendar, rectifying inaccuracies by incorporating leap years, which are additional days inserted to synchronize the calendar year with the solar year. Over a complete leap cycle spanning 400 years, the average duration of a Gregorian calendar year is 365.2425 days. While the majority of years consist of 365 days, 97 out of every 400 years are designated as leap years, extending to 366 days.

Despite the Gregorian calendar being the de-facto standard, other calendars exist:

  • Islamic Calendar: A lunar calendar consisting of 12 months in a year of 354 or 355 days.
  • Hebrew Calendar: A lunisolar calendar used by Jewish communities.
  • Chinese Calendar: A lunisolar calendar that determines traditional festivals and holidays in Chinese culture.

How many seconds in a day?

We all know that there are 60 seconds in a minute, 60 minutes in an hour, and 24 hours in a day, right? As it can be expected, these magic numbers aren’t a fundamental property of nature, but rather a result of two historical choices:

  • Dividing the day into 24 hours: This stems from ancient civilizations where early cultures divided the cycle of day and night into a convenient number of units, likely influenced by the phases of the moon or other natural patterns. Ancient egyptians used sundial subdivisions and split the cycle into two halves of 12 hours each: they used a duodecimal system already, likely influenced by the number of joints in our hands, excluding thumbs, and the ease of counting them. Twenty-four then became a common choice for counting hours.
  • Subdividing hours into 60 minutes and minutes into 60 seconds: This preference for base-60 systems comes from even earlier civilizations like the Sumerians and Babylonians. Their number systems were based on 60, again possibly due to the ease of counting on fingers and knuckles. This base-60 system was later adopted by the Greeks and eventually influenced our timekeeping.

Historically, dividing the natural day from sunrise to sunset into twelve hours, as the romans also did, meant that the hour had a variable measure that depended on season and latitude.

Equal hours were later standardized as 1/24 of the mean solar day.

Counting time

Historically, the concept of a second relied on the Earth’s rotation, but variability of the latter made this definition inadequate for scientific accuracy. In 1967, the second was redefined using the consistent properties of the caesium-133 atom. Presently, a second is defined as the time it takes for 9,192,631,770 oscillations of radiation during the transition between two energy states in caesium-133.

TAI and atomic clocks

Atomic clocks utilize the definition of a second for precise time measurement.

An array of atomic clocks globally synchronizes time through consensus: the clocks collectively determine the accurate time, with each clock adjusting to align with the consensus, known as International Atomic Time (TAI).

The International Bureau of Weights and Measures (BIPM) located near Paris, France, computes the International Atomic Time (TAI) by taking the weighted average of more than 300 atomic clocks from around the world, where the most stable clocks receive more weight in the calculation.

Operating on atomic seconds, TAI serves as the standard for calibrating various timekeeping instruments, including quartz and GPS clocks.

GPS time

Specifically, GPS satellites carry multiple atomic clocks for redundancy and better synchronization, but the accuracy of GPS timing requires different corrections, such as for both Special and General Relativity:

  • According to Special Relativity, clocks moving at high speeds will run slower relative to those on the ground. GPS satellites travel at approximately 14,000 km/h, which results in their clocks ticking slower by about 7.2 microseconds per day.
  • According to General Relativity, clocks in a weaker gravitational field run faster than those closer to a massive object. The altitude of GPS satellites (about 20,200 km) causes their clocks to tick faster by about 45.8 microseconds per day.
  • As the net effect, the combined relativistic effects cause the satellite clocks to run faster than ground-based clocks by about 38.6 microseconds per day.

To compensate, the satellite clocks are pre-adjusted on Earth before launch to tick slower than the ground clocks.

Once in orbit, GPS satellites continuously broadcast signals containing the exact transmission time and their positions.

A GPS receiver on the ground picks up these signals from multiple satellites and compares the reception time with the transmission time to calculate the time delays.

Using these delays, the receiver determines its distance from each satellite and, through trilateration, calculates its own position and the current time.

Precise timekeeping on GPS satellites is thus necessary, since small errors of 38.6 microseconds would result in big positional errors of roughly 11.4 km accumulated each day.

Notably, corrections must also consider that the elliptical satellite orbits cause variations in time dilation and gravitational frequency shifts over time. This eccentricity affects the clock rate difference between the satellite and the receiver, increasing or decreasing it based on the satellite’s altitude.

UTC and leap seconds

While TAI provides precise seconds, the Earth presents an irregular rotation: it gradually slows due to tidal acceleration.

On the other hand, for practical purposes, civil time is defined to agree with the Earth’s rotation.

Coordinated Universal Time (UTC) serves as the international standard for timekeeping. This time scale uses the same atomic seconds as TAI but adjusts for variations in the Earth’s rotation by adding or omitting leap seconds as necessary.

To determine if a leap second is required, universal time UT1 is utilized to measure mean solar time by monitoring the Earth’s rotation relative to the sun.

The variance between UT1 and UTC dictates the necessity of a leap second in UTC.

UTC has thus precise seconds and it is always kept within 0.9 seconds of the UT1 to synchronize with astronomical time: an additional second may be added to the last minute of a UTC year or June to compensate.

This is why the second is now precisely defined in the International System of Units (SI) as the atomic second, while days, hours, and minutes are only mentioned in the SI Brochure as accepted units for explanatory purposes: their use is deeply embedded in history and culture, but their definition is not precise as it refers to the mean solar day and it cannot account for the inconsistent rotational and orbital motion of the Earth.

Local time

Although UTC provides a universal time standard, individuals require a “local” time that is consistent to the position of the sun, ensuring that local noon (when the sun reaches its zenith) roughly aligns with 12:00 PM.

In a specific region, coordinating daily activities such as work and travel is more convenient when people adhere to a shared local time. For instance, if I start traveling to another nearby city at 9:00 in the morning, I can anticipate that it will be 11:00 AM after two hours of travel.

On the other hand, if I schedule a flight to a destination far away and the airline informs me that the arrival time is 11:00 AM local time, I can generally infer that the sun will be high in the sky.

Time zones

Time zones are designed to delineate geographical regions within which a convenient and consistent standard time is utilized.

A time zone is a set of rules that define a local time relative to the standard incremental time, i.e. UTC.

Imagine segmenting the Earth into 24 sections, each approximately 15 degrees of longitude apart. This process would roughly define geographical areas that differ by an hour, which is a time range small enough so that people living within each area can agree on a local time.

For this purpose, the Coordinated Universal Time (UTC), located at zero degrees longitude (the Prime Meridian), serves as the reference point. Each country or political entity declares its preferred time zone identified by its deviation from UTC, which spans from UTC−12:00 to UTC+14:00. Typically, these offsets are whole-hour increments, though certain zones like India and Nepal deviate by an additional 30 or 45 minutes.

This system ensures that different regions around the globe set their clocks according to their respective time zones.

There are actually more time zones than countries, even though time zones generally correspond to hourly divisions of the Earth’s geography.

At the boundaries of UTC offsets, the International Date Line (IDL) roughly follows the 180° meridian but zigzags to avoid splitting countries or territories, creating a sharp time difference where one side can be a whole day ahead of the other: i.e. when jumping from UTC-12 to UTC+12.

Some regions have chosen offsets that extend beyond the conventional range for economic or practical reasons. For example, Kiribati adjusted its time zones to ensure that all its islands are on the same calendar day. As a result, some areas of Kiribati observe UTC+14, hence why UTC offset range extends from -12 to +14.

Daylight Saving Time

Additionally, some nations adjust their clocks forward by one hour during warmer months. This practice, known as Daylight Saving Time (DST), aims to synchronize business hours with daylight hours, potentially boosting economic activity by extending daylight for commercial endeavors. The evening daylight extension also holds the promise of conserving energy by reducing reliance on artificial lighting. Typically, clocks are set forward in the spring and set back in the fall.

DST is less prevalent near the equator due to minimal variation in sunrise and sunset times, as well as in high latitudes, where a one-hour shift could lead to dramatic changes in daylight patterns. Consequently, certain countries maintain the same local time year-round.

Nigeria always observes West Africa Time (WAT), that is UTC+1. Japan observes Japan Standard Time (JST), that is UTC+9.

Instead, other countries observe DST during the summer by following a different UTC offset. That’s why Germany and other countries in Europe observe both Central European Time (CET) UTC+1 and Central European Summer Time (CEST) UTC+2 during the year, and that’s why CEST has such a name.

Interestingly, while the United Kingdom follows Greenwich Mean Time (GMT) UTC+0 in winter and British Summer Time (BST) UTC+1 in summer, Iceland uses GMT all year.

Time zone databases

Conceptually, a time zone is a region of the Earth that shares the same standard time, which can include both a standard time and a daylight saving time; these correspond to local time offsets with respect to UTC.

These conventions and rules change over time, hence why time zone databases exist and are regularly updated to reflect historical changes.

A standard reference is the IANA time zone database, also known as tz database or tzdata, which maintains a list of the existing time zones and their rules.

On Linux, macOS and Unix systems, you can navigate the timezone database at the standard path `/usr/share/zoneinfo/`.

Here’s a breakdown of the essential elements for navigating time zone databases:

  • The “time zone identifier” is typically a unique label in the form of Area/Location, e.g. America/New_York.
  • “Time zone abbreviations” are short forms used to denote specific time zones, often indicating whether it is standard or daylight saving time. For example, CET (Central European Time) that corresponds to UTC+01:00 and CEST (Central European Summer Time) that corresponds to UTC+02:00.
  • “Standard times” are the official local times observed in a region when daylight saving time is not in effect. For example, Central European Time (CET, UTC+01:00) is the standard time for many European countries during the winter months.

Time formats

For all practical purposes, UTC serves as the foundation for tracking the passage of time and as the reference for defining a local time.

However, all would be futile without a standardized representation of time that allows for consistent and accurate recording, processing, and exchange of time-related data across different systems.

Time formats are crucial for ensuring that time data is interpretable and usable by both humans and machines.

Some common formats have become the de-facto standards for interoperable time data.

ISO 8601, RFC 3339

ISO 8601 is a globally recognized standard for representing dates, times, and durations. It offers great versatility, accommodating a variety of date and time formats.

ISO 8601 strings represent date and time in a human-readable format with optional timezone information. It uses the Gregorian calendar, even for dates that precede its genesis.

Dates use the format `YYYY-MM-DD`, such as `2024–05–28`. Times use the format `HH:MM:SS`, such as `14:00:00`. Combined date and time can be formatted as `YYYY-MM-DDTHH:MM:SS`, for example `2024–05–28T14:00:00`. ISO 8601 strings can include timezone offsets, for example `2024–05–28T14:00:00+02:00`. `Z` is used to indicate Zulu time, the military name for UTC, for example `2024–05–28T14:00:00Z`.

ISO 8601 is further refined by RFC 3339 for use in internet protocols.

RFC 3339 is a profile of ISO 8601, meaning it adheres to the standard but imposes additional constraints to ensure consistency and avoid ambiguities:

  • Always specifies the time zone, either as Z for UTC or an offset. Example: `2024–05–28T14:00:00–05:00`.
  • Supports fractional seconds, such as in `2024–05–28T14:00:00.123Z`.
  • Fractional seconds, if used, must be preceded by a decimal point and can have any number of digits.
  • Does not allow the use of the truncated format of ISO 8601, such as `2024–05` for May 2024.

Both standards are not limited to representing only UTC times, as they can also express times that occurred before the introduction of UTC. Practically, they are most often used to represent UTC timestamps.

Since leap seconds are added to UTC to account for irregularities in the Earth’s rotation, ISO 8601 allows for the representation of the 60th second in a minute, which is the leap second.

For example, the latest leap second occurred at the end of 2016, so `2016–12–31T23:59:60+00:00` is a valid ISO 8601 time stamp.

Unix time

Unix time, or POSIX time, is a method of keeping track of time by counting the number of non-leap seconds that have passed since the Unix epoch. The Unix epoch is set at 00:00:00 UTC on January 1, 1970.

The Unix time format is thus a signed integer counting the number of non-leap seconds since the epoch.

For example, the Unix timestamp `1653484800` represents a specific second in time, that is 00:00:00 UTC on May 25, 2022.

For dates and times before the epoch, Unix time is represented as negative integers.

Unix time can also represent time with higher precision using fractional seconds (e.g., Unix timestamps with milliseconds, microseconds, or nanoseconds).

Note that Unix time is timezone-agnostic.

In addition, Unix time does not account for leap seconds, so it does not have a straightforward way to represent the 60th second (leap second) of a minute.

According to the POSIX.1 standard, Unix time handles a leap second by repeating the previous second, meaning that time appears to “stand still” for one second when fractional seconds are not considered. However, some systems, such as Google’s, use a technique called “leap smear.” Instead of inserting a leap second abruptly, they gradually adjust the system clock over a longer period, such as 24 hours, to distribute the one-second difference smoothly. As a result, certain Unix timestamps can be ambiguous: 1483142400 might refer to either the start of the leap second (2016–12–31 23:59:60) or one second later at the end of it (2017–01–01 00:00:00).

Note that every day in Unix time consists of exactly 86400 seconds. On the other hand, when leap seconds occur, the difference between two Unix times is not equal to the duration in seconds of the period between the two. Most applications however don’t require this level of accuracy, so Unix times are handy to compare and manipulate with basic additions and subtractions.

Clocks and synchronization

From atomic clocks to individual computers, time is synchronized through a hierarchical and redundant system using GPS, NTP, and other methods to ensure accuracy.

GPS satellites carry atomic clocks and broadcast time signals. Receivers on Earth can use these signals to determine precise time.

In Network Time Protocol (NTP), servers synchronize with atomic clocks via GPS or other means, and form a hierarchical distribution network of time information.

Radio stations, such as WWV/WWVH, can also broadcast time signals that can be picked up by radio receivers and used for synchronization.

Hardware clocks

Computers have hardware “clocks”, or “timers”, that synchronize all components and signals on the motherboard and are also used to track time.

Typically, modern clocks are generated by quartz crystal oscillators of about 20MHz: an oscillator on the motherboard ticks the “system clock”, and other clocks are generated from it by multiplying or dividing the frequency of the first oscillator thanks to phase-locked loops.

For example, the CPU clock is generated from the system clock and has the same purpose, but is only used on the CPU itself.

Since the clock determines the speed at which instructions are executed and the CPU needs to perform more operations per time than the motherboard, the CPU clock yields higher clock signals: e.g. 4GHz for a CPU core.

Some computers also allow the user to change the multipliers in order to “overclock” or “underclock” the CPU. Generally, this is used to speed up the processing capabilities of the computer, at the expense of increased power consumption, heat, and unreliability.

OS clock

Backed by hardware clocks, the operating system manages different software counters, still called “clocks”, such as the “system clock” which is used to track time.

The OS configures the hardware timers to generate periodic interrupts, and each interrupt allows the OS to increment a counter that represents the system uptime.

The actual time is derived from the uptime counter, typically starting from a predefined epoch (e.g., January 1, 1970, for Unix-based systems).

Real Time Clock

Modern computers have a separate hardware component dedicated to keeping track of time even when the computer is powered off: the Real-Time Clock (RTC).

The RTC is usually implemented as a small integrated circuit on the motherboard, powered by a small battery such as a coin-cell battery. It has its own low-power oscillator, separate from the main system clock.

The operating system reads the RTC during boot through standard communication protocols like I2C or SPI, and initializes the system time.

The OS periodically writes the system time back to the RTC to account for any adjustments (e.g., NTP updates).

Although a system clock is more precise for short-term operations due to high-frequency ticks, an RTC is sufficient for keeping the current date and time over long periods.

NTP

All hardware clocks drift due to temperature changes, aging hardware, and other factors.

This results in inaccurate timekeeping for the operating system, so regular synchronization of system time with NTP servers is needed to correct for this drift.

NTP adjusts the system clock of the OS in one of two ways:

  • By “stepping”, abruptly changing the time.
  • By “slewing”, slowly adjusting the clock speed so that it drifts toward the correct time: hardware timers continue to produce interrupts at their usual rate while the OS alters the rate at which it counts hardware interrupts.

Slewing should be preferred, but it’s only applicable for correcting small time inaccuracies.

NTP servers also provide leap second information, and systems need to be configured to handle these adjustments correctly, often by “smearing” the leap second over a long period, e.g. 24 hours.

Different OS clocks

The operating system generally keeps several time counters for use in various scenarios.

In Linux, for instance, `CLOCK_REALTIME` represents the current wall clock or time-of-day. However, other counters like `CLOCK_MONOTONIC` and `CLOCK_BOOTTIME` serve different purposes:

  • `CLOCK_MONOTONIC` remains unaffected by discontinuous changes in the system time (such as adjustments by NTP or manual changes by the system administrator) and excludes leap seconds.
  • `CLOCK_BOOTTIME` is similar to `CLOCK_MONOTONIC` but also accounts for any time the system spends in suspension.

For example, if you need to calculate the elapsed time between two events on a single machine without a reboot occurring in between, `CLOCK_MONOTONIC` is a suitable choice. This is commonly used for measuring time intervals, such as in code performance benchmarking.

Best practices

Storing universal time

For many applications, such as in log lines, time zones and Daylight Saving Time (DST) rules are not directly relevant when storing a timestamp: what matters is the exact moment in (universal) time when the event occurred, i.e. the relative ordering of events.

When storing timestamps, it is crucial to store them in a consistent and unambiguous manner.

Stored timestamps, as well as time calculations, should then be in UTC: i.e. without UTC offset. This avoids errors due to local time changes such as DST.

Timestamps should be converted to local time zones only for display purposes.

As per formatting, Unix timestamps are the preferred time format since they are simple numbers, are space-efficient, and are easy to compare and manipulate directly.

UTC timestamp strings as defined in RFC 3339 are useful when displaying time in a human-readable format.

Time zones and DST become relevant when converting, displaying, or processing timestamps in user-facing applications, especially when dealing with future events.

Here are some considerations for handling future events:

  • Store the base timestamp of the event in Unix time. This ensures that the event’s reference time is unambiguous and consistent.
  • Convert the stored UTC timestamp to the local time zone when displaying the event to users. Ensure that the conversion respects the UTC offset and DST rules of the target time zone.
  • Keep in mind that time zone rules change from time to time, for instance when a country stops observing DST, and the UTC timestamp would then be represented by a different future local time.

Time zone databases, like the IANA time zone database, are regularly updated to reflect historical changes. Applications must use the latest versions of these databases to ensure accurate timestamp conversions, correctly interpreting and displaying timestamps.

By all means, use established libraries or your language’s facilities to do timezone conversions and formatting. Do not implement these rules yourself.

Storing local time

Time zone databases must be updated because time zone rules and DST schedules can and do change. Governments and regulatory bodies may adjust time zone boundaries, introduce or abolish DST, or change the start and end dates of DST.

This is especially relevant if the application user is more interested in preserving the local time information of a scheduled event than its UTC time.

Indeed, while storing UTC time can help you track when the next solar eclipse will happen, imagine scheduling a conference call for 1 PM on some day next year and storing that time as a UTC timestamp. If the government changes or abolishes its DST then the timestamp still refers to the instant in time that was formerly going to be 1 PM at that date, but converting it back to local time would yield a different result.

For these situations storing UTC time is not ideal. Instead, calendar apps might store the event’s time with a day/date, a time-of-day, and a timezone, as three separate values; the UTC time can be derived if needed, given also the timezone rules that currently apply when deriving it.

However, it should be considered that converting local time to UTC time is not always unambiguous: thanks to DST some local timestamp might happen twice, e.g. when setting the clock backward, and some local timestamp might never happen, e.g. when setting the clock forward and skipping an hour.

Ideally, invalid or ambiguous times should not be stored in the first place, and stored times should be validated when new timezone rules are introduced.

Additional challenges arise with:

  • Recurrent events, where changes to timezone rules can alter offsets for some instances while leaving others unaffected.
  • Events where users come from multiple timezones, necessitating a single coordinating time zone for the recurrence. However, offsets must be recalculated for each involved time zone and every occurrence.

For exploring the topic of recurrent events and calendars further, RFC 5545 might be of interest since it presents the iCalendar data format for representing and exchanging calendaring and scheduling information.

For instance, it also describes a grammar for specifying “recurrence rules” where “the last work day of the month” could be represented as “FREQ=MONTHLY;BYDAY=MO,TU,WE,TH,FR;BYSETPOS=-1”, even though recurrence rules may generate recurrence instances with an invalid date or nonexistent local time, and that must still be accounted for.

Parsing and formatting time

Different time formats serve various purposes in timekeeping, incorporating concepts like date and time, universal time, calendar systems, and time zones. The specific format needed depends on the use case.

For example, using the naming convention from the W3C guidelines, “incremental time” like Unix time measures time in fixed integer units that increase from a specific point. In contrast, “floating time” refers to a date or time value in a calendar that is not tied to a specific instant. Applications use floating time values when the local wall time is more relevant than a precise timeline position. These values remain constant regardless of the user’s time zone.

Floating times are serialized in ISO8601 formats without local time offsets or time zone identifiers (like Z for UTC).

However, many programming libraries can make it too easy to deserialize an ISO8601 string into an incremental time (e.g. Unix time) aligned with UTC, leading to errors by implicitly assuming timezone information.

Programmers should carefully consider the intended use of time information to collect, parse, and format it correctly. Here are some guidelines for handling time inputs:

  • Use UTC or a consistent time zone when creating time-based values to facilitate comparisons across sources.
  • Allow users to choose a time zone and associate it with their session or profile.
  • Use exemplar cities to help users identify the correct time zone.
  • Consider the country as a hint, since most have a single time zone.
  • For data with local time offsets, adjust the zone offset to UTC before performing any date/time sensitive operations.
  • For time data not related to time zones, like birthdays, use a representation that does not imply a time zone or local time offset.
  • Since user input can be messy, avoid strict pattern matching and use a “lenient parse” approach: https://unicode.org/reports/tr35/tr35-10.html#Date_Format_Patterns
  • Be mindful of how libraries parse user input, for instance using the correct calendar: https://ericasadun.com/2018/12/25/iso-8601-yyyy-yyyy-and-why-your-year-may-be-wrong/

Distributed systems

Timekeeping and clocks are hardly reliable and consistent.

In theory, we have the means to agree on the notion of what time it is now: we know how to synchronize computer systems worldwide.

The real problem is not that information requires time to be transferred between systems either, instead it’s the fact that everything can break.

What makes it hard to build distributed systems that agree on a specific value is the combination of them being asynchronous and having physical components subject to failures.

Certain results provide insights into the challenges of achieving specific properties in distributed systems under particular conditions:

  • The “FLP result,” a 1985 paper, establishes that in asynchronous systems, no distributed algorithm can reliably solve the consensus problem. This is because, given an unbounded amount of time for processing, it’s impossible to determine if a system has crashed or is merely experiencing delays.
  • The “CAP theorem” explains that any distributed system can achieve at most two of the following three properties: Consistency (ensuring all data is up to date), Availability (ensuring the system is accessible), and Partition Tolerance (ensuring the system continues to function despite network partitions). In unreliable systems, it’s impossible to guarantee both consistency and availability simultaneously.
  • The “PACELC theorem,” introduced in 2010, extends the CAP theorem by highlighting that even in the absence of network partitions, distributed systems must make a trade-off between latency and consistency.

The net outcome is that in theory having a single absolute value of time is not meaningful in distributed systems. In practice, we can certainly plan for reducing the margin of error and de-risking our applications.

Here are some further considerations that can be relevant:

  • Ensure all nodes in the distributed system synchronize their clocks using NTP or similar protocols to minimize time drift between nodes. Use multiple NTP servers for redundancy and reliability.
  • Ensure that NTP servers and clients are configured to handle leap seconds correctly.
  • Consider also using GPS to provide an additional layer of accuracy.
  • Besides NTP, evaluate Precision Time Protocol (PTP) for applications that require nanosecond-level synchronization.
  • Account for network latency, for example by estimating round-trip times (RTTs), to avoid discrepancies caused by delays in message delivery.
  • Ensure timestamps have sufficient precision for the application’s requirements. Millisecond or microsecond precision may be necessary for high-frequency events.
  • Ensure that all nodes log events using UTC to maintain consistency in distributed logs.
  • Use distributed databases that handle time synchronization internally, such as Google Spanner, which uses TrueTime to provide globally consistent timestamps. This is mostly relevant for applications that require precise transaction ordering, such as financial systems or inventory management.
  • Consider using logical clocks (e.g., Lamport Timestamps or Vector Clocks) to order events in a causally consistent manner across distributed nodes, independent of physical time.
  • Consider Hybrid Logical Clocks (HLC) to provide a more reliable timestamp mechanism that combines physical and logical clocks.
  • Evaluate the use of Conflict-free Replicated Data Types (CRDT) to avoid the need of time or coordination for performing data updates.

Fun facts

  • UTC was formerly known as Greenwich Mean Time (GMT) because the Prime Meridian was arbitrarily designated to pass through the Royal Observatory in Greenwich.
  • The acronym UTC is a compromise between the English “Coordinated Universal Time” and the French “Temps Universel Coordonné.” The international advisory group selected UTC to avoid favoring any specific language. In contrast, TAI stands for the French “Temps Atomique International.”
  • Leap seconds need not be announced more than six months in advance, posing a challenge for second-accurate planning beyond this period.
  • NTP servers can use the leapfile directive in ntpd to announce an upcoming leap second. Most end-user installations do not implement this, relying instead on their upstream servers to handle it correctly.
  • Leap seconds can theoretically be both added and subtracted, although no leap second has ever been subtracted yet.
  • The 32-bit representation of Unix time will overflow on January 19, 2038, known as the Year 2038 problem. Transitioning to a 64-bit representation is necessary to extend the date range.
  • The year 2000 problem: https://en.wikipedia.org/wiki/Year_2000_problem
  • Atomic clocks are extremely precise. The primary time standard in the United States, the National Institute of Standards and Technology (NIST)’s caesium fountain clock, NIST-F2, has a time measurement uncertainty of only 1 second over 300 million years.
  • Leap seconds will be phased out in 2035, preferring a slowly drift of UTC from astronomical time over the complexities of managing leap seconds.
  • In 1998 the Swatch corporation introduced the “.beat time” for their watches, a decimal universal time system without time zones: https://en.wikipedia.org/wiki/Swatch_Internet_Time
  • February 30 has been a real date at least twice in history: https://www.timeanddate.com/date/february-30.html
  • Year 0 does not exist: https://en.wikipedia.org/wiki/Year_zero

Further reads

What was discussed here it’s just the tip of the iceberg to get a brief overview of the topic.

Some further resources might be interesting to deepen our understanding of timekeeping, its challenges, and edge cases:

Sunday, 21 July 2024

Make sure you commit anything you want to end up in the KDE Gear 24.08
releases to them

Next Dates  
  • July 25, 2024: 24.08 Freeze and Beta (24.07.80) tag & release
  • August  8, 2024: 24.08 RC (24.07.90) Tagging and Release
  • August 15, 2024: 24.08 Tagging
  • August 22, 2024: 24.08 Release

https://community.kde.org/Schedules/KDE_Gear_24.08_Schedule

Saturday, 20 July 2024

Yes, that’s right. The title just goes perfectly, with these long weeks! From Week 3 to Week 7! So, first a small back story on the title. On 25th June, my semester exam ended. I was returning to my hometown, and on my way, I got into a bike accident. My right hand got bruised onto the road. Luckily, it didn’t break. But, the sheer pain was enough to make me cry.

These past two weeks were big for Wayland accessibility support, as Nicolas Fella did a lot of work to improve support for sticky keys to equal the state they were in on X11. This work is not complete, but so far it’s taken a big bite out of the open issues! The work lands in a mixture of Plasma 6.1.3 and 6.2.0 (link 1, link 2, link 3, link 4, link 5, link 6).

Beyond this, it’s notable that Plasma developers fixed the five most common Plasma crashes, as well as a bunch of less common ones, which you’ll see mentioned below. These were caused by a mix of Qt regressions and our own code defects. The new automatic crash reporting system has been a huge boon here, allowing us to see which crashes are actually affecting people most. So please do continue to report them!

And of course there’s lots more too. Check it out:

New Features

Elisa now offers a feature to add a track, album, etc. directly to the playlist after current song, playing it next (Jack Hill, Elisa 24.08.0. Link):

System Settings’ Drawing Tablet page now has a calibration tool (Joshua Goins, Plasma 6.2.0. Link)

The default width of Icons-and-Text Task Manager items is now user- configurable, and should also exhibit slightly smarter shrinking behavior as space gets filled up (Kisaragi Hiu, Plasma 6.2.0. Link)

Added support for Plasma’s charging threshold feature to OpenBSD (Rafael Sadowski, Plasma 6.2.0. Link)

UI Improvements

Dolphin now features a lovely super premium user experience for installing Filelight if it’s not already installed (Felix Ernst, Dolphin 24.08.0. Link):

Filelight now has a more illuminating and welcoming homepage (me: Nate Graham, Filelight 24.08.0. Link):

Spectacle has now adopted the “inline messages touch the header” paradigm (me: Nate Graham, Spectacle 24.08.0. Link):

System Settings’ Night Light page now accepts creative custom times, no longer internally clamping them to a set of acceptable times (Vlad Zahorodnii, Plasma 6.1.3. Link)

Improved the smoothness of resizing Plasma widgets (Vlad Zahorodnii, Plasma 6.1.4. Link)

Made it impossible to remove administrator privileges from your current user unless there’s at least one other admin user on the system, to ensure that someone is an admin and can reverse the decision if needed! (Thomas Duckworth, Plasma 6.2.0. Link)

The “launch this app” shortcut for apps on System Settings’ Shortcuts page is now named “Launch”, making its purpose clear (me: Nate Graham, Plasma 6.2.0. Link)

Added some relevant clipboard-related keywords to System Settings’ General Behavior page, so you can find it in a search for things like “paste” and “selection” and stuff like that (Christoph Wolk, Plasma 6.2.0. Link)

Plasma’s Digital Clock widget now uses a typographic space to separate the time from the date when in single-row mode, so that it looks better especially when using a monospace font (Kisaragi Hiu, Plasma 6.2.0. Link)

Removed the filter for different job types from Plasma’s Printers widget, because it never worked (seriously) and apparently no one ever noticed because we didn’t even have any bug reports about it! (Mike Noe, Plasma 6.2.0. Link)

Bug Fixes

Filelight no longer fails to initiate a second scan after leaving the first one and going back to the Overview page (Harald Sitter, Filelight 24.08.0. Link)

Fixed one of the most common ways that Plasma could crash randomly, out of the blue (Akseli Lahtinen, Plasma 6.1.3. Link)

Fixed one of the most common ways that Plasma could crash on Wayland when a screen turns off (Marco Martin, Plasma 6.1.3. Link)

Fixed another one of the apparently many ways that Plasma can crash while handling various types of clipboard data, which appeared to be very common (David Edmundson, Plasma 6.1.3. Link)

Fixed a somewhat common way that Powerdevil could crash when waking the system from sleep with certain types of monitors (Jakob Petsovits, Plasma 6.1.3. Link)

Fixed an issue that caused Spectacle to crash after finishing a screen recording on systems using PipeWire 1.2.0 (Arjen Hiemstra, Plasma 6.1.3. Link)

Fixed an issue that caused Plasma on Wayland to crash when dragging a Task Manager task with no .desktop file associated with it (such as a Steam game) onto the desktop (Akseli Lahtinen, Plasma 6.1.3. Link)

Fixed a recent regression that caused System Tray icons for GTK2 apps to stop responding after the first time they’re clicked. Added an autotest to make sure this doesn’t regress again, too (David Edmundson and Fushan Wen, Plasma 6.1.3. Link)

It’s now possible to set your user avatar image to a file those full path contains “special characters” like spaces, ampersands, etc. (Daniil-Viktor Ratkin, Plasma 6.1.3. Link)

It’s now more reliable to change the date or time using System Settings on a distro not using Systemd (Fabio Bas, Plasma 6.1.3. Link)

Plasma’s RDP server now works properly after again a prior failed connection using a non-H.264-capable client app (Arjen Hiemstra, Plasma 6.1.3. Link)

The Fcitx input method’s “show input method info when switching input focus” setting is now compatible with Plasma’s zoom-out style edit mode, and no longer causes it to exit immediately when the input method popup is shown (Weng Xuetian, Plasma 6.1.3. Link)

Fixed a different somewhat common out-of-the-blue Plasma crash (Méven Car, Plasma 6.1.4. Link)

Fixed a case where KWin could crash on X11 when compositing gets toggled on or off (Vlad Zahorodnii, Plasma 6.1.4. Link)

Fixed a regression in Discover that caused various alerts and information items in certain apps’ description pages to not appear as expected (Aleix Pol Gonzalez, Plasma 6.1.4. Link)

Fixed a regression in Plasma 6.0 that caused the “remove this item” hover icons in KRunner’s history view to be invisible (Ivan Tkachenko, Plasma 6.1.4. Link)

OpenVPN VPNs requiring a challenge-response work again with NetworkManager 1.64 or later (Benjamin Robin, Plasma 6.2.0. Link)

“Text Only” System Monitor sensors on horizontal Plasma panels are once again correctly-sized with centered text (Arjen Hiemstra, Plasma 6.2.0. Link)

“Line Chart” System Monitor sensors now show their legends as expected when placed on a wide vertical Plasma panel (Arjen Hiemstra, Plasma 6.2.0. Link)

When using Plasma’s “Raise maximum volume” setting, it now applies to the per-app volume sliders in the System Tray popup as well as the device volume sliders (Roberto Chamorro, Plasma 6.2.0. Link)

Using KWin’s wide color gamut or ICC profile features no longer increases the transparency of already-transparent windows (Xaver Hugl, Plasma 6.2.0. Link)

Fixed the source of bizarre view corruption in Dolphin introduced in the recently-released Frameworks 6.4 (Vlad Zahorodnii, Frameworks 6.4.1. Link)

Fixed a case where Plasma could crash when trying to re-assign a shortcut for a widget to one already used by something else (Arjen Hiemstra, Frameworks 6.5. Link)

On the “Get New [thing]” dialogs, downloading one file from an entry that includes many now works from the details page (Akseli Lahtinen, Frameworks 6.5. Link)

Fixed another fairly common way that Plasma could crash on Wayland when a screen turns off (David Edmundson, Qt 6.7.3. Link)

Fixed yet another common way that Plasma could crash on Wayland, this time when showing notifications (David Edmundson, Qt 6.7.3. Link)

Fixed a case where Plasma could crash when you trigger the Meta+V keyboard shortcut to open the clipboard history menu over and over again in rapid succession (Vlad Zahorodnii, Qt 6.7.3. Link)

Fixed a case where trying to save a file in a Flatpak or Snap app using the standard Save dialog could fail and cause the saving app to quit instead! (Nicolas Fella, Qt 6.7.3. Link)

Other bug information of note:

Performance & Technical

Receiving a Plasma notification no longer blocks KWin’s “direct scan-out” feature, e.g. while playing a game, so it should no longer briefly reduce performance (Xaver Hugl, Plasma 6.1.3. Link)

Improved KWin’s detection for whether triple buffering on Wayland will improve things so that it won’t occasionally turn on and off repeatedly, impairing performance (Xaver Hugl, Plasma 6.1.3. Link)

Plasma’s RDP server is now capable of listening for IPv6 connections (Arjen Hiemstra, Plasma 6.1.3. Link)

KWin now disables 10 bits-per-color (BPC) support for monitors plugged into a dock, as these often limit the signal to 8 BPC but don’t tell KWin, causing issues when KWin tries to enable 10 BPC mode because it thinks it should be possible (Xaver Hugl, Plasma 6.1.4. Link)

KWin now operates with “realtime” capabilities on systemd using musl instead of glibc (Vlad Zahorodnii, Plasma 6.1.4. Link)

Plasma’s RDP server now also works as expected on systems using the OpenH264 video codec (Fabian Vogt, Plasma 6.2.0. Link)

Relevant only for cutting-edge distro-builders: It’s now possible to compile KWin with support for only the Wayland session, so support for X11 apps would be provided exclusively through XWayland (Neal Gompa, Plasma 6.2.0. Link)

Improved performance for everything in KDE that uses KFileItem::isHidden (Volker Krause, Frameworks 6.5. Link)

Created a new WindowStateSaver QML object you can add to apps’ windows to make them remember their size, maximization state (and position, on X11) (Joshua Goins, Frameworks 6.5. Link)

Apps storing their transient state data separately from their persistent configuration data now do so by putting the state config file in the standard XDG state folder of ~/.local/state/ (Harald Sitter, Frameworks 6.5. Link)

Automation & Systematization

Added an autotest to test clearing the clipboard history (Fushan Wen, link)

…And Everything Else

This blog only covers the tip of the iceberg! If you’re hungry for more, check out https://planet.kde.org, where you can find more news from other KDE contributors.

How You Can Help

If you use have multiple systems or an adventurous personality, you can really help us out by installing beta versions of Plasma using your distro’s available repos and reporting bugs. Arch, Fedora, and openSUSE Tumbleweed are examples of great distros for this purpose. So please please do try out Plasma beta versions. It truly does help us! Heck, if you’re very adventurous, live on the nightly repos. I’ve been doing this full-time for 5 years with my sole computer and it’s surprisingly stable.

Does that sound too scary? Consider donating today instead! That helps too.

Otherwise, visit https://community.kde.org/Get_Involved to discover other ways to be part of a project that really matters. Each contributor makes a huge difference in KDE; you are not a number or a cog in a machine! You don’t have to already be a programmer, either. I wasn’t when I got started. Try it, you’ll like it! We don’t bite!

The European Union must keep funding free software

Initially publishead by petites singularités. English translation provided by OW2.

If you want to sign this letter, please publish this text on your website and add yourself or your organization to the table you will find on the original site.

Open Letter to the European Commission

Since 2020, Next Generation Internet (NGI) programmes, part of European Commission's Horizon programme, fund free software in Europe using a cascade funding mechanism (see for example NLnet's calls).

Quite a few of KDE's projects have benefited from NGI's funding, including NeoChat, Kaidan, KDE Connect, KMail, and many others. KDE e.V. is a European non-profit with limited resources that relies on donations, sponsors and funding like that offered by NGI, to push the development of our projects forward.

However, this year, according to the Horizon Europe working draft detailing funding programmes for 2025, we notice that Next Generation Internet is not mentioned any more as part of Cluster 4.

NGI programmes have shown their strength and importance to supporting the European software infrastructure, as a generic funding instrument to fund digital commons and ensure their long-term sustainability. We find this transformation incomprehensible, moreover when NGI has proven efficient and economical to support free software as a whole, from the smallest to the most established initiatives. This ecosystem diversity backs the strength of European technological innovation, and maintaining the NGI initiative to provide structural support to software projects at the heart of worldwide innovation is key to enforce the sovereignty of a European infrastructure. Contrary to common perception, technical innovations often originate from European rather than North American programming communities, and are mostly initiated by small-scaled organizations.

Previous Cluster 4 allocated 27 million euros to:

  • "Human centric Internet aligned with values and principles commonly shared in Europe" ;
  • "A flourishing internet, based on common building blocks created within NGI, that enables better control of our digital life" ;
  • "A structured ecosystem of talented contributors driving the creation of new internet commons and the evolution of existing internet commons".

In the name of these challenges, more than 500 projects received NGI funding in the first 5 years, backed by 18 organisations managing these European funding consortia.

NGI contributes to a vast ecosystem, as most of its budget is allocated to fund third parties by the means of open calls, to structure commons that cover the whole Internet scope - from hardware to application, operating systems, digital identities or data traffic supervision. This third-party funding is not renewed in the current program, leaving many projects short on resources for research and innovation in Europe.

Moreover, NGI allows exchanges and collaborations across all the Euro zone countries as well as "widening countries"1, currently both a success and an ongoing progress, likewise the Erasmus programme before us. NGI also contributes to opening and supporting longer relationships than strict project funding does. It encourages implementing projects funded as pilots, backing collaboration, identification and reuse of common elements across projects, interoperability in identification systems and beyond, and setting up development models that mix diverse scales and types of European funding schemes.

While the USA, China or Russia deploy huge public and private resources to develop software and infrastructure that massively capture private consumer data, the EU can't afford this renunciation. Free and open source software, as supported by NGI since 2020, is by design the opposite of potential vectors for foreign interference. It lets us keep our data local and favors a community-wide economy and know-how, while allowing an international collaboration.

This is all the more essential in the current geopolitical context: the challenge of technological sovereignty is central, and free software allows to address it while acting for peace and sovereignty in the digital world as a whole.


1 As defined by Horizon Europe, widening Member States are Bulgaria, Croatia, Cyprus, the Czech Republic, Estonia, Greece, Hungary, Latvia, Lituania, Malta, Poland, Portugal, Romania, Slovakia and Slovenia. Widening associated countries (under condition of an association agreement) include Albania, Armenia, Bosnia, Feroe Islands, Georgia, Kosovo, Moldavia, Montenegro, Morocco, North Macedonia, Serbia, Tunisia, Turkey and Ukraine. Widening overseas regions are : Guadeloupe, French Guyana, Martinique, Reunion Island, Mayotte, Saint-Martin, The Azores, Madeira, the Canary Islands.

Friday, 19 July 2024

Let’s go for my web review for the week 2024-29.


The graying open source community needs fresh blood • The Register

Tags: tech, foss, community

This is indeed a problem. Somehow it became much harder to attract younger developers.

https://www.theregister.com/2024/07/15/opinion_open_source_attract_devs/


“Privacy-Preserving” Attribution: Mozilla Disappoints Us Yet Again

Tags: tech, browser, mozilla, privacy

You’d expect Mozilla to know better. This is disappointing, they’re no living up to their responsibility.

https://blog.privacyguides.org/2024/07/14/mozilla-disappoints-us-yet-again-2/


Commission sends preliminary findings to X for breach of DSA

Tags: tech, twitter, social-media, law

The European Commission starts showing it’s muscles. Twitter is an obvious one to pursue since it became the X cesspool.

https://ec.europa.eu/commission/presscorner/detail/en/IP_24_3761


Goldman Sachs: AI Is Overhyped, Wildly Expensive, and Unreliable

Tags: tech, ai, machine-learning, gpt, economics, ecology, criticism

I’m rarely on the side of a Goldman Sachs… Still this paper seems to be spot on. The equation between the costs (financial and ecological) and the value we get out of generative AI isn’t balanced at all. Also, since it is stuck on trying to improve mostly on model scale and amount of data it is doomed to plateau in its current form.

https://www.404media.co/goldman-sachs-ai-is-overhyped-wildly-expensive-and-unreliable/


Facebook Is the ‘Zombie Internet’

Tags: tech, ai, social-media, facebook

Or examples of the collapse of a shared reality. This has nothing to do with “social” media anymore. Very nice investigation in any case.

https://www.404media.co/email/24eb6cea-6fa6-4b98-a2d2-8c4ba33d6c04/


AT&T says criminals stole phone records of ‘nearly all’ customers in new data breach

Tags: tech, security, leak

Wow! This is a really bad data breach. Apparently related to the recent data theft on the Snowflake end.

https://techcrunch.com/2024/07/12/att-phone-records-stolen-data-breach/


git-pr: A new git collaboration service

Tags: tech, git, codereview, tools

Interesting approach to building a new code review system. I somehow doubt it’ll get traction unfortunately but it has nice ideas baked in.

https://pr.pico.sh/


gpu.cpp: portable GPU compute for C++ with WebGPU – Answer.AI

Tags: tech, c++, gpu, computation

Looks like an interesting library to build portable GPU compute workloads. Cleverly tries to leverage WebGPU.

https://www.answer.ai/posts/2024-07-11–gpu-cpp.html


C++ Design Patterns For Low-Latency Applications

Tags: tech, c++, performance, optimization, pattern

A paper listing patterns to reduce latency as much as possible. There are lesser known tricks in there.

https://hackaday.com/2024/07/13/c-design-patterns-for-low-latency-applications/


22 Common Filesystem Tasks in C++20

Tags: tech, c++

Nice little reference of what can be done with std::filesystem.

https://www.cppstories.com/2024/common-filesystem-cpp20/


C++ Must Become Safer

Tags: tech, c++, safety, memory

Definitely this. C++ isn’t going away anytime soon. Rewrites won’t be worth it in important cases, so improving the safety of the language matters.

https://www.alilleybrinker.com/blog/cpp-must-become-safer/


Django Migration Operations aka how to rename Models

Tags: tech, django, databases

Django doesn’t always generate the migration you’d expect. Read them before going to production. Also it’s fine to adjust them.

https://micro.webology.dev/2024/07/15/django-migration-operations.html


Gotchas with SQLite in Production

Tags: tech, backend, sqlite, databases

Where are the limitations of using SQLite in production for web applications? Here is a good list.

https://blog.pecar.me/sqlite-prod


SQLite Transactions

Tags: tech, databases, sqlite

Some improvements coming in SQLite transactions. Here are some early tests.

https://reorchestrate.com/posts/sqlite-transactions/


Tests you love to read, write and change

Tags: tech, tests

Three good advices on writing automated tests. This is necessary but not sufficient though.

https://jaywhy13.hashnode.dev/tests-you-love-to-read-write-and-change


Lessons learned in 35 years of making software – Jim Grey

Tags: tech, career

Quite a few good lessons in there. Again it’s more about social skills than technical skills.

https://dev.jimgrey.net/2024/07/03/lessons-learned-in-35-years-of-making-software/


Story Points are Pointless, Measure Queues

Tags: tech, project-management, product-management, estimates, agile

A bit long and a couple of mistakes when pointing out the flaws of story points. Still, it’s definitely a worthwhile read. Quite a lot of the criticism of story points is warranted and the proposed approach based on queue theory is welcome. This is stuff you can find in Kanban like approaches and mature XP.

https://www.brightball.com/articles/story-points-are-pointless-measure-queues


Managing Underperformers | Jack Danger

Tags: management

Nice advices to deal with underperforming teams or individuals. Making the distinction between refusal to align or failure to execute is particularly useful.

https://jackdanger.com/managing-underperformers/


All I Need to Know About Engineering Leadership I Learned From Leave No Trace - Jacob Kaplan-Moss

Tags: tech, leadership, engineering, ecology, funny

Funny experiment at drawing parallels between engineering leadership and how you should behave when hiking in nature. This works surprisingly well.

https://jacobian.org/2024/jul/12/lnt-for-engineering-leadership/


Progress can be slow

Tags: work, life, improving, coaching, habits

This one is more self-help than I’m usually comfortable with… somehow something rung true to me with it. It’s indeed a good reminder that changing habits takes a while. It’s an exercise in patience and there are good reasons for it.

https://jeanhsu.substack.com/p/progress-can-be-slow?isFreemail=true&post_id=146457673


German Navy still uses 8-inch floppy disks, working on emulating a replacement | Ars Technica

Tags: tech, hardware, storage, history

We keep finding floppies in use at surprising places. There’s clearly lot of inertia for technologies getting replaced.

https://arstechnica.com/gadgets/2024/07/german-navy-still-uses-8-inch-floppy-disks-working-on-emulating-a-replacement/



Bye for now!

Thursday, 18 July 2024

An over-the-air (OTA) update capability is an increasingly critical part of any embedded product to close cybersecurity vulnerabilities, allow just-in-time product rollouts, stomp out bugs, and deliver new features. We’ve talked about some of the key structural elements that go into an embedded OTA architecture before. But what about the back end? Let’s address some of those considerations now.

The challenges of embedded connectivity

The ideal of a constant Internet connection is more aspiration than reality for many embedded devices. Sporadic connections, costly cellular or roaming charges, and limited bandwidth are common hurdles. These conditions necessitate smart management of update payloads and robust retry strategies that can withstand interruptions, resuming where they left off without getting locked in a continually restarting update cycle.

There are other ways to manage spotty connections. Consider using less frequent update schedules or empower users to initiate updates. These strategies however have trade-offs, including the potential to miss critical security patches. One way to strike a balance is to implement updates as either optional or mandatory, or flag updates as mandatory only when critical, allowing users to pace out updates when embedded connectivity isn’t reliable.

To USB or not to USB

When network access is very unreliable, or even just plain absent, then USB updates are indispensable for updating device software. These updates can also serve as effective emergency measures or for in-field support. While the process of downloading and preparing a USB update can often be beyond a normal user’s capability, it’s a critical fallback and useful tool for technical personnel.

OTA servers: SaaS or self-hosted

Deciding between software as a service (SaaS) and self-hosted options for your OTA server is a decision that impacts not just the update experience but also compliance with industry and privacy regulations. While SaaS solutions can offer ease and reliability, certain scenarios may necessitate on-premise servers. If you do need to host an OTA server yourself, you’ll need to supply the server hardware and assign a maintenance team to manage it. But you may not have to build it all from scratch – you can still call in the experts with proven experience in setting up self-hosted OTA solutions.

Certificates: The bedrock of OTA security

SSL certificates are non-negotiable for genuine and secure OTA updates. They verify your company as the authentic source of updates. Choosing algorithms with the longest (comparatively equivalent) key lengths will extend the reliable lifespan of these certificates. However, remember that certificates do expire; having a game plan in place to deal with expired certificates will allow you to avoid the panic of an emergency scramble if it should happen unexpectedly.

Accurate timekeeping is also essential for validating SSL certificates. A functioning and accurate real-time clock that is regularly NTP/SNTP synchronized is critical. If timekeeping fails, your certificates won’t be validated properly, causing all sorts of issues. (We recommend reading our OTA best practice guide for advice on what to do proactively and reactively with invalidated or expired certificates.

Payload encryption: Non-negotiable

Encrypted update payloads are imperative as a safeguard against reverse-engineering and content tampering. This is true for OTA updates as well as any USB or offline updates. Leveraging the strongest possible encryption keys that your device can handle will enhance security significantly.

Accommodating the right to repair

The growing ‘right to repair’ movement and associated legislation imply that devices should support updates outside of your organization’s tightly controlled processes. This may mean that you need to provide a manual USB update to meet repair requirements without exposing systems to unauthorized OTA updates. To prevent your support team from struggling with amateur software updates, you’ll want to configure your device to set a flag when unauthorized software has been loaded. This status can be checked by support teams to invalidate support or warranty agreements.

Summary

By carefully navigating the critical aspects of OTA updates, such as choosing the right hosting option and managing SSL certificates and encryption protocols, your embedded systems can remain up-to-date and secure under any operating conditions. While this post introduces the issues involved in embedded-system updates, there is much more to consider for a comprehensive strategy. For a deeper exploration and best practices in managing an embedded product software update strategy, please visit our best practice guide, Updates Outside the App Store.

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

The post Behind the Scenes of Embedded Updates appeared first on KDAB.

Design Considerations

One of the main focuses while designing this engine was flexibility, to ensure the library is usable for a wide variety of Mancala variants. While certain abstractions are necessary to achieve this goal, it’s also important not to overly abstract the library.

Provided Functionality

MankalaEngine provides classes to assist in creating computerized opponents for many Mancala variants. As mentioned earlier, these classes are designed with a degree of abstraction to allow for greater flexibility on the developer’s end. In addition to these base classes, a concrete implementation tailored for the Bohnenspiel variant [1] is also provided.

The Board struct

The board struct is the base struct for a Mancala game board. It is used to specify the structure of the board used by a variant. As of now, this struct allows for a board with an arbitrary number of holes and two stores, one per player, as this seems to be the case for most Mancala games.

The Rules class

The rules class is the base class for the rules of a Mancala variant. It is used to specify the behaviour when making a move, what constitutes a valid move, when the game is over, etc. The only rule assumed to be shared by all variants is that the winning player is the one with more pebbles in their store, as this seems to be relatively common in Mancala variants.

The moveselection functions

The functions provided in the moveselection file are general adversarial search functions that can be used to select moves for Mancala games. In addition to Minimax [2] and MTDF-f [3], random selection and user selection functions are also provided.

The Minimax move selection function

Minimax works by recursively exploring the game tree, considering each player’s moves and assuming that each player chooses the optimal move to maximize their chances of winning. If we’re scoring a Mancala position using an evaluation function that subtracts the pebbles in Player 2’s store from the pebbles in Player 1’s store, this means that Player 1 will want to maximize the score, while Player 2 will want to minimize it. The diagram below shows how a position is evaluated using the Minimax algorithm.

Minimax Diagram

Each node represents a specific board configuration, and each level of the tree represents a turn of play. The tree nodes are squares on Player 1’s turn and circles on Player 2’s turn. Leaf nodes (nodes without children) are scored using the evaluation function. The rest of the nodes are scored by selecting the best score out of their children nodes - highest score if it’s Player 1’s turn and lowest score if it’s Player 2’s turn.

The Minimax implementation in the library also uses alpha-beta pruning, a technique used to reduce the number of nodes evaluated in the tree by eliminating branches that are guaranteed to be worse than previously examined branches.

A great explanation of Minimax and Alpha-beta prunning can be found in Sebastian Lague’s Youtube video about this algorithm.

The MTD-f move selection function

MTD-f works by repeatedly calling Minimax until it converges to a value. The Minimax used by MTD-f is implemented using alpha-beta pruning.

Since MTD-f calls Minimax several times, it’s also important to use a transposition table, which is a data structure that stores previously evaluated positions and their scores.

Below is Aske Plaat’s pseudo-code for the algorithm.

function MTDF(root : node_type; f : integer; d : integer) : integer;

   g := f;
   upperbound := +INFINITY;
   lowerbound := -INFINITY;
   repeat
       if g == lowerbound then beta := g + 1 else beta := g;
       g := Minimax(root, beta - 1, beta, d);
       if g < beta then upperbound := g else lowerbound := g;
   until lowerbound >= upperbound;
   return g;

Evaluating Mancala positions

The static evaluation function used in this library consists of subtracting the pebbles in Player 2’s store from the pebbles in Player 1’s store. This is the same function that was used when solving the Mancala variant Kalah [4].

This way of scoring Mancala positions is particulary suitable for MTD-f, since, according to Plaat, the static evaluation function used should be coarse-grained, meaning that we should avoid using heuristics with little weight. As he says in his post about MTD(f), “The coarser the grain of eval, the less passes MTD(f) has to make to converge to the minimax value. Some programs have a fine grained evaluation function, where positional knowledge can be worth as little as one hundredst of a pawn.” [3].

The MankalaEngine class

The MankalaEngine class ties everything together. When instatiating it, you’ll need to choose a move selection function. It then provides a function, play, that, given the player whose turn it is to play, the rules to use, and the board in which the move will be played, executes the move selected by its move selection function. This allows reusing the common structure of a play across all Mancala variants while deferring variant-specific behaviour to the rules object.

bool MankalaEngine::play(Player player, const Rules& rules,
                         Board& state) const {
    if (rules.gameOver(player, state)) {
        const Player winner = player == player_1 ? player_2 : player_1;
        rules.finishGame(winner, state);
        return false;
    }
    const int move = _selectMove(player, rules, state);
    rules.move(move, player, state);
    return true;
}

Next steps

The idea is to continue adding concrete variant implementations to the library so that developers wanting to create a Mancala game don’t have to implement them themselves. Additionally, adding the option to choose the difficulty of an opponent is also relevant. This may be implemented, for example, by allowing changes to the cutoff depth for the Minimax and MTD-f opponents, which is not currently supported.

As of now, the implemented Minimax only uses alpha-beta prunning and transposition tables. Adding more optimizations, such as move ordering, per example, might also be of interest.

Another possible route is developing a Qt application for playing Mancala that uses this engine. This would likely help generate interest in the project within the broader community.

If you’re interested in this project, you can check it out on Invent and come talk to us on Matrix.

References

1. “Das Bohnenspiel”, Wikipedia, 2023. https://en.wikipedia.org/wiki/Das_Bohnenspiel.

2. “Algorithms - Minimax” https://cs.stanford.edu/people/eroberts/courses/soco/projects/2003-04/intelligent-search/minimax.html.

3. “Aske Plaat: MTD(f), a new chess algorithm.” https://people.csail.mit.edu/plaat/mtdf.html.

4. G. Irving, J. Donkers, and J. Uiterwijk, “Solving Kalah”, Icga journal, vol. 23, no. 3, pp. 139–147, Sep. 2000, doi: 10.3233/ICG-2000-23303.

Wednesday, 17 July 2024

The Akademy 2024 Program is now available.

This year's Akademy will take place in Würzburg, a beautiful city where you can enjoy interesting and fascinating talks, panels and keynotes. And for those who prefer to participate remotely, Akademy will also be available online.

Akademy officially kicks off with a welcome event on Friday 6 September, followed by a series of talks on Saturday 7 September and Sunday 8 September. From Monday 9 to Thursday 12 September, there will be BoFs (Birds of a Feather), workshops, meetings, daytrip and training sessions.

The talks will cover KDE's goals, how we're doing with implementing other languages to code for KDE (Rust anyone?), what's new in the latest wave of desktop and mobile applications, how KDE Eco is saving the environment, backends, frontends, KDE for work, life and fun.

For example, Nicolas Fella will tell us what a software maintainer does and why they are crucial to a project's survival, Aleix Pol Gonzalez will demystify embedded Linux, and Kevin Ottens will take us deep into the core of KDE Neon. You will also learn more about Plasma Mobile, funding your dream project and cool new KWin effects.

You can expect much, much more from a schedule packed with exciting talks and eye-opening presentations. Just take a look at the full program to discover everything that will be happening.

And that is not all! Stay tuned for the announcement of our two keynote speakers, coming soon here on Planet.

During the week KDE community members will attend BoFs and meet with colleagues with similar interests to work on their projects. They will also attend workshops, meetings, training sessions and daytrip until the event closes on 12 September.

Tuesday, 16 July 2024

Say hello to LabPlot 2.11!

This brand new release comes with many new features, improvements and performance enhancements in several areas, as well as support for more data formats and visualisation types.

The main new features are outlined below. For a more detailed overview of the changes in this release, please refer to the ChangeLog file.

The source code of LabPlot, the Flatpak and Snap packages for Linux, as well as the installer for Windows and the image for MacOS are available from our download page.

What’s new in 2.11?

Worksheet

This release includes more visualisations, usability improvements and a new worksheet preview panel:

  • You can now use Lollipop, Q-Q and KDE plots
  • We have implemented error bars for bar plots
  • There is a new preview panel for all available worksheets in the project
  • You can use the navigation panel in the presenter widget to select, zoom and navigate in the presenter mode
  • You can lock worksheet elements to prevent accidental changes
  • LabPlot 2.11 allows you to show or hide the entry in the legend for all supported plot types and not just xy-curve
  • You can give your worksheets a fresh new look with the Dracula theme

Spreadsheet

Spreadsheets gain more functions and operations to modify, generate and understand the data:

  • We have extended the search and replace features
  • You can check statistical properties of the parent in a new child spreadsheet
  • We have added sparklines in the header of a spreadsheet
  • LabPlot 2.11 comes with spreadsheet linking to synchronize the number of rows across multiple spreadsheets
  • We have implemented triangular distributions for PDF, CDF, and pseudorandom number generation
  • Equidistant value generation has been extended

Analysis

Analysis tools added to LabPlot 2.11 include:

  • Note showing the fit results
  • Faster computation of the baseline removal (we switched to Eigen3 internally)

Import/Export

LabPlot 2.11 adds support for new file formats and multiple optimizations to improve the handling of edge-case scenarios:

  • You can now import Open Document Spreadsheet (ODS) files.
  • Templates for ASCII and Binary import filters allow you to save and load current filter settings
  • There is a new feature that allows you to to specify the data range to be read (start/end values for columns and records) when importing from SQL databases
  • LabpPlot can now gracefully handle out-of-memory situations when importing large amounts of data
  • LabPlot 2.11 displays better error messages during the import
  • We provide additional information about BLF files (application name with which the file was created with, etc.)
  • We have made several fixes and improvements to the import of Origin’s OPJ files

Notebook

The 2.11 release adds a number of usability enhancements to the Notebook interface:

  • You can now export the notebook to PDF
  • We provide statistics and a “plot data” action from the context menu in the project explorer for variables created in the Notebook
  • There is a new option in the application settings to run a selected CAS engine on startup