AffûtOS

Until now, the installer itself copied my settings into the system: desktop configuration, the template for new accounts, system rules. It worked on install day and never again: a fixed setting never reached machines already installed, and no setting could ever be removed.

Seven packages, three rules

Settings now belong to affutos-* packages in the AffûtOS repository: system, boot, AMD GPU, KDE, Firefox, Steam, Decky. Installing a package applies its settings, removing it takes them away, updating it fixes them on every machine — with one exception, forced by the second rule. The image build rejects any setting copied outside a package; the installer keeps only what happens once, such as the restored backup or the firmware boot entry.

• A setting KDE reads as a global default goes into /etc/xdg: changed for everyone on update, and the account can always override it;
• a per-account setting — Steam, Firefox, files Plasma rewrites — goes into the template for new accounts, /etc/skel, as a file owned by the package. The downside: an existing account keeps its copy, which neither an update nor a removal touches;
• a service is enabled by a link shipped in the package, not by a command: removing the package is enough to disable it. The one exception is Decky: its service is named after the account, which the package cannot know in advance, so it enables it for the first account and disables it when removed.

The CPU, memory and disk monitors in the taskbar go through a Plasma update script instead, run once per account at login: they land right after the task list, as on my main machine.

The culprit was pacman

The affutos-firefox package installs the system language pack when it is installed, then puts it back after every browser update. On an already installed machine with a flawless network, reinstalling it failed at the download: curl: (7) Failed to connect to archive.mozilla.org:443 after 1 ms. One millisecond is not a slow DNS: the connection never left the machine.

The answer was in the pacman.conf manual: since pacman 7.1, hooks and package install scripts run by default in a fresh network namespace, with no interface but loopback. No download gets through. That was the cause of the “could not resolve host” errors seen during installs since late September — which I had first blamed on the dangling DNS link suspected in mid-August, then filed as “cause unknown” on 4 October.

A hook can ask for the network (NetworkAccess = allowed), a package install script cannot — and the pack is also installed when the package itself is. So both the hook and the script ask systemd to start a service, which downloads outside the isolated namespace: systemd's sockets are files, reachable even without a network. During installation, where systemd is not yet running in the target system, a post-install step replays the download.

Three fewer errors in the system journal

Three errors cluttered the system journal. Two came from the Game Mode session, which starts a Steam notification daemon and an input method whose programs are not installed: their services are now skipped while their program is missing. The third came from the service that keeps the Firefox profile in RAM: it also started for root, which has no profile, and only runs for regular accounts now.

Checked both ways

On a test machine, every package was removed and reinstalled: its system files and services disappeared and came back identical. After that, two full installs in a virtual machine with the new images: none of the three errors in the system journal, and Decky running from the first boot.

One trap along the way: on a fresh install, Decky refused to start for lack of a folder my test account already had. A service checked on an existing account proves nothing for a new one: every service is now also tested on an account created for the occasion.

The lesson: as long as the failure only happened in the installer, I searched the installer. It took seeing it on an ordinary machine for me to finally look at pacman.

For two months AffûtOS has drifted away from CachyOS one fix at a time, each with its reason written in a comment. What I had never done was install both side by side and look at the difference, file by file.

The bench

Two virtual machines on the same computer, each on its own disk, both fed by the same local package cache: CachyOS handheld with its defaults — except the filesystem and the bootloader, set to ext4 and systemd-boot like mine — and my working version of AffûtOS. Then an offline survey of both installed systems — packages, files owned by no package, enabled services — and their boot timings.

• Boot: 4.56 s against 6.78 s, even though my working version starts a few more services — the initramfs phase takes 0.07 s with booster against 1.65 s;
• and five seconds of boot menu on top for CachyOS, against none for AffûtOS.

A wrong explanation, in production for weeks

In mid-August I had blamed an installer step that, I thought, replaced the DNS configuration with a dangling link, and I had added a step to repair it. In September, a Firefox language pack that failed to download during an install — “could not resolve host” — seemed to prove me right. The comparison sank that story: the live system has had no such link since 2022, the step copies an ordinary file identical to the one already in place — and CachyOS, with no repair step at all, ends up with exactly the same file.

The repair step is gone. The Firefox symptom is real, and its cause is still unknown; what no longer depends on it is that downloads outside pacman — language pack, Decky, Steam client — already happen in the post-install steps, where the network works.

Update, 8 October: the cause has been found — since pacman 7.1, package hooks have no network by default. See “Every setting in a package”.

A clean-up you can check

Every difference then had to justify itself against the original. Result: 1,224 lines removed and 391 added, over ten commits — network-boot entries nobody used, up to three minutes of waiting before the installer opened, for a mirror ranking it only reads once installing starts, a copy of my script that nothing read anymore, duration estimates exceeded at nearly every install. Reinstalled after the clean-up, AffûtOS had the same packages, the same services, and only the file differences that were intended: same behaviour, less code.

Along the way, a fix CachyOS had published on its main branch — a mirror-ranking timer to mask on the live system — was ported too.

The lesson echoes August's: reading your own code convinces you, putting it next to the original does not lie.

The goal: shave seconds off the time from power button to desktop on my main machine. The first question came before any change: how to measure. systemd only sees what happens after the firmware; to compare a classic bootloader with a kernel launched directly by the firmware, you need the firmware's own clock — an ACPI table records the moment it hands over. Each configuration was booted three to six times.

• Starting point, mkinitcpio: 12.83 s from power-on to desktop.
• An “optimized” mkinitcpio — fewer hooks, uncompressed modules, maximum compression: 12.81 s. Nothing.
• booster, which only packs what this machine needs (under 10 MB instead of 16.7): 11.52 s, 1.3 s saved. On my other laptop, 2.5 s.
• A unified kernel image launched by the firmware, with no bootloader at all: 11.47 s against 11.45 s. Nothing either: the bootloader's 0.3 s is mostly the firmware reading the kernel from disk, which it has to do anyway.

The second nobody sees

The boot menu, set to one second so I could still pick an entry, actually cost 1.2 seconds per boot; on my other laptop the bootloader alone took up to 1.8 seconds. The timeout is now zero, and holding a key brings the menu back.

Two traps

• booster's strip option, which slims the modules down, also strips their signatures: the kernel then marks itself as tainted. It stays off — the size difference is under a megabyte.
• Creating or deleting a UEFI boot entry makes my machine's firmware run a full initialization on the next boot: 7.3 s instead of 4.4. Any measurement taken right after touching the boot entries is wrong; it takes one warm-up boot first.

Where to stop looking

Sampling the rest of the boot showed that the login screen, at 6.8 s, is held back by two drivers: the AMD GPU, which only starts probing at 5.75 s, and the Wi-Fi card applying its radio regulations. Whatever is left on systemd's side is worth about 0.2 s. The measurements say where to stop digging as much as where to dig.

A read through the code had named a perfect culprit: packages preloaded at boot did not land where the installer looked for them, and would therefore be downloaded all over again. Two minutes to claw back, on paper.

The fix was a regression

Twice over. The installer does not call pacman --root but pacman --sysroot — and that option makes it read the configuration and the cache inside the system being installed, not on the live medium. The original code was already pointing at the right place. My “fix” was diverting 746 MB into a cache nobody reads, under filenames the installer never looks for.

Twelve packages picked at random among the preloaded ones: twelve downloaded again. The live cache had not grown by a single byte over the whole install. Only a real run showed it — reading the code had convinced two successive analyses.

The real win was elsewhere, and it is huge

A package cache on the local network, on the other hand, changes everything. Measured on the living-room machine, full install:

• 5 minutes instead of 12, for 1271 packages installed;
• 341 MB pulled from the Internet instead of 4.2 GB;
• 3347 files served by the local cache against 177 fetched from outside, or 95% of the requests.

The lesson fits on one line: an assumption about a cache is checked against file timestamps after the fact, never by reading command-line options.

Until now, preparing the installer meant patching it on the fly: over a thousand lines of substitutions applied to its configuration file, on every boot of the live system. It worked, and it was untenable — no way to tell what differed from upstream without walking the script by hand.

The configuration now lives in the image, in plain text, as an overlay the installer reads before its own. It can be read back, diffed, versioned. The launcher checks in the log which configuration was actually opened and stops if it is the wrong one: a silent fallback would install the upstream distribution without warning.

In the same pass, post-install left its background script to become a step called by the installer. And a full review dropped 925 lines: package groups a console edition cannot display, a bootloader nothing knew how to configure, startup files nothing ever read.

A detail measured along the way: with no explicit setting, the kernel tooling rebuilt an initramfs and a boot entry already produced by the previous step — 64 MB wasted on a 196 MB boot partition.

AffûtOS network boot started refusing the system image: checksum error. The usual reflex: regenerate the checksum. It would have been the worst possible call.

The disk was telling the truth, memory was lying

Comparing the published image with the one in the working directory, a single byte differed out of 2,823,114,752 — and it differed by one bit: 0xA2 had become 0xAA. The rest of the image, kernel and initramfs included, was identical down to the bit.

The timeline placed the accident in a window of a second and a half, while the image was being copied into the ISO file. The source file on disk, read back later bypassing the page cache, was perfectly sound.

The proof, a few hours later

The same day, a package in the cache started failing its signature check. Read normally — so from memory — its signature was invalid. Read straight from the disk, it was perfect. And the faulty bit was the same one as in the image: bit 3.

The machine has non-ECC memory, four sticks pushed past the frequencies the vendor validated. It has no way to detect, let alone correct, that kind of accident.

What I took from it

• A checksum that fails is good news: it did its job. Regenerating it amounts to officially certifying corrupted data.
• A check that reads a file back right after writing it only checks memory. You have to drop the cache before reading back, otherwise you validate the suspect copy against itself.
• Nothing gets published now without a disk read-back of every file, not just the biggest one.

The server side of this incident

The goal: join Wi-Fi from the initramfs, before the system exists, so a machine can be installed with no network cable. The script was written, injected into the image, and never ran. No message. No error. Nothing.

Everything was right, except one thing

The file was there in the image — checked. Its syntax was valid — checked with the exact interpreter the initramfs uses. Sourcing it did define the expected function — checked. It was listed among the scripts to run — checked. And still it never ran.

The answer was in the boot system's own code, a single line:

[ -x "/hooks/$hook" ] || continue

A non-executable script is ignored, without so much as a message. It is even the official way to disable one: the function meant to switch a script off simply strips its execute bit.

Rebuilding the image by hand, I had recreated the file as 0644 instead of 0755. I had been disabling my own script without knowing it, on every single attempt.

Two lessons in method

• I added a version number printed on screen by the script. Without it there is no way to know which version is actually running — I spent an hour testing fixes that were never loaded.
• The initramfs console does not render accented characters: every one of my French diagnostic messages showed up as unreadable symbols. They had been there from the start, I simply could not read them.

I wanted three Game Mode settings applied from install time, with no manual step: never dim the screen on AC power, never suspend, and always show the battery percentage. The key names looked obvious. They did not work.

The right key in the wrong place

The configuration file contains two blocks with the same name. The software reads only one: the deeper one. I was writing into the other. The key was right, the value was right, the location was wrong — so nothing happened, without the slightest warning.

Same trap for the battery percentage a few hours later: the key had to live in one specific sub-block, alongside three sibling keys written by the same component. At the root it stayed a dead letter.

The method that worked

Two complementary approaches. First the empirical one: snapshot the files, change the setting by hand in the UI, compare. The file then confesses exactly what to write, and where.

Then reading the sources. Steam's interface is no longer a compiled library but minified JavaScript: a hundred-odd megabytes of unreadable yet perfectly searchable code. In it you find the exact key name, the function that writes it, and the one that reads it back at startup — hence the certainty that a value set in advance will really be picked up.

One trap to avoid at all costs

A trick going around for disabling the first-boot wizard: create a certain configuration file with a magic line. When the launcher meets that line, it runs an rm -rf on the entire application folder — on every startup. The correct solution uses a bootstrap file the vendor provides, which nobody seems to use.