#archlinux — Public Fediverse posts on home.social

How do I troubleshoot a black screen on #KDE #Plasma? I am able to launch apps just fine, but:
- the desktop is all black (no start menu, global menu etc, and the desktop background isn't getting rendered)
- the window decorations are HUGE.

Is there any easy way to debug this? I had initially suspected this was due to my using #Stylix on stock #archlinux, but no, disabling it didn't change a thing (apart from the theme).

I do have custom desktop effects and window decorations installed; so I suspect that might be the issue. How do I disable them if systemsettings does not launch (for some weird reason)?

It's definitely not a kernel issue (I've tried on the stock Arch kernel and the CachyOS kernel), and it certainly isn't a graphics issue (stock mesa etc, and I don't have a dGPU on this machine).

Will send screenshots in a while. Any and all help appreciated! Thanks in advance. Toots and boosts welcome.

#linux #kde #plasma #askfedi #linuxhelp #linuxquestions

#askfedi #linux #linuxhelp #linuxquestions #kde #plasma

ZeStig :emacs: :nix: :rust: :archlinux: @[email protected] · 2026-05-14 · 12:52 UTC

How do I troubleshoot a black screen on #KDE #Plasma? I am able to launch apps just fine, but:
- the desktop is all black (no start menu, global menu etc, and the desktop background isn't getting rendered)
- the window decorations are HUGE.

Is there any easy way to debug this? I had initially suspected this was due to my using #Stylix on stock #archlinux, but no, disabling it didn't change a thing (apart from the theme).

I do have custom desktop effects and window decorations installed; so I suspect that might be the issue. How do I disable them if systemsettings does not launch (for some weird reason)?

It's definitely not a kernel issue (I've tried on the stock Arch kernel and the CachyOS kernel), and it certainly isn't a graphics issue (stock mesa etc, and I don't have a dGPU on this machine).

Will send screenshots in a while. Any and all help appreciated! Thanks in advance. Toots and boosts welcome.

#linux #kde #plasma #askfedi #linuxhelp #linuxquestions

#kde #plasma #stylix #archlinux #askfedi #linux

ZeStig :emacs: :nix: :rust: :archlinux: @[email protected] · 2026-05-14 · 12:52 UTC

How do I troubleshoot a black screen on #KDE #Plasma? I am able to launch apps just fine, but:
- the desktop is all black (no start menu, global menu etc, and the desktop background isn't getting rendered)
- the window decorations are HUGE.

Is there any easy way to debug this? I had initially suspected this was due to my using #Stylix on stock #archlinux, but no, disabling it didn't change a thing (apart from the theme).

I do have custom desktop effects and window decorations installed; so I suspect that might be the issue. How do I disable them if systemsettings does not launch (for some weird reason)?

It's definitely not a kernel issue (I've tried on the stock Arch kernel and the CachyOS kernel), and it certainly isn't a graphics issue (stock mesa etc, and I don't have a dGPU on this machine).

Will send screenshots in a while. Any and all help appreciated! Thanks in advance. Toots and boosts welcome.

#linux #kde #plasma #askfedi #linuxhelp #linuxquestions

#kde #plasma #stylix #archlinux #askfedi #linux

ZeStig :emacs: :nix: :rust: :archlinux: @[email protected] · 2026-05-14 · 12:52 UTC

How do I troubleshoot a black screen on #KDE #Plasma? I am able to launch apps just fine, but:
- the desktop is all black (no start menu, global menu etc, and the desktop background isn't getting rendered)
- the window decorations are HUGE.

Is there any easy way to debug this? I had initially suspected this was due to my using #Stylix on stock #archlinux, but no, disabling it didn't change a thing (apart from the theme).

I do have custom desktop effects and window decorations installed; so I suspect that might be the issue. How do I disable them if systemsettings does not launch (for some weird reason)?

It's definitely not a kernel issue (I've tried on the stock Arch kernel and the CachyOS kernel), and it certainly isn't a graphics issue (stock mesa etc, and I don't have a dGPU on this machine).

Will send screenshots in a while. Any and all help appreciated! Thanks in advance. Toots and boosts welcome.

#linux #kde #plasma #askfedi #linuxhelp #linuxquestions

#linuxquestions #linuxhelp #linux #askfedi #archlinux #stylix

ZeStig :emacs: :nix: :rust: :archlinux: @[email protected] · 2026-05-14 · 12:52 UTC

How do I troubleshoot a black screen on #KDE #Plasma? I am able to launch apps just fine, but:
- the desktop is all black (no start menu, global menu etc, and the desktop background isn't getting rendered)
- the window decorations are HUGE.

Is there any easy way to debug this? I had initially suspected this was due to my using #Stylix on stock #archlinux, but no, disabling it didn't change a thing (apart from the theme).

I do have custom desktop effects and window decorations installed; so I suspect that might be the issue. How do I disable them if systemsettings does not launch (for some weird reason)?

It's definitely not a kernel issue (I've tried on the stock Arch kernel and the CachyOS kernel), and it certainly isn't a graphics issue (stock mesa etc, and I don't have a dGPU on this machine).

Will send screenshots in a while. Any and all help appreciated! Thanks in advance. Toots and boosts welcome.

#linux #kde #plasma #askfedi #linuxhelp #linuxquestions

#kde #plasma #stylix #archlinux #askfedi #linux

Špoljarević :redhat: :ansible: @[email protected] · 2026-05-14 · 11:15 UTC

For those who are wondering why my projects are so silent, I moved to my own Forgejo instance at https://code.spoljarevic.sh

From there I will continue my journey.

Tho I didn’t have lots of time to work on them, and they will slightly change.

Due to an upcoming server migration, new tools and my personal life, everything got slow.

But I already finished some open issues on the Ansible Project and will go into testing after my vacation.

It shall be finished this month or I’m no longer trusting myself.

Next month will be nothing since I’m gonna focus on chess openings and the month after I’m working on a Knowledge Base (fork of hyprland wiki)

After that, everything should go normal again.

#ansible #archlinux #project #forgejo #server #ServerMigration #wiki #KnowledgeBase #chess #chessopenings

#ansible #archlinux #project #forgejo #server #servermigration

Špoljarević :redhat: :ansible: @[email protected] · 2026-05-14 · 11:15 UTC

For those who are wondering why my projects are so silent, I moved to my own Forgejo instance at https://code.spoljarevic.sh

From there I will continue my journey.

Tho I didn’t have lots of time to work on them, and they will slightly change.

Due to an upcoming server migration, new tools and my personal life, everything got slow.

But I already finished some open issues on the Ansible Project and will go into testing after my vacation.

It shall be finished this month or I’m no longer trusting myself.

Next month will be nothing since I’m gonna focus on chess openings and the month after I’m working on a Knowledge Base (fork of hyprland wiki)

After that, everything should go normal again.

#ansible #archlinux #project #forgejo #server #ServerMigration #wiki #KnowledgeBase #chess #chessopenings

#chessopenings #chess #knowledgebase #wiki #servermigration #server

Špoljarević :redhat: :ansible: @spoljarevic · 2026-05-14 · 11:15 UTC

For those who are wondering why my projects are so silent, I moved to my own Forgejo instance at https://code.spoljarevic.sh

From there I will continue my journey.

Tho I didn’t have lots of time to work on them, and they will slightly change.

Due to an upcoming server migration, new tools and my personal life, everything got slow.

But I already finished some open issues on the Ansible Project and will go into testing after my vacation.

It shall be finished this month or I’m no longer trusting myself.

Next month will be nothing since I’m gonna focus on chess openings and the month after I’m working on a Knowledge Base (fork of hyprland wiki)

After that, everything should go normal again.

#ansible #archlinux #project #forgejo #server #ServerMigration #wiki #KnowledgeBase #chess #chessopenings

#ansible #archlinux #project #forgejo #server #servermigration

Ein Philosophiker @[email protected] · 2026-05-14 · 11:07 UTC

How can I take part in update a package in #Aur? #ArchLinux

#aur #archlinux

Ein Philosophiker @[email protected] · 2026-05-14 · 11:07 UTC

How can I take part in update a package in #Aur? #ArchLinux

#aur #archlinux

Ein Philosophiker @[email protected] · 2026-05-14 · 11:07 UTC

How can I take part in update a package in #Aur? #ArchLinux

#aur #archlinux

Ein Philosophiker @[email protected] · 2026-05-14 · 11:07 UTC

How can I take part in update a package in #Aur? #ArchLinux

#aur #archlinux

UndeadLeech @UndeadLeech · 2026-05-13 · 23:16 UTC

Apparently the Fairphone 5 microphone not working with automatic wireguard call profile switching was just a race condition with q6voiced:

https://gitlab.postmarketos.org/postmarketOS/q6voiced/-/work_items/3

This means I now have calls working at a usable level with great quality on the Fairphone 5.

Since ArchLinux users must love the bleeding edge, I'm shipping this as default with my packages, so it should work out of the box going forward:

https://git.sr.ht/~undeadleech/catacomb_packages/commit/3150b440e4050516b180a17ce2b8244198e5c301

#linuxmobile #archlinux #fairphone #catacomb

Really Lazy Bear :debian_logo: @[email protected] · 2026-05-13 · 21:14 UTC

Yeah I'm just vibin now on osu. Controller's got a few problems but I'll fix it in the future.

Kinda sleepy but anyways should I play Ultimate Custom Night and torture myself?

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton #osugame #osulazer #osu

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource

Really Lazy Bear :debian_logo: @[email protected] · 2026-05-13 · 21:14 UTC

Yeah I'm just vibin now on osu. Controller's got a few problems but I'll fix it in the future.

Kinda sleepy but anyways should I play Ultimate Custom Night and torture myself?

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton #osugame #osulazer #osu

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource

Really Lazy Bear :debian_logo: @[email protected] · 2026-05-13 · 21:14 UTC

Yeah I'm just vibin now on osu. Controller's got a few problems but I'll fix it in the future.

Kinda sleepy but anyways should I play Ultimate Custom Night and torture myself?

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton #osugame #osulazer #osu

#osu #osulazer #osugame #proton #valve #steam

Really Lazy Bear :debian_logo: @[email protected] · 2026-05-13 · 21:14 UTC

Yeah I'm just vibin now on osu. Controller's got a few problems but I'll fix it in the future.

Kinda sleepy but anyways should I play Ultimate Custom Night and torture myself?

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton #osugame #osulazer #osu

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource

Trey Fordham @[email protected] · 2026-05-13 · 19:51 UTC

People keep asking me how I knew to stock up on CPUs and RAM before the AI hardware choke. Anyway, I think I'm just about done configuring my optiplex #archlinux router. It's currently operating behind my modem, but it is responsible for the whole wireless network right now. And wow! Just using the hagezi configuration file for blocking is great. next, I think I'll recreate my "stream hole" configuration for the rest of the TVs in the house. #selfhosted

#archlinux #selfhosted

Linux Easy @[email protected] · 2026-05-13 · 16:53 UTC

Archcraft Prime aggiorna la sua ISO con nuovi compositor Wayland, Hyprland v55 e miglioramenti per Openbox, bspwm e i3wm. #Linux #ArchLinux #Archcraft #Wayland #Hyprland #Openbox #Bspwm #i3wm

https://www.linuxeasy.org/archcraft-prime-aggiorna-iso-hyprland-v55-wayland/?utm_source=mastodon&utm_medium=jetpack_social

#linux #archlinux #archcraft #wayland #hyprland #openbox

Kevin @[email protected] · 2026-05-12 · 21:56 UTC

When two Hetzner servers died at the same time

On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

Several hours went into hypotheses that turned out to be wrong:

The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd

#archlinux #bootfailure #debugging #devops #dhcp #dropbear

Kevin @[email protected] · 2026-05-12 · 21:56 UTC

When two Hetzner servers died at the same time

On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

Several hours went into hypotheses that turned out to be wrong:

The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd

#archlinux #bootfailure #debugging #devops #dhcp #dropbear

Kevin @[email protected] · 2026-05-12 · 21:56 UTC

When two Hetzner servers died at the same time

On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

Several hours went into hypotheses that turned out to be wrong:

The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd

#archlinux #bootfailure #debugging #devops #dhcp #dropbear

Kevin @[email protected] · 2026-05-12 · 21:56 UTC

When two Hetzner servers died at the same time

On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

Several hours went into hypotheses that turned out to be wrong:

The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd

#systemdnetworkd #sysadmin #serveroutage #pythoncli #postmortem #pacman

Kevin @[email protected] · 2026-05-12 · 21:56 UTC

When two Hetzner servers died at the same time

On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

Several hours went into hypotheses that turned out to be wrong:

The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

What it almost certainly was

Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"

One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

Why it matters for anyone running this stack

If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

Tooling

While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

→ github.com/kevinveenbirkenbach/hetzner-arch-luks

Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd

#archlinux #bootfailure #debugging #devops #dhcp #dropbear

(BOT) Really Lazy Bear's bot @[email protected] · 2026-05-12 · 20:45 UTC

Revisiting the guys from back then in Ultimate Custom Night.

#youtube: https://youtube.com/@ReallyLazyBear
#twitch: https://www.twitch.tv/reallylazybear

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton

#proton #valve #steam #cachyos #archlinux #horror

(BOT) Really Lazy Bear's bot @[email protected] · 2026-05-12 · 20:45 UTC

Revisiting the guys from back then in Ultimate Custom Night.

#youtube: https://youtube.com/@ReallyLazyBear
#twitch: https://www.twitch.tv/reallylazybear

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton

#youtube #twitch #linuxgaming #ultimatecustomnight #ucn #fnaf

unfa🇺🇦 @[email protected] · 2026-05-12 · 20:30 UTC

Today I learned #BespokeSynth on #ArchLinux freezes when trying to save a project. After I force terminate the application, the save file dialog window appears. On #MXLinux I was able to get my music done though, so that's good!

#bespokesynth #archlinux #mxlinux

(BOT) Really Lazy Bear's bot @[email protected] · 2026-05-12 · 19:44 UTC

Revisiting the guys from back then in Ultimate Custom Night.

#youtube: https://youtube.com/@ReallyLazyBear
#twitch: https://www.twitch.tv/reallylazybear

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton

#proton #valve #steam #cachyos #archlinux #horror

(BOT) Really Lazy Bear's bot @[email protected] · 2026-05-12 · 19:44 UTC

Revisiting the guys from back then in Ultimate Custom Night.

#youtube: https://youtube.com/@ReallyLazyBear
#twitch: https://www.twitch.tv/reallylazybear

#linuxgaming #ultimatecustomnight #ucn #fnaf #fivenightsatfreddys #opensource #indiegame #horror #archlinux #cachyos #steam #valve #proton

#youtube #twitch #linuxgaming #ultimatecustomnight #ucn #fnaf

Arch Linux CZ @[email protected] · 2026-05-12 · 19:14 UTC

Arch Tip týdne: pacman jako profík

📦 pacman umí mnohem víc než jen instalovat balíčky.
Tady jsou dotazy, které ti pomůžou pochopit, co máš v systému.

═══════════════════════════════════════════
🔍 ZÁKLADNÍ DOTAZY (-Q = QUERY)
═══════════════════════════════════════════

výpis všech nainstalovaných balíčků

pacman -Q

kolik balíčků mám?

pacman -Q | wc -l

jen explicitně nainstalované (bez závislostí)

pacman -Qe

jen závislosti (instalované jako "side effect")

pacman -Qd

balíčky z AUR / mimo oficiální repo

pacman -Qm

balíčky JEN z oficiálních repo

pacman -Qn

osiřelé balíčky (závislosti, které už nikdo nepotřebuje)

pacman -Qtdq

odstranění osiřelých

sudo pacman -Rns $(pacman -Qtdq)

═══════════════════════════════════════════
📋 pacman -Qi → INFO O BALÍČKU
═══════════════════════════════════════════

detailní info

pacman -Qi firefox

co uvidíš:

- verze, popis, URL

- licence

- závislosti (Depends On)

- co na něm závisí (Required By)

- volitelné závislosti (Optional Deps)

- velikost, datum instalace

- důvod instalace (Explicitly installed / Dependency)

rozšířené info (přidá Backup Files = konfiguráky chráněné při update)

pacman -Qii firefox

info z repa (i pro nenainstalované)

pacman -Si firefox

jen velikost balíčku

pacman -Qi firefox | grep "Installed Size"

řazení balíčků podle velikosti (potřebuje: pacman -S expac)

expac -H M "%m\t%n" | sort -h | tail -20

═══════════════════════════════════════════
📂 pacman -Ql → SEZNAM SOUBORŮ
═══════════════════════════════════════════

co všechno balíček nainstaloval

pacman -Ql firefox

jen binárky

pacman -Ql firefox | grep '/bin/'

konfigurační soubory balíčku (vše v /etc)

pacman -Ql firefox | grep '^firefox /etc'

počet souborů v balíčku

pacman -Ql firefox | wc -l

soubory nenainstalovaného balíčku (z repa)

nejdřív aktualizuj file database:

sudo pacman -Fy
pacman -Fl firefox

═══════════════════════════════════════════
🔎 pacman -Qo → KDO VLASTNÍ TENTO SOUBOR?
═══════════════════════════════════════════

který balíček nainstaloval daný soubor?

pacman -Qo /usr/bin/firefox

výstup:

/usr/bin/firefox is owned by firefox 123.0-1

najdi balíček podle příkazu v PATH

pacman -Qo $(which nvim)

najdi balíček, kterému patří soubor – i když ho nemáš nainstalovaný

sudo pacman -Fy
pacman -F /usr/bin/htop

═══════════════════════════════════════════
🛠️ pacnew / pacsave – CHRÁNĚNÉ KONFIGURÁKY
═══════════════════════════════════════════

Backup Files = soubory označené v balíčku jako "neredisej moji verzi"

Když přijde update a soubor jsi upravil, pacman vytvoří .pacnew

(nová verze z balíčku) a tvoji verzi nechá být.

seznam backup files konkrétního balíčku

pacman -Qii pacman | awk '/^Backup Files/,/^$/'

najdi všechny .pacnew/.pacsave v systému (čekající na merge)

sudo find /etc ( -name ".pacnew" -o -name ".pacsave" )

interaktivní merge (z pacman-contrib)

sudo pacman -S pacman-contrib
sudo pacdiff

═══════════════════════════════════════════
🌳 pactree → STROM ZÁVISLOSTÍ
═══════════════════════════════════════════

instalace (pokud nemáš)

sudo pacman -S pacman-contrib

co všechno balíček potřebuje (závislosti)

pactree firefox

obrácený strom – kdo závisí NA tomto balíčku

pactree -r glibc

omezení hloubky

pactree -d 1 firefox

jen názvy, bez ozdob

pactree -u firefox

grafický výstup (potřebuje: pacman -S graphviz)

pactree --graph firefox | dot -Tpng > firefox-deps.png

═══════════════════════════════════════════
🔧 BONUS – DALŠÍ UŽITEČNÉ DOTAZY
═══════════════════════════════════════════

hledání balíčku v repech

pacman -Ss firefox

hledání v nainstalovaných

pacman -Qs firefox

poslední full upgrade systému

grep "starting full system upgrade" /var/log/pacman.log | tail -5

co se naposled instalovalo/aktualizovalo

tail -50 /var/log/pacman.log

pretty výpis posledních akcí (expac)

expac --timefmt='%Y-%m-%d %T' '%l\t%n' | sort -r | head -20

změnit důvod instalace (z dependency na explicit)

sudo pacman -D --asexplicit firefox

označit jako závislost (půjde uklidit přes -Rns)

sudo pacman -D --asdeps nějaký-balíček

═══════════════════════════════════════════
💡 TYPICKÝ WORKFLOW
═══════════════════════════════════════════

1. co mi žere místo?

expac -H M "%m\t%n" | sort -h | tail -20

2. co dělá tenhle balíček?

pacman -Qi <balíček>

3. můžu ho smazat? Co na něm závisí?

pactree -r <balíček>

4. odkud se vzal tenhle soubor?

pacman -Qo /cesta/k/souboru

5. úklid

sudo pacman -Rns $(pacman -Qtdq)
sudo paccache -rk2

#archlinux #pacman #linux #cli

Arch Linux CZ @[email protected] · 2026-05-12 · 19:14 UTC

Arch Tip týdne: pacman jako profík

📦 pacman umí mnohem víc než jen instalovat balíčky.
Tady jsou dotazy, které ti pomůžou pochopit, co máš v systému.

═══════════════════════════════════════════
🔍 ZÁKLADNÍ DOTAZY (-Q = QUERY)
═══════════════════════════════════════════

výpis všech nainstalovaných balíčků

pacman -Q

kolik balíčků mám?

pacman -Q | wc -l

jen explicitně nainstalované (bez závislostí)

pacman -Qe

jen závislosti (instalované jako "side effect")

pacman -Qd

balíčky z AUR / mimo oficiální repo

pacman -Qm

balíčky JEN z oficiálních repo

pacman -Qn

osiřelé balíčky (závislosti, které už nikdo nepotřebuje)

pacman -Qtdq

odstranění osiřelých

sudo pacman -Rns $(pacman -Qtdq)

═══════════════════════════════════════════
📋 pacman -Qi → INFO O BALÍČKU
═══════════════════════════════════════════

detailní info

pacman -Qi firefox

co uvidíš:

- verze, popis, URL

- licence

- závislosti (Depends On)

- co na něm závisí (Required By)

- volitelné závislosti (Optional Deps)

- velikost, datum instalace

- důvod instalace (Explicitly installed / Dependency)

rozšířené info (přidá Backup Files = konfiguráky chráněné při update)

pacman -Qii firefox

info z repa (i pro nenainstalované)

pacman -Si firefox

jen velikost balíčku

pacman -Qi firefox | grep "Installed Size"

řazení balíčků podle velikosti (potřebuje: pacman -S expac)

expac -H M "%m\t%n" | sort -h | tail -20

═══════════════════════════════════════════
📂 pacman -Ql → SEZNAM SOUBORŮ
═══════════════════════════════════════════

co všechno balíček nainstaloval

pacman -Ql firefox

jen binárky

pacman -Ql firefox | grep '/bin/'

konfigurační soubory balíčku (vše v /etc)

pacman -Ql firefox | grep '^firefox /etc'

počet souborů v balíčku

pacman -Ql firefox | wc -l

soubory nenainstalovaného balíčku (z repa)

nejdřív aktualizuj file database:

sudo pacman -Fy
pacman -Fl firefox

═══════════════════════════════════════════
🔎 pacman -Qo → KDO VLASTNÍ TENTO SOUBOR?
═══════════════════════════════════════════

který balíček nainstaloval daný soubor?

pacman -Qo /usr/bin/firefox

výstup:

/usr/bin/firefox is owned by firefox 123.0-1

najdi balíček podle příkazu v PATH

pacman -Qo $(which nvim)

najdi balíček, kterému patří soubor – i když ho nemáš nainstalovaný

sudo pacman -Fy
pacman -F /usr/bin/htop

═══════════════════════════════════════════
🛠️ pacnew / pacsave – CHRÁNĚNÉ KONFIGURÁKY
═══════════════════════════════════════════

Backup Files = soubory označené v balíčku jako "neredisej moji verzi"

Když přijde update a soubor jsi upravil, pacman vytvoří .pacnew

(nová verze z balíčku) a tvoji verzi nechá být.

seznam backup files konkrétního balíčku

pacman -Qii pacman | awk '/^Backup Files/,/^$/'

najdi všechny .pacnew/.pacsave v systému (čekající na merge)

sudo find /etc ( -name ".pacnew" -o -name ".pacsave" )

interaktivní merge (z pacman-contrib)

sudo pacman -S pacman-contrib
sudo pacdiff

═══════════════════════════════════════════
🌳 pactree → STROM ZÁVISLOSTÍ
═══════════════════════════════════════════

instalace (pokud nemáš)

sudo pacman -S pacman-contrib

co všechno balíček potřebuje (závislosti)

pactree firefox

obrácený strom – kdo závisí NA tomto balíčku

pactree -r glibc

omezení hloubky

pactree -d 1 firefox

jen názvy, bez ozdob

pactree -u firefox

grafický výstup (potřebuje: pacman -S graphviz)

pactree --graph firefox | dot -Tpng > firefox-deps.png

═══════════════════════════════════════════
🔧 BONUS – DALŠÍ UŽITEČNÉ DOTAZY
═══════════════════════════════════════════

hledání balíčku v repech

pacman -Ss firefox

hledání v nainstalovaných

pacman -Qs firefox

poslední full upgrade systému

grep "starting full system upgrade" /var/log/pacman.log | tail -5

co se naposled instalovalo/aktualizovalo

tail -50 /var/log/pacman.log

pretty výpis posledních akcí (expac)

expac --timefmt='%Y-%m-%d %T' '%l\t%n' | sort -r | head -20

změnit důvod instalace (z dependency na explicit)

sudo pacman -D --asexplicit firefox

označit jako závislost (půjde uklidit přes -Rns)

sudo pacman -D --asdeps nějaký-balíček

═══════════════════════════════════════════
💡 TYPICKÝ WORKFLOW
═══════════════════════════════════════════

1. co mi žere místo?

expac -H M "%m\t%n" | sort -h | tail -20

2. co dělá tenhle balíček?

pacman -Qi <balíček>

3. můžu ho smazat? Co na něm závisí?

pactree -r <balíček>

4. odkud se vzal tenhle soubor?

pacman -Qo /cesta/k/souboru

5. úklid

sudo pacman -Rns $(pacman -Qtdq)
sudo paccache -rk2

#archlinux #pacman #linux #cli

Arch Linux CZ @[email protected] · 2026-05-12 · 19:14 UTC

Arch Tip týdne: pacman jako profík

📦 pacman umí mnohem víc než jen instalovat balíčky.
Tady jsou dotazy, které ti pomůžou pochopit, co máš v systému.

═══════════════════════════════════════════
🔍 ZÁKLADNÍ DOTAZY (-Q = QUERY)
═══════════════════════════════════════════

výpis všech nainstalovaných balíčků

pacman -Q

kolik balíčků mám?

pacman -Q | wc -l

jen explicitně nainstalované (bez závislostí)

pacman -Qe

jen závislosti (instalované jako "side effect")

pacman -Qd

balíčky z AUR / mimo oficiální repo

pacman -Qm

balíčky JEN z oficiálních repo

pacman -Qn

osiřelé balíčky (závislosti, které už nikdo nepotřebuje)

pacman -Qtdq

odstranění osiřelých

sudo pacman -Rns $(pacman -Qtdq)

═══════════════════════════════════════════
📋 pacman -Qi → INFO O BALÍČKU
═══════════════════════════════════════════

detailní info

pacman -Qi firefox

co uvidíš:

- verze, popis, URL

- licence

- závislosti (Depends On)

- co na něm závisí (Required By)

- volitelné závislosti (Optional Deps)

- velikost, datum instalace

- důvod instalace (Explicitly installed / Dependency)

rozšířené info (přidá Backup Files = konfiguráky chráněné při update)

pacman -Qii firefox

info z repa (i pro nenainstalované)

pacman -Si firefox

jen velikost balíčku

pacman -Qi firefox | grep "Installed Size"

řazení balíčků podle velikosti (potřebuje: pacman -S expac)

expac -H M "%m\t%n" | sort -h | tail -20

═══════════════════════════════════════════
📂 pacman -Ql → SEZNAM SOUBORŮ
═══════════════════════════════════════════

co všechno balíček nainstaloval

pacman -Ql firefox

jen binárky

pacman -Ql firefox | grep '/bin/'

konfigurační soubory balíčku (vše v /etc)

pacman -Ql firefox | grep '^firefox /etc'

počet souborů v balíčku

pacman -Ql firefox | wc -l

soubory nenainstalovaného balíčku (z repa)

nejdřív aktualizuj file database:

sudo pacman -Fy
pacman -Fl firefox

═══════════════════════════════════════════
🔎 pacman -Qo → KDO VLASTNÍ TENTO SOUBOR?
═══════════════════════════════════════════

který balíček nainstaloval daný soubor?

pacman -Qo /usr/bin/firefox

výstup:

/usr/bin/firefox is owned by firefox 123.0-1

najdi balíček podle příkazu v PATH

pacman -Qo $(which nvim)

najdi balíček, kterému patří soubor – i když ho nemáš nainstalovaný

sudo pacman -Fy
pacman -F /usr/bin/htop

═══════════════════════════════════════════
🛠️ pacnew / pacsave – CHRÁNĚNÉ KONFIGURÁKY
═══════════════════════════════════════════

Backup Files = soubory označené v balíčku jako "neredisej moji verzi"

Když přijde update a soubor jsi upravil, pacman vytvoří .pacnew

(nová verze z balíčku) a tvoji verzi nechá být.

seznam backup files konkrétního balíčku

pacman -Qii pacman | awk '/^Backup Files/,/^$/'

najdi všechny .pacnew/.pacsave v systému (čekající na merge)

sudo find /etc ( -name ".pacnew" -o -name ".pacsave" )

interaktivní merge (z pacman-contrib)

sudo pacman -S pacman-contrib
sudo pacdiff

═══════════════════════════════════════════
🌳 pactree → STROM ZÁVISLOSTÍ
═══════════════════════════════════════════

instalace (pokud nemáš)

sudo pacman -S pacman-contrib

co všechno balíček potřebuje (závislosti)

pactree firefox

obrácený strom – kdo závisí NA tomto balíčku

pactree -r glibc

omezení hloubky

pactree -d 1 firefox

jen názvy, bez ozdob

pactree -u firefox

grafický výstup (potřebuje: pacman -S graphviz)

pactree --graph firefox | dot -Tpng > firefox-deps.png

═══════════════════════════════════════════
🔧 BONUS – DALŠÍ UŽITEČNÉ DOTAZY
═══════════════════════════════════════════

hledání balíčku v repech

pacman -Ss firefox

hledání v nainstalovaných

pacman -Qs firefox

poslední full upgrade systému

grep "starting full system upgrade" /var/log/pacman.log | tail -5

co se naposled instalovalo/aktualizovalo

tail -50 /var/log/pacman.log

pretty výpis posledních akcí (expac)

expac --timefmt='%Y-%m-%d %T' '%l\t%n' | sort -r | head -20

změnit důvod instalace (z dependency na explicit)

sudo pacman -D --asexplicit firefox

označit jako závislost (půjde uklidit přes -Rns)

sudo pacman -D --asdeps nějaký-balíček

═══════════════════════════════════════════
💡 TYPICKÝ WORKFLOW
═══════════════════════════════════════════

1. co mi žere místo?

expac -H M "%m\t%n" | sort -h | tail -20

2. co dělá tenhle balíček?

pacman -Qi <balíček>

3. můžu ho smazat? Co na něm závisí?

pactree -r <balíček>

4. odkud se vzal tenhle soubor?

pacman -Qo /cesta/k/souboru

5. úklid

sudo pacman -Rns $(pacman -Qtdq)
sudo paccache -rk2

#archlinux #pacman #linux #cli