Search
1000 results for “downgrade”
-
Launching the upgrade from Fedora 43 to Fedora 44 on an up to date machine today, and it *downgrades* the kernel from 7.0.6 to 7.0.4 🤔
-
Sad to see it go, because of how vibrant it was at the time. Downloaded my history, and scrolling back through time was emotional, and I'm glad I was able to get that archive for posterity. But, like so many, I just can't anymore. #riptwitter
-
Sad to see it go, because of how vibrant it was at the time. Downloaded my history, and scrolling back through time was emotional, and I'm glad I was able to get that archive for posterity. But, like so many, I just can't anymore. #riptwitter
-
the end game in civ 7 is definitely really underwhelming tbh. the non-military victories are like "slot resources in your towns and then wait a while" and "dig up 15 artifacts". major downgrade from the win conditions of civ 6. and the future era tech seems to be completely absent from the game
-
the end game in civ 7 is definitely really underwhelming tbh. the non-military victories are like "slot resources in your towns and then wait a while" and "dig up 15 artifacts". major downgrade from the win conditions of civ 6. and the future era tech seems to be completely absent from the game
-
the end game in civ 7 is definitely really underwhelming tbh. the non-military victories are like "slot resources in your towns and then wait a while" and "dig up 15 artifacts". major downgrade from the win conditions of civ 6. and the future era tech seems to be completely absent from the game
-
the end game in civ 7 is definitely really underwhelming tbh. the non-military victories are like "slot resources in your towns and then wait a while" and "dig up 15 artifacts". major downgrade from the win conditions of civ 6. and the future era tech seems to be completely absent from the game
-
the end game in civ 7 is definitely really underwhelming tbh. the non-military victories are like "slot resources in your towns and then wait a while" and "dig up 15 artifacts". major downgrade from the win conditions of civ 6. and the future era tech seems to be completely absent from the game
-
:fediverse: #Pew: #FreshPotOfCoffee - Check ✅️
:fediverse: #Pew: #Antonov-#AN225: #Downloaded - Check ✅️
:fediverse: #Pew: #Playlist #Confirmed - Check ✅️And, #QuoteToots; #StillQuiteCool...
With #Bullets; #LotsAndLots of #Bullets... And, #Spoons...!
🧙⚔️🤖🐺🤖⚔️🧙 | 🛫☕️️🍪🦄🍪☕️🛬
-
https://www.europesays.com/ch/63225/ ABB India’s Profit Dips Despite Strong Orders, Prompting Analyst Downgrade #ABB #ABBIndia #ABBIndiaMargin #ABBIndiaOrderInflow #ABBIndiaQ1Earnings #ABBIndiaStockRating #ABBIndiaTargetPrice #AbbStock #DataCenterSector #Electrification #IndianManufacturing #IndustrialCapex #PrabhudasLilladher
-
USGS CoNED (TopoBathy) WebMap Viewer & (Open) Data Downloader
--
https://topotools.cr.usgs.gov/topobathy_viewer/ <-- shared #USGS Viewer webmap & download selector
--
https://www.usgs.gov/coastal-changes-and-impacts/coned <-- shared USGS CoNED overview/entry page
--
[I used to shore dive in the Straits Of Juan de Fucca, Washington State side, and Crescent Lake - so I chose that area as a CoNED example to explore; good memories, including of the 18 Wheeler Burger with pie & coffee in Joyce, WA on drizzly days]
#GIS #spatial #mapping #remotesensing #model #modeling #CoastalNationalElevationDatabase #CoNED #elevation #topography #bathymetry #topobathy #opendata #openscience #fedscience #publicgood #usecase #naturalworld #coastal #coast #depth #hydrospatial #hydrography #download #shoreline #inundation #sedimenttransport #sealevelrise #SLR #stormsurge #models #modeling #extremeweather #tsunami #impact #risk #hazard #assessment #climatechange #geomorphology #geomorphometry #Restoration, #redevelopment #protection #cliff #flood #flooding #webmap
@USGS -
USGS CoNED (TopoBathy) WebMap Viewer & (Open) Data Downloader
--
https://topotools.cr.usgs.gov/topobathy_viewer/ <-- shared #USGS Viewer webmap & download selector
--
https://www.usgs.gov/coastal-changes-and-impacts/coned <-- shared USGS CoNED overview/entry page
--
[I used to shore dive in the Straits Of Juan de Fucca, Washington State side, and Crescent Lake - so I chose that area as a CoNED example to explore; good memories, including of the 18 Wheeler Burger with pie & coffee in Joyce, WA on drizzly days]
#GIS #spatial #mapping #remotesensing #model #modeling #CoastalNationalElevationDatabase #CoNED #elevation #topography #bathymetry #topobathy #opendata #openscience #fedscience #publicgood #usecase #naturalworld #coastal #coast #depth #hydrospatial #hydrography #download #shoreline #inundation #sedimenttransport #sealevelrise #SLR #stormsurge #models #modeling #extremeweather #tsunami #impact #risk #hazard #assessment #climatechange #geomorphology #geomorphometry #Restoration, #redevelopment #protection #cliff #flood #flooding #webmap
@USGS -
USGS CoNED (TopoBathy) WebMap Viewer & (Open) Data Downloader
--
https://topotools.cr.usgs.gov/topobathy_viewer/ <-- shared #USGS Viewer webmap & download selector
--
https://www.usgs.gov/coastal-changes-and-impacts/coned <-- shared USGS CoNED overview/entry page
--
[I used to shore dive in the Straits Of Juan de Fucca, Washington State side, and Crescent Lake - so I chose that area as a CoNED example to explore; good memories, including of the 18 Wheeler Burger with pie & coffee in Joyce, WA on drizzly days]
#GIS #spatial #mapping #remotesensing #model #modeling #CoastalNationalElevationDatabase #CoNED #elevation #topography #bathymetry #topobathy #opendata #openscience #fedscience #publicgood #usecase #naturalworld #coastal #coast #depth #hydrospatial #hydrography #download #shoreline #inundation #sedimenttransport #sealevelrise #SLR #stormsurge #models #modeling #extremeweather #tsunami #impact #risk #hazard #assessment #climatechange #geomorphology #geomorphometry #Restoration, #redevelopment #protection #cliff #flood #flooding #webmap
@USGS -
USGS CoNED (TopoBathy) WebMap Viewer & (Open) Data Downloader
--
https://topotools.cr.usgs.gov/topobathy_viewer/ <-- shared #USGS Viewer webmap & download selector
--
https://www.usgs.gov/coastal-changes-and-impacts/coned <-- shared USGS CoNED overview/entry page
--
[I used to shore dive in the Straits Of Juan de Fucca, Washington State side, and Crescent Lake - so I chose that area as a CoNED example to explore; good memories, including of the 18 Wheeler Burger with pie & coffee in Joyce, WA on drizzly days]
#GIS #spatial #mapping #remotesensing #model #modeling #CoastalNationalElevationDatabase #CoNED #elevation #topography #bathymetry #topobathy #opendata #openscience #fedscience #publicgood #usecase #naturalworld #coastal #coast #depth #hydrospatial #hydrography #download #shoreline #inundation #sedimenttransport #sealevelrise #SLR #stormsurge #models #modeling #extremeweather #tsunami #impact #risk #hazard #assessment #climatechange #geomorphology #geomorphometry #Restoration, #redevelopment #protection #cliff #flood #flooding #webmap
@USGS -
USGS CoNED (TopoBathy) WebMap Viewer & (Open) Data Downloader
--
https://topotools.cr.usgs.gov/topobathy_viewer/ <-- shared #USGS Viewer webmap & download selector
--
https://www.usgs.gov/coastal-changes-and-impacts/coned <-- shared USGS CoNED overview/entry page
--
[I used to shore dive in the Straits Of Juan de Fucca, Washington State side, and Crescent Lake - so I chose that area as a CoNED example to explore; good memories, including of the 18 Wheeler Burger with pie & coffee in Joyce, WA on drizzly days]
#GIS #spatial #mapping #remotesensing #model #modeling #CoastalNationalElevationDatabase #CoNED #elevation #topography #bathymetry #topobathy #opendata #openscience #fedscience #publicgood #usecase #naturalworld #coastal #coast #depth #hydrospatial #hydrography #download #shoreline #inundation #sedimenttransport #sealevelrise #SLR #stormsurge #models #modeling #extremeweather #tsunami #impact #risk #hazard #assessment #climatechange #geomorphology #geomorphometry #Restoration, #redevelopment #protection #cliff #flood #flooding #webmap
@USGS -
Downloaded tonight's episode of "Curse Of Oak Island". S13E21. <sigh>
This episode is a 2-hour special. And the start of the episode is different. Crossing my fingers this indicates they finally found something after ~230 years of searching for treasure on this mysterious island. #OakIsland
-
Downloaded tonight's episode of "Curse Of Oak Island". S13E21. <sigh>
This episode is a 2-hour special. And the start of the episode is different. Crossing my fingers this indicates they finally found something after ~230 years of searching for treasure on this mysterious island. #OakIsland
-
Downloaded tonight's episode of "Curse Of Oak Island". S13E21. <sigh>
This episode is a 2-hour special. And the start of the episode is different. Crossing my fingers this indicates they finally found something after ~230 years of searching for treasure on this mysterious island. #OakIsland
-
Downloaded tonight's episode of "Curse Of Oak Island". S13E21. <sigh>
This episode is a 2-hour special. And the start of the episode is different. Crossing my fingers this indicates they finally found something after ~230 years of searching for treasure on this mysterious island. #OakIsland
-
Downloaded tonight's episode of "Curse Of Oak Island". S13E21. <sigh>
This episode is a 2-hour special. And the start of the episode is different. Crossing my fingers this indicates they finally found something after ~230 years of searching for treasure on this mysterious island. #OakIsland
-
In the capitalist global system, every purchase is an action the determines how much power we give as consumers to big corporations.
And this current system is not working for people or the planet as we know it, but only amplifying the polycrisis (*insert* mourning in future tense).
I made a radical switch from iPhone to Android after almost 8 years or so. I downloaded all my 155 GBs of pictures from icloud and unsubscribed to the cloud storage (for life), everything self-hosted—soon.
I switched to not a very normal Android device, but surely a fairer one. Fairphone is one of the most repairable phones out there, designed in the Netherlands. Promoting users to repair their own devices actively, a phone that you own not the other way around. The two visible screws (Torx T5) on the back even encourages user to open it themselves.
On the other hand, even a new EU mandate requires every smartphone sold in Europe including Apple devices to feature a battery that can be replaced by the user without specialist tools, without voiding a warranty, and without sending the device to a manufacturer approved service center.
The designers and the team at Fairphone even went so far as to incorporate the map of the Democratic Republic of the Congo, sourcing conflict-free tin and tantalum from mines. FYI: The same mines that produced almost 50% of uranium oxide (i.e. 8.59 million kg) for the U.S. Manhattan Project.
Read the full 2025 Fairphone's Impact Report here: https://lnkd.in/ejWvdxiB
A small step towards more circular electronics. ☀
#fasttech #saynotofasttech #circular #sustaianble #electronics #fairphone #EuropeanUnion #Minerals #Sourcing #Fast #Tech
-
In the capitalist global system, every purchase is an action the determines how much power we give as consumers to big corporations.
And this current system is not working for people or the planet as we know it, but only amplifying the polycrisis (*insert* mourning in future tense).
I made a radical switch from iPhone to Android after almost 8 years or so. I downloaded all my 155 GBs of pictures from icloud and unsubscribed to the cloud storage (for life), everything self-hosted—soon.
I switched to not a very normal Android device, but surely a fairer one. Fairphone is one of the most repairable phones out there, designed in the Netherlands. Promoting users to repair their own devices actively, a phone that you own not the other way around. The two visible screws (Torx T5) on the back even encourages user to open it themselves.
On the other hand, even a new EU mandate requires every smartphone sold in Europe including Apple devices to feature a battery that can be replaced by the user without specialist tools, without voiding a warranty, and without sending the device to a manufacturer approved service center.
The designers and the team at Fairphone even went so far as to incorporate the map of the Democratic Republic of the Congo, sourcing conflict-free tin and tantalum from mines. FYI: The same mines that produced almost 50% of uranium oxide (i.e. 8.59 million kg) for the U.S. Manhattan Project.
Read the full 2025 Fairphone's Impact Report here: https://lnkd.in/ejWvdxiB
A small step towards more circular electronics. ☀
#fasttech #saynotofasttech #circular #sustaianble #electronics #fairphone #EuropeanUnion #Minerals #Sourcing #Fast #Tech
-
In the capitalist global system, every purchase is an action the determines how much power we give as consumers to big corporations.
And this current system is not working for people or the planet as we know it, but only amplifying the polycrisis (*insert* mourning in future tense).
I made a radical switch from iPhone to Android after almost 8 years or so. I downloaded all my 155 GBs of pictures from icloud and unsubscribed to the cloud storage (for life), everything self-hosted—soon.
I switched to not a very normal Android device, but surely a fairer one. Fairphone is one of the most repairable phones out there, designed in the Netherlands. Promoting users to repair their own devices actively, a phone that you own not the other way around. The two visible screws (Torx T5) on the back even encourages user to open it themselves.
On the other hand, even a new EU mandate requires every smartphone sold in Europe including Apple devices to feature a battery that can be replaced by the user without specialist tools, without voiding a warranty, and without sending the device to a manufacturer approved service center.
The designers and the team at Fairphone even went so far as to incorporate the map of the Democratic Republic of the Congo, sourcing conflict-free tin and tantalum from mines. FYI: The same mines that produced almost 50% of uranium oxide (i.e. 8.59 million kg) for the U.S. Manhattan Project.
Read the full 2025 Fairphone's Impact Report here: https://lnkd.in/ejWvdxiB
A small step towards more circular electronics. ☀
#fasttech #saynotofasttech #circular #sustaianble #electronics #fairphone #EuropeanUnion #Minerals #Sourcing #Fast #Tech
-
In the capitalist global system, every purchase is an action the determines how much power we give as consumers to big corporations.
And this current system is not working for people or the planet as we know it, but only amplifying the polycrisis (*insert* mourning in future tense).
I made a radical switch from iPhone to Android after almost 8 years or so. I downloaded all my 155 GBs of pictures from icloud and unsubscribed to the cloud storage (for life), everything self-hosted—soon.
I switched to not a very normal Android device, but surely a fairer one. Fairphone is one of the most repairable phones out there, designed in the Netherlands. Promoting users to repair their own devices actively, a phone that you own not the other way around. The two visible screws (Torx T5) on the back even encourages user to open it themselves.
On the other hand, even a new EU mandate requires every smartphone sold in Europe including Apple devices to feature a battery that can be replaced by the user without specialist tools, without voiding a warranty, and without sending the device to a manufacturer approved service center.
The designers and the team at Fairphone even went so far as to incorporate the map of the Democratic Republic of the Congo, sourcing conflict-free tin and tantalum from mines. FYI: The same mines that produced almost 50% of uranium oxide (i.e. 8.59 million kg) for the U.S. Manhattan Project.
Read the full 2025 Fairphone's Impact Report here: https://lnkd.in/ejWvdxiB
A small step towards more circular electronics. ☀
#fasttech #saynotofasttech #circular #sustaianble #electronics #fairphone #EuropeanUnion #Minerals #Sourcing #Fast #Tech
-
In the capitalist global system, every purchase is an action the determines how much power we give as consumers to big corporations.
And this current system is not working for people or the planet as we know it, but only amplifying the polycrisis (*insert* mourning in future tense).
I made a radical switch from iPhone to Android after almost 8 years or so. I downloaded all my 155 GBs of pictures from icloud and unsubscribed to the cloud storage (for life), everything self-hosted—soon.
I switched to not a very normal Android device, but surely a fairer one. Fairphone is one of the most repairable phones out there, designed in the Netherlands. Promoting users to repair their own devices actively, a phone that you own not the other way around. The two visible screws (Torx T5) on the back even encourages user to open it themselves.
On the other hand, even a new EU mandate requires every smartphone sold in Europe including Apple devices to feature a battery that can be replaced by the user without specialist tools, without voiding a warranty, and without sending the device to a manufacturer approved service center.
The designers and the team at Fairphone even went so far as to incorporate the map of the Democratic Republic of the Congo, sourcing conflict-free tin and tantalum from mines. FYI: The same mines that produced almost 50% of uranium oxide (i.e. 8.59 million kg) for the U.S. Manhattan Project.
Read the full 2025 Fairphone's Impact Report here: https://lnkd.in/ejWvdxiB
A small step towards more circular electronics. ☀
#fasttech #saynotofasttech #circular #sustaianble #electronics #fairphone #EuropeanUnion #Minerals #Sourcing #Fast #Tech
-
CW: 🔞NSFW🔞
👠 😍 The Shitty Chronicles (video)☝️ :heart_fire:
This is a collection of what I consider to be my best video edits. Even though I already posted them all separately, some of them received a bunch of tweaks in this version.
🧲As it's too BIG to be uploaded here, I'll share a link: https://motherless.com/509F2CF
(If you want to download it from there, I can recommend this browser extension: Video Downloader Professional.)
Peace!
-
When two Hetzner servers died at the same time
On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same
pacman -Syyuthe day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back.nmap -Pn -p 22showedfilteredfrom anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.Several hours went into hypotheses that turned out to be wrong:
- The
encryptsshinitcpio hook referencing a/usr/lib/initcpio/udev/11-dm-initramfs.rulesfile that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway. PermitRootLogin noinsshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd showsclosed, notfiltered.- Predictable interface-naming drift after the systemd 260 upgrade. Patched the
.networkconfig to match by MAC. Useful hardening; not the cause. - Stale GRUB stage1 +
core.imgin the MBR. Arch never re-runsgrub-installafter agrubpackage upgrade. Refreshed it. Still filtered. - Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.
The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before
systemd-journaldcould flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.What it almost certainly was
Hetzner Dedicated servers configure the initramfs network with
ip=dhcpon the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:- Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
- New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays
filtered.
Hetzner’s own documentation has been quietly moving away from
ip=dhcptoward static IPv4 in the kernel command line. The fix is exactly that:GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"One line in
/etc/default/grub,grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.Why it matters for anyone running this stack
If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the
ip=dhcpshipped byinstallimageis a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routinepacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.Tooling
While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (
hal) — includinghal fix static-ip, which derives the static cmdline directly from your existingsystemd-networkd.networkfile:→ github.com/kevinveenbirkenbach/hetzner-arch-luks
Single command, idempotent, reversible (the original
#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd/etc/default/grubis backed up to.hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you. - The
-
When two Hetzner servers died at the same time
On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same
pacman -Syyuthe day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back.nmap -Pn -p 22showedfilteredfrom anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.Several hours went into hypotheses that turned out to be wrong:
- The
encryptsshinitcpio hook referencing a/usr/lib/initcpio/udev/11-dm-initramfs.rulesfile that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway. PermitRootLogin noinsshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd showsclosed, notfiltered.- Predictable interface-naming drift after the systemd 260 upgrade. Patched the
.networkconfig to match by MAC. Useful hardening; not the cause. - Stale GRUB stage1 +
core.imgin the MBR. Arch never re-runsgrub-installafter agrubpackage upgrade. Refreshed it. Still filtered. - Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.
The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before
systemd-journaldcould flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.What it almost certainly was
Hetzner Dedicated servers configure the initramfs network with
ip=dhcpon the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:- Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
- New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays
filtered.
Hetzner’s own documentation has been quietly moving away from
ip=dhcptoward static IPv4 in the kernel command line. The fix is exactly that:GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"One line in
/etc/default/grub,grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.Why it matters for anyone running this stack
If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the
ip=dhcpshipped byinstallimageis a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routinepacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.Tooling
While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (
hal) — includinghal fix static-ip, which derives the static cmdline directly from your existingsystemd-networkd.networkfile:→ github.com/kevinveenbirkenbach/hetzner-arch-luks
Single command, idempotent, reversible (the original
#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd/etc/default/grubis backed up to.hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you. - The
-
When two Hetzner servers died at the same time
On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same
pacman -Syyuthe day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back.nmap -Pn -p 22showedfilteredfrom anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.Several hours went into hypotheses that turned out to be wrong:
- The
encryptsshinitcpio hook referencing a/usr/lib/initcpio/udev/11-dm-initramfs.rulesfile that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway. PermitRootLogin noinsshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd showsclosed, notfiltered.- Predictable interface-naming drift after the systemd 260 upgrade. Patched the
.networkconfig to match by MAC. Useful hardening; not the cause. - Stale GRUB stage1 +
core.imgin the MBR. Arch never re-runsgrub-installafter agrubpackage upgrade. Refreshed it. Still filtered. - Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.
The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before
systemd-journaldcould flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.What it almost certainly was
Hetzner Dedicated servers configure the initramfs network with
ip=dhcpon the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:- Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
- New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays
filtered.
Hetzner’s own documentation has been quietly moving away from
ip=dhcptoward static IPv4 in the kernel command line. The fix is exactly that:GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"One line in
/etc/default/grub,grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.Why it matters for anyone running this stack
If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the
ip=dhcpshipped byinstallimageis a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routinepacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.Tooling
While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (
hal) — includinghal fix static-ip, which derives the static cmdline directly from your existingsystemd-networkd.networkfile:→ github.com/kevinveenbirkenbach/hetzner-arch-luks
Single command, idempotent, reversible (the original
#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd/etc/default/grubis backed up to.hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you. - The
-
When two Hetzner servers died at the same time
On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same
pacman -Syyuthe day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back.nmap -Pn -p 22showedfilteredfrom anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.Several hours went into hypotheses that turned out to be wrong:
- The
encryptsshinitcpio hook referencing a/usr/lib/initcpio/udev/11-dm-initramfs.rulesfile that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway. PermitRootLogin noinsshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd showsclosed, notfiltered.- Predictable interface-naming drift after the systemd 260 upgrade. Patched the
.networkconfig to match by MAC. Useful hardening; not the cause. - Stale GRUB stage1 +
core.imgin the MBR. Arch never re-runsgrub-installafter agrubpackage upgrade. Refreshed it. Still filtered. - Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.
The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before
systemd-journaldcould flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.What it almost certainly was
Hetzner Dedicated servers configure the initramfs network with
ip=dhcpon the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:- Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
- New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays
filtered.
Hetzner’s own documentation has been quietly moving away from
ip=dhcptoward static IPv4 in the kernel command line. The fix is exactly that:GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"One line in
/etc/default/grub,grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.Why it matters for anyone running this stack
If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the
ip=dhcpshipped byinstallimageis a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routinepacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.Tooling
While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (
hal) — includinghal fix static-ip, which derives the static cmdline directly from your existingsystemd-networkd.networkfile:→ github.com/kevinveenbirkenbach/hetzner-arch-luks
Single command, idempotent, reversible (the original
#ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd/etc/default/grubis backed up to.hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you. - The