home.social

#debugging — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #debugging, aggregated by home.social.

  1. Every team learns differently — that’s why KDAB’s in-company #training programs are built around your goals. Customize the curriculum, schedule, and location to fit your needs. Explore topics like #QtDev, #Cpp, #QML, #3D, #Debugging, and more.

    Learn more: training.kdab.com/in-company-t
    training.kdab.com/in-company-t

  2. Every team learns differently — that’s why KDAB’s in-company #training programs are built around your goals. Customize the curriculum, schedule, and location to fit your needs. Explore topics like #QtDev, #Cpp, #QML, #3D, #Debugging, and more.

    Learn more: training.kdab.com/in-company-t
    training.kdab.com/in-company-t

  3. Every team learns differently — that’s why KDAB’s in-company #training programs are built around your goals. Customize the curriculum, schedule, and location to fit your needs. Explore topics like #QtDev, #Cpp, #QML, #3D, #Debugging, and more.

    Learn more: training.kdab.com/in-company-t
    training.kdab.com/in-company-t

  4. Every team learns differently — that’s why KDAB’s in-company #training programs are built around your goals. Customize the curriculum, schedule, and location to fit your needs. Explore topics like #QtDev, #Cpp, #QML, #3D, #Debugging, and more.

    Learn more: training.kdab.com/in-company-t
    training.kdab.com/in-company-t

  5. Every team learns differently — that’s why KDAB’s in-company programs are built around your goals. Customize the curriculum, schedule, and location to fit your needs. Explore topics like , , , , , and more.

    Learn more: training.kdab.com/in-company-t
    training.kdab.com/in-company-t

  6. 🔍 Spark + Elasticsearch Debugging 🧵

    Building a cybersecurity analytics platform. Hit 2 blockers:

    ❌ JAR path mismatch → Fixed absolute path
    ❌ No data nodes (single-node Docker ES) → Added es.nodes.wan.only=true

    ✅ Result: 89 records loaded. Working pipeline!

    Lesson: Verify JAR paths + disable node discovery for single-node ES.

    #PySpark #Elasticsearch #DataEngineering #CyberSecurity #Debugging

  7. 🔍 Spark + Elasticsearch Debugging 🧵

    Building a cybersecurity analytics platform. Hit 2 blockers:

    ❌ JAR path mismatch → Fixed absolute path
    ❌ No data nodes (single-node Docker ES) → Added es.nodes.wan.only=true

    ✅ Result: 89 records loaded. Working pipeline!

    Lesson: Verify JAR paths + disable node discovery for single-node ES.

    #PySpark #Elasticsearch #DataEngineering #CyberSecurity #Debugging

  8. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  9. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  10. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  11. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  12. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  13. While probing a GPIO pin on my oscilloscope, I noticed it toggled between 2.4V and 3.3V. Using another reference pin, toggling at exactly the same time swinged correctly between 0V and 3.3V

    I think I might have damaged a pin on my #rp2350 :( #debugging #signals #embedded #pico2

  14. While probing a GPIO pin on my oscilloscope, I noticed it toggled between 2.4V and 3.3V. Using another reference pin, toggling at exactly the same time swinged correctly between 0V and 3.3V

    I think I might have damaged a pin on my #rp2350 :( #debugging #signals #embedded #pico2

  15. Uso #ExtremeProgramming da quando facevo #debugging all'università, oltre 20 anni fa.

    Una cosa non cambia: scrivere codice è facile, definirne tutto il comportamento no.

    🎤 A #PyConIT 2026 porterò un #workshop su #Python 🐍 e #TDD:
    dalle variabili al primo test, per chiarire cosa il software debba fare prima di implementarlo.

    2026.pycon.it/it/event/primi-p

    #TalksAndWorkshops #python #pytest #BestPractices

  16. Uso #ExtremeProgramming da quando facevo #debugging all'università, oltre 20 anni fa.

    Una cosa non cambia: scrivere codice è facile, definirne tutto il comportamento no.

    🎤 A #PyConIT 2026 porterò un #workshop su #Python 🐍 e #TDD:
    dalle variabili al primo test, per chiarire cosa il software debba fare prima di implementarlo.

    2026.pycon.it/it/event/primi-p

    #TalksAndWorkshops #python #pytest #BestPractices

  17. 🎮🤖 Apparently, PS3 emulator devs are drowning in AI-generated pull requests from wannabe vibecoders who think #debugging is a new TikTok dance. It seems the real "game" is getting human contributors who know that 'AI slop' isn't the latest fast-food trend. 🍔💻
    kotaku.com/playstation-3-emula #PS3Emulator #AIgenerated #PullRequests #GameDevelopment #HackerNews #ngated

  18. 🎮🤖 Apparently, PS3 emulator devs are drowning in AI-generated pull requests from wannabe vibecoders who think #debugging is a new TikTok dance. It seems the real "game" is getting human contributors who know that 'AI slop' isn't the latest fast-food trend. 🍔💻
    kotaku.com/playstation-3-emula #PS3Emulator #AIgenerated #PullRequests #GameDevelopment #HackerNews #ngated

  19. 🎮🤖 Apparently, PS3 emulator devs are drowning in AI-generated pull requests from wannabe vibecoders who think #debugging is a new TikTok dance. It seems the real "game" is getting human contributors who know that 'AI slop' isn't the latest fast-food trend. 🍔💻
    kotaku.com/playstation-3-emula #PS3Emulator #AIgenerated #PullRequests #GameDevelopment #HackerNews #ngated

  20. 🎮🤖 Apparently, PS3 emulator devs are drowning in AI-generated pull requests from wannabe vibecoders who think #debugging is a new TikTok dance. It seems the real "game" is getting human contributors who know that 'AI slop' isn't the latest fast-food trend. 🍔💻
    kotaku.com/playstation-3-emula #PS3Emulator #AIgenerated #PullRequests #GameDevelopment #HackerNews #ngated

  21. 🎮🤖 Apparently, PS3 emulator devs are drowning in AI-generated pull requests from wannabe vibecoders who think #debugging is a new TikTok dance. It seems the real "game" is getting human contributors who know that 'AI slop' isn't the latest fast-food trend. 🍔💻
    kotaku.com/playstation-3-emula #PS3Emulator #AIgenerated #PullRequests #GameDevelopment #HackerNews #ngated