home.social

#postmortem — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #postmortem, aggregated by home.social.

  1. Баги, которые нас воспитали: инженерные истории с Go Loto

    Каждый разработчик помнит тот самый момент, когда система, которая только что работала идеально, вдруг начинает вести себя так, будто сошла с ума. Когда дашборд в Grafana показывает что-то страшное, а ты стоишь перед ним с кружкой остывшего кофе и не понимаешь, с чего начать. На нашем мероприятии Avito Go Loto разработчики поделились своим опытом без прикрас. О блоате в полтора терабайта, о девяти инстансах, которые передрались за один звонок, о бэкенд-разработчице, которая в пятницу вечером открыла чужой фронтовый проект, о нагрузочных тестах за несколько месяцев до большой рекламной кампании, и о транзакции, которую забыли закоммитить тоже в пятницу вечером. Спойлер: все выжили. Но стали другими людьми.

    habr.com/ru/companies/avito/ar

    #go #bloat #vacuum #микросервисы #распределенные_системы #нагрузочное_тестирование #postmortem

  2. Баги, которые нас воспитали: инженерные истории с Go Loto

    Каждый разработчик помнит тот самый момент, когда система, которая только что работала идеально, вдруг начинает вести себя так, будто сошла с ума. Когда дашборд в Grafana показывает что-то страшное, а ты стоишь перед ним с кружкой остывшего кофе и не понимаешь, с чего начать. На нашем мероприятии Avito Go Loto разработчики поделились своим опытом без прикрас. О блоате в полтора терабайта, о девяти инстансах, которые передрались за один звонок, о бэкенд-разработчице, которая в пятницу вечером открыла чужой фронтовый проект, о нагрузочных тестах за несколько месяцев до большой рекламной кампании, и о транзакции, которую забыли закоммитить тоже в пятницу вечером. Спойлер: все выжили. Но стали другими людьми.

    habr.com/ru/companies/avito/ar

    #go #bloat #vacuum #микросервисы #распределенные_системы #нагрузочное_тестирование #postmortem

  3. Баги, которые нас воспитали: инженерные истории с Go Loto

    Каждый разработчик помнит тот самый момент, когда система, которая только что работала идеально, вдруг начинает вести себя так, будто сошла с ума. Когда дашборд в Grafana показывает что-то страшное, а ты стоишь перед ним с кружкой остывшего кофе и не понимаешь, с чего начать. На нашем мероприятии Avito Go Loto разработчики поделились своим опытом без прикрас. О блоате в полтора терабайта, о девяти инстансах, которые передрались за один звонок, о бэкенд-разработчице, которая в пятницу вечером открыла чужой фронтовый проект, о нагрузочных тестах за несколько месяцев до большой рекламной кампании, и о транзакции, которую забыли закоммитить тоже в пятницу вечером. Спойлер: все выжили. Но стали другими людьми.

    habr.com/ru/companies/avito/ar

    #go #bloat #vacuum #микросервисы #распределенные_системы #нагрузочное_тестирование #postmortem

  4. Баги, которые нас воспитали: инженерные истории с Go Loto

    Каждый разработчик помнит тот самый момент, когда система, которая только что работала идеально, вдруг начинает вести себя так, будто сошла с ума. Когда дашборд в Grafana показывает что-то страшное, а ты стоишь перед ним с кружкой остывшего кофе и не понимаешь, с чего начать. На нашем мероприятии Avito Go Loto разработчики поделились своим опытом без прикрас. О блоате в полтора терабайта, о девяти инстансах, которые передрались за один звонок, о бэкенд-разработчице, которая в пятницу вечером открыла чужой фронтовый проект, о нагрузочных тестах за несколько месяцев до большой рекламной кампании, и о транзакции, которую забыли закоммитить тоже в пятницу вечером. Спойлер: все выжили. Но стали другими людьми.

    habr.com/ru/companies/avito/ar

    #go #bloat #vacuum #микросервисы #распределенные_системы #нагрузочное_тестирование #postmortem

  5. Как auto-update n8n нашёл мину которая лежала 8 месяцев в node_modules

    20 мая в 06:01:55 МСК Watchtower по расписанию проверил 14 контейнеров на нашем VPS, нашёл 5 обновлений и пересоздал. Среди обновлённых - n8n, который крутит production-вебхуки нескольких проектов студии (в том числе SaaS Подробнее

    habr.com/ru/articles/1037434/

    #n8n #docker #watchtower #monitoring #incident #postmortem #selfhosted #observability #crashloop #devops

  6. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  7. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  8. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  9. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  10. When two Hetzner servers died at the same time

    On May 12, 2026, two of my Arch Linux + LUKS servers at Hetzner became unreachable at the same moment. Both had been running for 4+ months without issue. Both had received the same pacman -Syyu the day before, but had stayed on the old kernel until the morning the websites stopped responding. I rebooted — SSH never came back. nmap -Pn -p 22 showed filtered from anywhere. No ping. No banner. The Hetzner Robot panel insisted the hardware was fine.

    Several hours went into hypotheses that turned out to be wrong:

    • The encryptssh initcpio hook referencing a /usr/lib/initcpio/udev/11-dm-initramfs.rules file that no longer exists. Real bug, no boot impact — the initramfs rebuilds anyway.
    • PermitRootLogin no in sshd_config. Real misconfiguration, fixed it, didn’t help. A refusing sshd shows closed, not filtered.
    • Predictable interface-naming drift after the systemd 260 upgrade. Patched the .network config to match by MAC. Useful hardening; not the cause.
    • Stale GRUB stage1 + core.img in the MBR. Arch never re-runs grub-install after a grub package upgrade. Refreshed it. Still filtered.
    • Kernel 7.0.5 regression. Downgraded to 6.18.3, the kernel that had run for 4 months. Still filtered. So the kernel itself wasn’t it either.

    The clue was in the persistent journal: a single recorded boot from December 31 to May 12 10:13 UTC, and absolutely nothing after. Every reboot since the upgrade was failing before systemd-journald could flush to disk — so the failure had to be in the initramfs, before the root filesystem was even mounted.

    What it almost certainly was

    Hetzner Dedicated servers configure the initramfs network with ip=dhcp on the kernel command line. That depends on Hetzner’s DHCP server replying to whatever request format the current kernel sends. Somewhere between kernel 6.18 / iproute2 6.18 and kernel 7.0 / iproute2 7.0, the request format changed enough that Hetzner’s DHCP stopped responding. Effects:

    • Old kernel at runtime kept the interface already configured (Phase A — 32 hours of healthy operation after the package upgrade).
    • New kernel cold-boots, hits DHCP, never gets an IP, dropbear cannot listen, port 22 stays filtered.

    Hetzner’s own documentation has been quietly moving away from ip=dhcp toward static IPv4 in the kernel command line. The fix is exactly that:

    GRUB_CMDLINE_LINUX="cryptdevice=/dev/md1:cryptroot ip=A.B.C.D::GATEWAY:255.255.255.255:hostname:eth0:none"
    

    One line in /etc/default/grub, grub-mkconfig, reboot. No more dependency on Hetzner’s DHCP responding to whatever your current kernel sends.

    Why it matters for anyone running this stack

    If you run Arch on Hetzner Dedicated with full-disk encryption and remote unlock via dropbear, the ip=dhcp shipped by installimage is a latent bug. It can keep working for years and then break overnight, on every machine you have, after a routine pacman -Syyu. The static-IP version is what Hetzner now recommends and removes the entire dependency.

    Tooling

    While debugging, I turned the whole rescue / chroot / diagnose / fix workflow into a Python CLI (hal) — including hal fix static-ip, which derives the static cmdline directly from your existing systemd-networkd .network file:

    github.com/kevinveenbirkenbach/hetzner-arch-luks

    Single command, idempotent, reversible (the original /etc/default/grub is backed up to .hal-backup). If you’re on this stack, switch to static IP before the next kernel upgrade catches you.

    #ArchLinux #bootFailure #debugging #DevOps #DHCP #Dropbear #fullDiskEncryption #GRUB #Hetzner #initramfs #kernelUpgrade #Linux #LUKS #mkinitcpio #pacman #postmortem #PythonCLI #serverOutage #sysadmin #systemdNetworkd
  11. L’âme mécanique est en ligne.
    Pas de format « nouvelle »… Des épisodes à suivre !
    À lire sur mon espace Panodyssey : chk.me/uRlZVks
    #amemecanique #haroldcath #IA #robotique #postmortem #jeuneauteur #1erroman

  12. A year ago I released my turn-based horror strategy about underground oil drilling — and it quietly taught me more than I expected.

    So I wrote a postmortem. Here are a few things I'd tell myself at the start.

    Full postmortem here 👇

    reddit.com/r/GameDevelopment/c

    Anoxia didn't become a big hit. But it's slowly recouping its budget, and we're already deep into our next game — Bonereader, a Balatro-like deck-builder set in a shamanic Purgatory.

    #gamedev #indiegame #postmortem

  13. Раскатил учётные записи с помощью Ansible playbook и... чёрт побери! В плейбуке пользователям задан /bin/bash, а на серверах с FreeBSD его нет. Разумеется, зайти уже не получалось по SSH и консольно.

    Выручило то, что у хостера есть cloud-init. Добавил в запуск
    runcmd:
    - chsh -s /bin/sh root
    и после перезагрузки смог зайти.

    Дальше уже в плейбуке поменял порядок стадий, чтобы сначала ставился bash и другие пакеты, а потом уже по пользователям побежали настройки.

    #devops #freebsd #postMortem

  14. Раскатил учётные записи с помощью Ansible playbook и... чёрт побери! В плейбуке пользователям задан /bin/bash, а на серверах с FreeBSD его нет. Разумеется, зайти уже не получалось по SSH и консольно.

    Выручило то, что у хостера есть cloud-init. Добавил в запуск
    runcmd:
    - chsh -s /bin/sh root
    и после перезагрузки смог зайти.

    Дальше уже в плейбуке поменял порядок стадий, чтобы сначала ставился bash и другие пакеты, а потом уже по пользователям побежали настройки.

    #devops #freebsd #postMortem

  15. Раскатил учётные записи с помощью Ansible playbook и... чёрт побери! В плейбуке пользователям задан /bin/bash, а на серверах с FreeBSD его нет. Разумеется, зайти уже не получалось по SSH и консольно.

    Выручило то, что у хостера есть cloud-init. Добавил в запуск
    runcmd:
    - chsh -s /bin/sh root
    и после перезагрузки смог зайти.

    Дальше уже в плейбуке поменял порядок стадий, чтобы сначала ставился bash и другие пакеты, а потом уже по пользователям побежали настройки.

    #devops #freebsd #postMortem

  16. Раскатил учётные записи с помощью Ansible playbook и... чёрт побери! В плейбуке пользователям задан /bin/bash, а на серверах с FreeBSD его нет. Разумеется, зайти уже не получалось по SSH и консольно.

    Выручило то, что у хостера есть cloud-init. Добавил в запуск
    runcmd:
    - chsh -s /bin/sh root
    и после перезагрузки смог зайти.

    Дальше уже в плейбуке поменял порядок стадий, чтобы сначала ставился bash и другие пакеты, а потом уже по пользователям побежали настройки.

    #devops #freebsd #postMortem

  17. Раскатил учётные записи с помощью Ansible playbook и... чёрт побери! В плейбуке пользователям задан /bin/bash, а на серверах с FreeBSD его нет. Разумеется, зайти уже не получалось по SSH и консольно.

    Выручило то, что у хостера есть cloud-init. Добавил в запуск
    runcmd:
    - chsh -s /bin/sh root
    и после перезагрузки смог зайти.

    Дальше уже в плейбуке поменял порядок стадий, чтобы сначала ставился bash и другие пакеты, а потом уже по пользователям побежали настройки.

    #devops #freebsd #postMortem

  18. 🚀 Oh, look! Yet another grandiloquent prize fest that ends with a whimper and a "blameless #postmortem." 🙄 Congratulations, QDay Prize organizers! You've achieved the remarkable feat of discovering that nobody cares. 🎉
    algassert.com/post/2601 #grandiloquentprize #QDayPrize #nobodycares #prizefest #HackerNews #ngated

  19. 🚀 Oh, look! Yet another grandiloquent prize fest that ends with a whimper and a "blameless #postmortem." 🙄 Congratulations, QDay Prize organizers! You've achieved the remarkable feat of discovering that nobody cares. 🎉
    algassert.com/post/2601 #grandiloquentprize #QDayPrize #nobodycares #prizefest #HackerNews #ngated

  20. 🚀 Oh, look! Yet another grandiloquent prize fest that ends with a whimper and a "blameless #postmortem." 🙄 Congratulations, QDay Prize organizers! You've achieved the remarkable feat of discovering that nobody cares. 🎉
    algassert.com/post/2601 #grandiloquentprize #QDayPrize #nobodycares #prizefest #HackerNews #ngated

  21. 🚀 Oh, look! Yet another grandiloquent prize fest that ends with a whimper and a "blameless #postmortem." 🙄 Congratulations, QDay Prize organizers! You've achieved the remarkable feat of discovering that nobody cares. 🎉
    algassert.com/post/2601 #grandiloquentprize #QDayPrize #nobodycares #prizefest #HackerNews #ngated

  22. 🚀 Oh, look! Yet another grandiloquent prize fest that ends with a whimper and a "blameless #postmortem." 🙄 Congratulations, QDay Prize organizers! You've achieved the remarkable feat of discovering that nobody cares. 🎉
    algassert.com/post/2601 #grandiloquentprize #QDayPrize #nobodycares #prizefest #HackerNews #ngated

  23. Matrix.org - Post-mortem of the September 2 outage

    "Matrix, the open protocol for secure decentralised communications"

    Link: matrix.org/blog/2025/10/post-m

    #linkdump #blogpost #matrix #postmortem #story

  24. Matrix.org - Post-mortem of the September 2 outage

    "Matrix, the open protocol for secure decentralised communications"

    Link: matrix.org/blog/2025/10/post-m

  25. Matrix.org - Post-mortem of the September 2 outage

    "Matrix, the open protocol for secure decentralised communications"

    Link: matrix.org/blog/2025/10/post-m

    #linkdump #blogpost #matrix #postmortem #story

  26. Matrix.org - Post-mortem of the September 2 outage

    "Matrix, the open protocol for secure decentralised communications"

    Link: matrix.org/blog/2025/10/post-m

    #linkdump #blogpost #matrix #postmortem #story

  27. Matrix.org - Post-mortem of the September 2 outage

    "Matrix, the open protocol for secure decentralised communications"

    Link: matrix.org/blog/2025/10/post-m

    #linkdump #blogpost #matrix #postmortem #story

  28. Matrix.org - Post-mortem of the September 2 outage

    "Matrix, the open protocol for secure decentralised communications"

    Link: matrix.org/blog/2025/10/post-m

    #linkdump #blogpost #matrix #postmortem #story