home.social

#oopsie — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #oopsie, aggregated by home.social.

  1. Mastodon Incident Report / Root cause analysis:

    Earlier today, users experienced timeouts with Search, Hashtags, and Autocomplete.

    Root Cause: Our setup separates the Mastodon frontend VPS (Hetzner) from backend services (for example Elasticsearch) via an OPNSense firewall. Suricata (our IPS) triggered a false-positive on internal traffic and aggressively blocked the VPS IP, severing the connection to the search database.

    Resolution: We identified the false-positive, added the frontend IP to the whitelist, and traffic immediately normalized. Everything is back to green!

    #mastodon #mastoadmin #burningboard #elasticsearch #firewall #opnsense #suricata #oopsie

  2. Mastodon Incident Report / Root cause analysis:

    Earlier today, users experienced timeouts with Search, Hashtags, and Autocomplete.

    Root Cause: Our setup separates the Mastodon frontend VPS (Hetzner) from backend services (for example Elasticsearch) via an OPNSense firewall. Suricata (our IPS) triggered a false-positive on internal traffic and aggressively blocked the VPS IP, severing the connection to the search database.

    Resolution: We identified the false-positive, added the frontend IP to the whitelist, and traffic immediately normalized. Everything is back to green!

    #mastodon #mastoadmin #burningboard #elasticsearch #firewall #opnsense #suricata #oopsie

  3. Mastodon Incident Report / Root cause analysis:

    Earlier today, users experienced timeouts with Search, Hashtags, and Autocomplete.

    Root Cause: Our setup separates the Mastodon frontend VPS (Hetzner) from backend services (for example Elasticsearch) via an OPNSense firewall. Suricata (our IPS) triggered a false-positive on internal traffic and aggressively blocked the VPS IP, severing the connection to the search database.

    Resolution: We identified the false-positive, added the frontend IP to the whitelist, and traffic immediately normalized. Everything is back to green!

    #mastodon #mastoadmin #burningboard #elasticsearch #firewall #opnsense #suricata #oopsie

  4. Mastodon Incident Report / Root cause analysis:

    Earlier today, users experienced timeouts with Search, Hashtags, and Autocomplete.

    Root Cause: Our setup separates the Mastodon frontend VPS (Hetzner) from backend services (for example Elasticsearch) via an OPNSense firewall. Suricata (our IPS) triggered a false-positive on internal traffic and aggressively blocked the VPS IP, severing the connection to the search database.

    Resolution: We identified the false-positive, added the frontend IP to the whitelist, and traffic immediately normalized. Everything is back to green!

    #mastodon #mastoadmin #burningboard #elasticsearch #firewall #opnsense #suricata #oopsie

  5. Mastodon Incident Report / Root cause analysis:

    Earlier today, users experienced timeouts with Search, Hashtags, and Autocomplete.

    Root Cause: Our setup separates the Mastodon frontend VPS (Hetzner) from backend services (for example Elasticsearch) via an OPNSense firewall. Suricata (our IPS) triggered a false-positive on internal traffic and aggressively blocked the VPS IP, severing the connection to the search database.

    Resolution: We identified the false-positive, added the frontend IP to the whitelist, and traffic immediately normalized. Everything is back to green!

    #mastodon #mastoadmin #burningboard #elasticsearch #firewall #opnsense #suricata #oopsie

  6. At least the Dark OP’s of the Cheney Bush years were somewhat ennobled by honest brokers and decent intentions following the trauma of 911.

    The Trump team just doesn’t give a merde about deporting a migrant protected by asylum. #Oopsie He was persecuted and disappeared like a Deep State plot. I can’t abide agreeing with Joe Rogan but his call-out of the Black Hole problem was approaching sane. Mind twisting events. Props to Dems Resistance #corybooker #pramilajayapal

    youtu.be/eQjYci6S6x4?si=t0NMQa