#hachyinfra — Public Fediverse posts

This morning in #hachyderm infrastructure - BACKLOG.

We had a **great** response to our call for volunteers, so I'm slowly working through and reaching out as I recover from AWS #reinvent.

If you reached out to me and I haven't replied, bump me again. The Masto DM UX makes it a little hard to keep track of things! 😅

I'm also working on the infra backlog in general to make sure we have a set of "good first issue" items for folks to hop on as they onboard.

#hachyinfra #sre #devops

this morning in #hachyderm infrastructure:

our post about the resilience plans for infra got a lot of rotation across the fedi. 🎉

we've had several folks reach out and offer to volunteer, which made me look at the backlog and realize it's not super consumable.

spending some time prepping good work for new folks. 🔧

have i gone overboard with the tags? yes.

note: our repo is currently private, but we are looking at ways of extracting key things to share!

#hachyinfra #github #devops

This morning in #hachyderm infrastructure, we finally posted our risk analysis and possible action items to strengthen Hachyderm's position against changing laws in the USA.

https://hachyderm.io/@hachyderm/113572384384017395

tl;dr - we're in pretty good shape, but we can do more.

Read through it, ask questions, poke at ideas, and if you have some free time on your hands, join the crew! There's so much work to do.

$ :terminal_blinker:

#hachyinfra #devops #sre #mastoadmin

hey, #hachyderm - did you see when we did the restart? probably not - it was pretty fast!

near term future, we'll add an additional host to serve mastodon-web and you *really* won't see it!

#hachyinfra

applying good progressive exposure practices in ansible has been fun.

#hachyderm's infra footprint is still grokkable at human-scale, so we can do things like prompts for attention -- something little to break out of potential doldrums of repetitive actions.

next will probably be implementing a concept like "rings" or "update domains" in our harness script, e.g.:

run X against all of our edge nodes, but roll it across the fleet by risk (lowest traffic --> highest traffic)

#hachyinfra

this morning in #hachyderm infra, doing our first prod roll outs of some targeted #ansible runbooks!

#hachyinfra

howdy #hachyderm & fediverse - I'm just about to close out the draft of our infrastructure threat analysis, but one thing I'd like to include is reading material for folks who aren't familiar with building a personal opsec plan/model.

what are your favorite articles, talks, resources, etc. for defining your personal online security threat model and building a plan to mitigate the risks you've identified?

#security #opsec #hachyinfra

this morning in #hachyderm infrastructure - picking up where I left off last night: applying #ansible roles to existing hachyderm infrastructure, starting with our dev instance, hachyderm.wtf.

also - i yanked the #tailscale to prometheus.yml code and open sourced it here: https://github.com/hachyderm/ansible-role-prometheus-tailscale-discovery. Note that it may not work standalone: use this as a way to see our thinking. I'll work on cleaning it up to be a proper role that we use. /cc @tebriel

#hachyinfra

and more @tailscale -- automating our #prometheus scrape configs based on tags attached to machines. such clean, very template 🐕

#hachyderm #hachyinfra

this (early) morning in #hachyderm infrastructure:

finished up the base #ansible common role that does prep and basic hardening of new nodes.

also:

- sets up our basic "daily driver" user
- installs @tailscale and connects it to our tailnet
- sets up log shipping to loki via promtail
- hardens sshd and enables fail2ban
- sets up basic node metrics with prometheus-node-exporter
- works on both deb12 & arch
- set up the rad motd we will use forever, thx @nova :blobfoxheartcute:

#hachyinfra

@tailscale it would be SUPER NEAT if you could wildcard allowed tags for OAuth clients. (or made the allowable tags editable, but that might be harder/not possible)

we use tags pretty extensively as a service discovery mechanism + data push down (to say prometheus exporters).

when the tag set changes, then we have to issue a new oauth client. the web UX doesn't make it easy to create a new one, as you have to click each individual tag. we have ~30. 🏷️

#hachyderm #hachyinfra

Hey, folks - I recently joined the @shipit podcast to talk shop about #hachyderm infra. give it a listen and hear what goes into keeping this thing running!

thanks for having me, autumn & justin! :blobfoxheadphones:

https://changelog.com/shipit/130

#hachyinfra

once i found the right setting to enable it, scraping `pgcat` for metrics from prometheus just worked (tm).

#hachyderm #hachyinfra #sre

Hey, Hachyderm - with recent US election results and the policy changes it implies, we have been thinking deeply here on the #hachyderm infrastructure team about how to better position Hachy to keep our users safe.

We'll share a strategy blog post soon that has a set of thematic improvements we can make to further harden our position from both a security and long-term service continuity perspective.

In the meantime (and after!), we welcome any questions you have.

:blobfoxninja:

#hachyinfra

Howdy, @tailscale! I'm a Hachyderm infra admin. We use #tailscale to stitch things together across clouds to make Hachyderm work. With yesterday's election results, in the event some kind of law were to be enacted restricting LGBTQIA+ content in the US, would we be in breach of the https://tailscale.com/terms and have our service terminated?

If yes, what are our self-hosting or alternative options?

#hachyinfra

@renchap poking at the new #otel support in 4.3.0, and there's a bunch of neat data here. yay!

is there any value in keeping these repeated ZSCAN traces? they're the vast majority of our current event usage, and they don't seem to be terribly meaningful. usually (maybe exclusively) just a span that says ZSCAN.

safe to drop these? i don't think we'll lose any meaningful observability by doing so?

#hachyderm #hachyinfra #MastoInfra #opentelemetry

@honeycombio we may, uh, need to do sampling. :blobfoxsciencewoozy:

#otel #opentelemetry #hachyderm #hachyinfra #forscience

experimenting with the new #opentelemetry (otel) capabilities in mastodon 4.3.0.

we're just getting started, but getting some super neat visibility by Just Sending Stuff to @honeycombio!

#hachyderm #hachyinfra

i learned so much about `certbot` today #HachyInfra

#hachyderm please enjoy your freshly renewed certificates :blobfoxheartcute:

@stux (and others!) what pointers do you have for periodic full backups of the masto db? pg_dump is nice & slow. :blobfoxcry2:

#mastoadmin #HachyInfra

Just finished doing a mega `terraform import` of the hachyderm.io and hachyderm.media DNS zones using #terraform and https://github.com/cycloidio/terracognita.

Pretty fun - first time using Terracognita - it did a nice job of scaffolding out the HCL & terraform state. A few regexes later, some rearranging, some state surgery, and everything's there!

#hachyderm #HachyInfra #devops