#mstdndk — Public Fediverse posts

Beep-beep-boop-de-boop! #MSTDNDK was down for a brief moment, while we applied the latest update.

Beep-beep-boop-de-boop! #MSTDNDK was down for a brief moment, while we applied the latest update.

#MSTDNDK is going down for a moment due to critical updates. BRB - I hope ;-)

#MSTDNDK is going down for a moment due to critical updates. BRB - I hope ;-)

For some odd reason, I at one point disabled #DKIM signatures on outgoing e-mail on my #Postfix / #OpenDKIM setup. With the #DEMARC and #SPF policies still in place, some of the e-mail being relayed, including that of #MSTDNDK must unfortunately have been rejected at the destination mail servers. The issue is fixed now.

Shoutout to https://mailtested.com - it came in handy when testing if signatures were in place! :-)

For some odd reason, I at one point disabled #DKIM signatures on outgoing e-mail on my #Postfix / #OpenDKIM setup. With the #DEMARC and #SPF policies still in place, some of the e-mail being relayed, including that of #MSTDNDK must unfortunately have been rejected at the destination mail servers. The issue is fixed now.

Shoutout to https://mailtested.com - it came in handy when testing if signatures were in place! :-)

#Crowdsec has been re-enabled on #Midgaard - the #Kubernetes cluster running #MSTDNDK. We're doing this without CloudFlare and virtualization - it's #baremetal and homegrown #firewall all the way.

Should you experience connectivity issues, either as a user or a federated instance, please reach out to us.

#Crowdsec has been re-enabled on #Midgaard - the #Kubernetes cluster running #MSTDNDK. We're doing this without CloudFlare and virtualization - it's #baremetal and homegrown #firewall all the way.

Should you experience connectivity issues, either as a user or a federated instance, please reach out to us.

#mstdndk was just migrated to the #traefik ingress controller from the now obsolete #nginx. Since we're lifting and shifting (I feel so filthy just saying that) and not doing it as a big bang, it means https://mstdn.dk just got a new set of IP addresses. Hopefully you'll never notice, but let me know if something acts up.

#mstdndk was just migrated to the #traefik ingress controller from the now obsolete #nginx. Since we're lifting and shifting (I feel so filthy just saying that) and not doing it as a big bang, it means https://mstdn.dk just got a new set of IP addresses. Hopefully you'll never notice, but let me know if something acts up.

The language rules on #mstdndk are based on the abilities of our moderators. Allowing the languages of all Denmark's neighboring countries except Germany seemed wrong, so we're making an effort and adding that to the list. It's not a hard rule, so if you reply to a couple of toots in Swahili, that fine to! 🙂

The language rules on #mstdndk are based on the abilities of our moderators. Allowing the languages of all Denmark's neighboring countries except Germany seemed wrong, so we're making an effort and adding that to the list. It's not a hard rule, so if you reply to a couple of toots in Swahili, that fine to! 🙂

Numerous technical and security improvements on the infrastructure that supports https://mstdn.dk

• DNS simplified extensively by migrating public facing secondary nameservers to #NSD using #CatalogZones from PowerDNS + DNSDist.
• #DNSSEC reenabled
• #ExternalDNS and #CertManager configuration vastly simplified.
• #Ingress controller migrated from #Nginx to #Traefik

Bottom line: https://sikkerpånettet.dk/ now gives the site a 100% #security score. There are still improvements to be made (weirdly enough) - specifically I'm looking into supporting DANE for #TLS certificate signatures in #DNS.

Now that's off the TODO-list :-)

#mstdndk

Numerous technical and security improvements on the infrastructure that supports https://mstdn.dk

• DNS simplified extensively by migrating public facing secondary nameservers to #NSD using #CatalogZones from PowerDNS + DNSDist.
• #DNSSEC reenabled
• #ExternalDNS and #CertManager configuration vastly simplified.
• #Ingress controller migrated from #Nginx to #Traefik

Bottom line: https://sikkerpånettet.dk/ now gives the site a 100% #security score. There are still improvements to be made (weirdly enough) - specifically I'm looking into supporting DANE for #TLS certificate signatures in #DNS.

Now that's off the TODO-list :-)

#mstdndk

#MSTDNDK was down for a moment there - seems the PostgreSQL database ran full, and didn't exactly recover gracefully. Will investigate when I'm not drinking beers in #Schiphol 🙂

#MSTDNDK was down for a moment there - seems the PostgreSQL database ran full, and didn't exactly recover gracefully. Will investigate when I'm not drinking beers in #Schiphol 🙂

Some time this weekend, we'll be upgrading the #Mastodon #PostgreSQL cluster on #MSTDNDK. The cluster is being managed by #CloudNativePG and upgrades have been successfully tested on multiple other clusters - even upgrades directly from version 15 to 18.

While this is an incredibly smooth and easy process, major upgrades requires that #CNPG takes down the whole database cluster, meaning https://mstdn.dk will be unavailable for a period of time. Exactly how long is unknown, since we've never actually done this before with clusters of this size.

I'll keep you posted and give you fair warning 🙂

Some time this weekend, we'll be upgrading the #Mastodon #PostgreSQL cluster on #MSTDNDK. The cluster is being managed by #CloudNativePG and upgrades have been successfully tested on multiple other clusters - even upgrades directly from version 15 to 18.

While this is an incredibly smooth and easy process, major upgrades requires that #CNPG takes down the whole database cluster, meaning https://mstdn.dk will be unavailable for a period of time. Exactly how long is unknown, since we've never actually done this before with clusters of this size.

I'll keep you posted and give you fair warning 🙂

I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.

I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.

Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.

I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.

This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.

#Linux #MSTDNDK #K8s

I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.

I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.

Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.

I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.

This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.

#Linux #MSTDNDK #K8s

One of the #MSTDNDK nodes is currently experiencing issues with an #NVMe disk that started reporting #SMART errors. It's still up, and we're talking to #Hetzner support about scheduling a replacement. Once we've agreed on a plan moving forward, the node will be taken down and hardware will be replaced. Hopefully you won't feel a thing.

One of the #MSTDNDK nodes is currently experiencing issues with an #NVMe disk that started reporting #SMART errors. It's still up, and we're talking to #Hetzner support about scheduling a replacement. Once we've agreed on a plan moving forward, the node will be taken down and hardware will be replaced. Hopefully you won't feel a thing.

Hård uge. Arbejder og sover. Heldigvis holder @leeleedee styr på #mstdndk imens jeg jamrer og klager! Tak for det! 😊

.. og med de ord vil jeg se dyner! 🤣

Hård uge. Arbejder og sover. Heldigvis holder @leeleedee styr på #mstdndk imens jeg jamrer og klager! Tak for det! 😊

.. og med de ord vil jeg se dyner! 🤣

@Lodahl Hver server kan have sin egen opsætning af den maksimale størrelse på et indlæg/toot. Her på #mstdndk er det 1.000 tegn. Hvis du vil lave et indlæg der er længere end det, skal du bruge threaded mode, hvilket i praksis vil sige at du lave en række indlæg der er bundet sammen.

@Lodahl Hver server kan have sin egen opsætning af den maksimale størrelse på et indlæg/toot. Her på #mstdndk er det 1.000 tegn. Hvis du vil lave et indlæg der er længere end det, skal du bruge threaded mode, hvilket i praksis vil sige at du lave en række indlæg der er bundet sammen.

The #MSTDNDK Mastodon instance just migrated away from Bitnami's #Redis helm chart and container images to #Valkey.

#Bitnami and Redis both seem to be exiting the open source community, which could prevent us from staying current, meaning always running the latest versions of the software components that make up this instance.

We understand the need to monetize software, but see that as an opportunity to add premium paid features, not take away existing ones from open source. Doing so will make your potential future customers look for other solutions. Looking at you, #MinIO, Redis and Bitnami.

Please report any issues you might experience as a result of the move to Valkey. So far it looks peachy.

The #MSTDNDK Mastodon instance just migrated away from Bitnami's #Redis helm chart and container images to #Valkey.

#Bitnami and Redis both seem to be exiting the open source community, which could prevent us from staying current, meaning always running the latest versions of the software components that make up this instance.

We understand the need to monetize software, but see that as an opportunity to add premium paid features, not take away existing ones from open source. Doing so will make your potential future customers look for other solutions. Looking at you, #MinIO, Redis and Bitnami.

Please report any issues you might experience as a result of the move to Valkey. So far it looks peachy.

Velkommen til @anduin som - hvis ikke #nypåmastodon - så i hvertfald ny på #mstdndk 🙂

Velkommen til @anduin som - hvis ikke #nypåmastodon - så i hvertfald ny på #mstdndk 🙂

The #Matrix Homeserver loosely associated with #MSTDNDK - https://mtrix.dk/ - is currently undergoing maintenance. We'll probably be dropping #KeyCloak in favor of the new Matrix Authentication Service to simplify the setup. Stay tuned.

#mtrixdk

The #Matrix Homeserver loosely associated with #MSTDNDK - https://mtrix.dk/ - is currently undergoing maintenance. We'll probably be dropping #KeyCloak in favor of the new Matrix Authentication Service to simplify the setup. Stay tuned.

#mtrixdk

Hej #mstdndk, jeg er #nypåmastodon. Jeg er tysker og lærer dansk. Jeg håber jeg kan træne at skrive på dansk her, fordi jeg mangler muligheden at gøre det i mit hverdagsliv i Tyskland. 🙂

Hej #mstdndk, jeg er #nypåmastodon. Jeg er tysker og lærer dansk. Jeg håber jeg kan træne at skrive på dansk her, fordi jeg mangler muligheden at gøre det i mit hverdagsliv i Tyskland. 🙂

Velkommen til #mstdndk, @uldkaktus !

#nypåmastodon

Velkommen til #mstdndk, @uldkaktus !

#nypåmastodon

Carefully taking the first steps required for switching from the now obsolete #nginx #ingress controller to #traefik. The plan is to spin it up on alternative HTTP(S) ports and try to move a couple of ingress resources to that. If I do this right, you shouldn't feel a thing! ;-)

#kubernetes #k8s #mstdndk #ingresscontroller

Carefully taking the first steps required for switching from the now obsolete #nginx #ingress controller to #traefik. The plan is to spin it up on alternative HTTP(S) ports and try to move a couple of ingress resources to that. If I do this right, you shouldn't feel a thing! ;-)

#kubernetes #k8s #mstdndk #ingresscontroller

Velkommen til #mstdndk , @mikkelbue !

Velkommen til #mstdndk , @mikkelbue !

Kære medborgere på #mstdndk! Glædelig bagjul og godt nytår! Tag fordi I er her, og tak for jeres tålmodighed! Det er en rejse at drive et setup som dette, og engang imellem koster det lidt nedetid at blive klogere. Med fare for a jinxe det, så har det kørt rigtig godt et stykke tid nu, og vi satser på at det fortsætter sådan! 🙂 Antallet af aktive brugere er lidt dalende, men det er jo som bekendt ikke kvantitet men kvalitet der tæller! Det er en fornøjelse at følge med i det lokale feed, og mit indtryk er at vi har skabt et rimelig udholdeligt lille hjørne af Fødiverset her! Pas godt på jer selv derude, og vi ses i 2026! 🙂🎆🎇

Kære medborgere på #mstdndk! Glædelig bagjul og godt nytår! Tag fordi I er her, og tak for jeres tålmodighed! Det er en rejse at drive et setup som dette, og engang imellem koster det lidt nedetid at blive klogere. Med fare for a jinxe det, så har det kørt rigtig godt et stykke tid nu, og vi satser på at det fortsætter sådan! 🙂 Antallet af aktive brugere er lidt dalende, men det er jo som bekendt ikke kvantitet men kvalitet der tæller! Det er en fornøjelse at følge med i det lokale feed, og mit indtryk er at vi har skabt et rimelig udholdeligt lille hjørne af Fødiverset her! Pas godt på jer selv derude, og vi ses i 2026! 🙂🎆🎇

Velkommen til #mstdndk, @mafiltenborg !

Velkommen til #mstdndk, @mafiltenborg !

Velkommen til #mstdndk, @C_P !

Velkommen til #mstdndk, @C_P !

Made a backup account over on dice.camp . Just in case our #mstdndk server is out for a bit and I need to provide situation updates. You can follow that here:

@[email protected]

Made a backup account over on dice.camp . Just in case our #mstdndk server is out for a bit and I need to provide situation updates. You can follow that here:

@[email protected]

Monitoring is slowly getting up and running.. now if only I knew how to read all that! ;-)

#mastodon #postgresql #cloudnativepg #grafana #mstdndk

Monitoring is slowly getting up and running.. now if only I knew how to read all that! ;-)

#mastodon #postgresql #cloudnativepg #grafana #mstdndk

Velkommen til #mstdndk @lmjhe

Velkommen til #mstdndk @lmjhe

One of the nodes of the Midgaard #Kubernetes cluster running #mstdndk is currently down, and awaiting a disk replacement. Luckily clustering works, and you shouldn't feel a thing ;-)

Resiliency for the win! :-)

One of the nodes of the Midgaard #Kubernetes cluster running #mstdndk is currently down, and awaiting a disk replacement. Luckily clustering works, and you shouldn't feel a thing ;-)

Resiliency for the win! :-)

Tonight is #Halloween, which also means it's the #birthday of #mstdndk! Three years ago exactly, the domain was registered and we deployed #Mastodon with my home grown Helm chart. A couple of horrible crashes and a lot of experience later, we're still alive and kicking! With backups and everything! :mastowink:

Have a great day! 🙂

Tonight is #Halloween, which also means it's the #birthday of #mstdndk! Three years ago exactly, the domain was registered and we deployed #Mastodon with my home grown Helm chart. A couple of horrible crashes and a lot of experience later, we're still alive and kicking! With backups and everything! :mastowink:

Have a great day! 🙂

#mstdndk and the underlying Kubernetes cluster died a couple of days ago. Working on recovering.

#mstdndk and the underlying Kubernetes cluster died a couple of days ago. Working on recovering.

Considering switching the #MinIO backend of #mstdndk to #Garage by #deuxfleurs. 3 replicas on #Kubernetes. Anyone with real life experience and/or tips? :-)

https://garagehq.deuxfleurs.fr/

#mastodon #s3

Considering switching the #MinIO backend of #mstdndk to #Garage by #deuxfleurs. 3 replicas on #Kubernetes. Anyone with real life experience and/or tips? :-)

https://garagehq.deuxfleurs.fr/

#mastodon #s3