#sysadminnery — Public Fediverse posts

I swear there is nothing more annoying in my job than the complete and total nightmare of domain names, distinct auth systems, and self-competing branding that fell out of SolarWinds’ dismemberment.

I never saw the value in this mess and still don’t. I only ever touch any of it when it makes noise, which it only does when the tooling itself is breaking in ways that I can't even comprehend based on the alerts.

#Sysadminnery

I swear there is nothing more annoying in my job than the complete and total nightmare of domain names, distinct auth systems, and self-competing branding that fell out of SolarWinds’ dismemberment.

I never saw the value in this mess and still don’t. I only ever touch any of it when it makes noise, which it only does when the tooling itself is breaking in ways that I can't even comprehend based on the alerts.

#Sysadminnery

I confess:
My aversion to LLMs is based entirely on not being able to imagine any problem that could conceivably be solved with a computer that I would want solved by a LLM rather than doing it myself or finding someone else's proven solution or just not bothering.

E.g. most of the work I need to do is resolving tech debt. Lots of updates of substantially out of date software, largely because clients resist maybe-breaking updates and any downtime. No LLM is going to do that.

#Sysadminnery

I confess:
My aversion to LLMs is based entirely on not being able to imagine any problem that could conceivably be solved with a computer that I would want solved by a LLM rather than doing it myself or finding someone else's proven solution or just not bothering.

E.g. most of the work I need to do is resolving tech debt. Lots of updates of substantially out of date software, largely because clients resist maybe-breaking updates and any downtime. No LLM is going to do that.

#Sysadminnery

I pay for my lack of responsibility for Sales by also lacking the authority to decline to work with clients who insist on using LLMs as their Level 1 tech support while behaving like it's a Level 3 solution that I am required to deploy. However, I do have permission to treat LLM output with all the respect it merits, as long as I explain to clients why it is bad/wrong with proper explanations for why and citations as available.

That's billable Sr. Staff time.

#Sysadminnery

Just so that anyone unfamiliar with the fundamentals of #Sysadminnery doesn’t misunderstand:

If OpenAI had wanted a closed testing sandbox, they would have actually constructed and used a closed testing sandbox. THIS IS NOT HARD.

Sure, it's possible that they have shit netadmins and sysadmins who couldn’t contain a test planned by shit devoopsies and project managers unwilling to put out the effort to build REAL containment. They intentionally tested against production: the Internet at large.

Just so that anyone unfamiliar with the fundamentals of #Sysadminnery doesn’t misunderstand:

If OpenAI had wanted a closed testing sandbox, they would have actually constructed and used a closed testing sandbox. THIS IS NOT HARD.

Sure, it's possible that they have shit netadmins and sysadmins who couldn’t contain a test planned by shit devoopsies and project managers unwilling to put out the effort to build REAL containment. They intentionally tested against production: the Internet at large.

I’m starting to think that the only programmers who understand how to work with email use Perl.

e.g. imapsync is reliably good and reasonably documented, with excellent exception handling/error descriptions, while imap-tools-rs is reliably garbage with inconsistent docs and error messages literally only decipherable by the developer (or someone willing to learn the code.)

#Sysadminnery #shitpost

I hadn’t built OpenSSL from source in *decades*, and I don’t recall it using so much Perl. I’ve been doing the configure/fix/configure/make/fix/make/fix.. process with a dozen yum and/or cpan invocations as the fixes. Needed on a CentOS 6.10 machine with Perl 5.10 as the base.

Oh. The whole test stage of the build is Perl. Cool.

#Sysadminnery

I hadn’t built OpenSSL from source in *decades*, and I don’t recall it using so much Perl. I’ve been doing the configure/fix/configure/make/fix/make/fix.. process with a dozen yum and/or cpan invocations as the fixes. Needed on a CentOS 6.10 machine with Perl 5.10 as the base.

Oh. The whole test stage of the build is Perl. Cool.

#Sysadminnery

I’m building a toolchain to get OpenSSH 10 running on CentOS 6 on a production box to fix backups, in case anyone is envying my July 4 festivities.

#Sysadminnery

I’m building a toolchain to get OpenSSH 10 running on CentOS 6 on a production box to fix backups, in case anyone is envying my July 4 festivities.

#Sysadminnery

My boss’ biggest flaw: he's too reluctant to criticize me.
In 2017, that contributed to me being fired. He never passed along to me the fact that the CTO (since departed) was peeved at my chaotic schedule.
Today I found a deeply embarrassing error that I made some months ago which just became visible because another system was updated. Easy enough to fix once I figured it out & no harm done (unless someone wants a PIT restore of that Redmine server from the last few months...)
#Sysadminnery

My boss’ biggest flaw: he's too reluctant to criticize me.
In 2017, that contributed to me being fired. He never passed along to me the fact that the CTO (since departed) was peeved at my chaotic schedule.
Today I found a deeply embarrassing error that I made some months ago which just became visible because another system was updated. Easy enough to fix once I figured it out & no harm done (unless someone wants a PIT restore of that Redmine server from the last few months...)
#Sysadminnery

I am having a “recover from updates” day.
Updated my backup server FreeBSD v13->v14 and one of my 4 authoritative DNS servers 14->15.

It turns out that the default MTA is now dma. That’s probably a good idea, except that dma installed on a machine that has a working mail setup relying on features not in dma does stupid shit. Also, the ssh update forced a host key update so all the backups broke. AND: OpenSSL on 15 doesn’t like my antique CGP server: weak dh params.

#Sysadminnery

I wrote more Perl yesterday than I had in the past 3 years.
It was embarrassing, because the only way to actually test this code, a CGI program, was on the live system. That meant committing every change to SVN and each one needed a comment, and my capacity for minor LLM-like syntax errors: they look like they ought to be right, but somehow are not, in visible ways.

Anyway, my non-programmership is confirmed.

#Sysadminnery

Very reminiscent of my time in Big Corp IT.
I got asked for a detailed weekly report on the performance of mail servers and spam filters issued weekly. It took me months to get the report form & content just right for the manager that requested it.

He left. His replacement didn’t recognize my weekly report mail as something useful and rather than telling me to stop, he just kept marking it as spam.

#Sysadminnery @aj https://gts.sadauskas.id.au/@aj/statuses/01KVZBAX3H4TDKD8T0CETDEDW1

Very reminiscent of my time in Big Corp IT.
I got asked for a detailed weekly report on the performance of mail servers and spam filters issued weekly. It took me months to get the report form & content just right for the manager that requested it.

He left. His replacement didn’t recognize my weekly report mail as something useful and rather than telling me to stop, he just kept marking it as spam.

#Sysadminnery @aj https://gts.sadauskas.id.au/@aj/statuses/01KVZBAX3H4TDKD8T0CETDEDW1

It is always an iffy proposition to open a bug in the CPAN RT (https://rt.cpan.org/Ticket/Display.html?id=179191) when all the unresolved tickets for a module are years old and all the resolved ones are far older.

Anyway: if the latest Net::CIDR::Lite update broke your use of it, apply the simple fix in that ticket. Who knows when it will actually get addressed in the canonical version…

#Sysadminnery #Perl

It is always an iffy proposition to open a bug in the CPAN RT (https://rt.cpan.org/Ticket/Display.html?id=179191) when all the unresolved tickets for a module are years old and all the resolved ones are far older.

Anyway: if the latest Net::CIDR::Lite update broke your use of it, apply the simple fix in that ticket. Who knows when it will actually get addressed in the canonical version…

#Sysadminnery #Perl

RE: https://mastodon.online/@rfceditor/116757407381105796

I felt a shudder in the fabric of spacetime as a million shell scripts choked on the 5th digit.

#Sysadminnery

No, really, creating a login role account so multiple people can have access to support a system is very bad.

— How I make developers look at me incredulously.

#InfoSec #Sysadminnery

RE: https://infosec.exchange/@bob_zim/116754944378922259

Like we wouldn’t all recognize Oracle's support system...
#Sysadminnery

RE: https://infosec.exchange/@bob_zim/116754944378922259

Like we wouldn’t all recognize Oracle's support system...
#Sysadminnery

LOL…
Just wasted most of a day and all of my capacity for diplomacy on a customer whose problem amounted to “Microsoft has a feature I dislike.”

It’s called “Focused Inbox” and while I also now believe it to be a flawed feature, I believe that it provides some users with something they want, and that their provider of $REDACTED just has to deal with not being that important to users…

#Sysadminnery #spam

ALSO: The "raw unmodified email message" does NOT mean what you get from "view source" on your Outlook Web Access UI.

I weep for the state of the computing workforce.

#Sysadminnery

#PSA: When your email admin asks you clearly and explicitly for a copy of a raw original email message that is problematic sent to them as an attachment for analysis, they DO NOT mean just hit 'forward' in Outlook & go home for the day.

If you do this, expect them to metaphorically pants you in a reply-all for not following basic simple instructions.

ALSO: If you work in a technical field that ever involves sending email, you should have a basic understanding of RFC532{1,2} .
#Sysadminnery

Web server DDoS has made me resort to:

A cron job launches a simple while loop which once every 90 seconds kills all running processes named $x and $y, processes which are launched by the (valid) URLs being hit, which generate oom-kills of random processes otherwise. It runs for 900 minutes (15 hours) during the period when the DDoS is most intense. Needed to allow other important stuff to run.

Whoever is behind this insanity needs to die in a fire as soon as possible.

#Sysadminnery

RE: https://toad.social/@grumpybozo/116692326777473465

“Seem” is a favorite word…
The machine was gummed up to the point where fail2ban saw my failed logins as an attack.

#Sysadminnery

The metric of the day is “oom-kills per hour”

I is SRE

#Sysadminnery

The metric of the day is “oom-kills per hour”

I is SRE

#Sysadminnery

Cool. After an hour of trying to log into an actively DDoS’d webserver, all 5 retry cycles seem to have reached authentication…

#Sysadminnery

It helps that *every* *single* *time* anyone has insulted me by offering slop as an answer to a problem, it has been wrong in some egregious way.

#Sysadminnery

RE: https://infosec.exchange/@JessTheUnstill/116584328658916926

This is part of why I don’t let *ANY* slop emission at me to go unpunished.
I’m confident each step of my management chain appreciates what my humanity brings to my job, but customers are a different matter; they hire us because they lack IT chops. LLM-excreta give them the false impression that they can get an answer in seconds that I might take a day to write up a bit differently. E.g., with 'not' inserted in many places...

#Sysadminnery

Had an account manager feed me slop to “help” me address a MS365 problem today. Turns out Copilot is poorly informed about how today’s MS365 operates.

In its defense, it DID give me the key phrase that I needed to search for to understand the problem. It just told me to look for it in places where it is not. An actual web search enlightened me.

#Sysadminnery

Had an account manager feed me slop to “help” me address a MS365 problem today. Turns out Copilot is poorly informed about how today’s MS365 operates.

In its defense, it DID give me the key phrase that I needed to search for to understand the problem. It just told me to look for it in places where it is not. An actual web search enlightened me.

#Sysadminnery

Today MS365 forgot the existence of both of the 2 rarely-used accounts I have with them solely for the purpose of supporting one of our largest clients. Then an hour later they remembered my existence, so I could reset the password that needed resetting.

How do people use this garbage and find it acceptable?

#Sysadminnery #Microsoft

Today MS365 forgot the existence of both of the 2 rarely-used accounts I have with them solely for the purpose of supporting one of our largest clients. Then an hour later they remembered my existence, so I could reset the password that needed resetting.

How do people use this garbage and find it acceptable?

#Sysadminnery #Microsoft

Cloudflare Turnstile is garbage. If you use it, you will turn away Safari 26 users.

The troubleshooting process can get Safari users to a page where CF admits that they are requiring browsers to be fingerprintable.

FUCK THAT.

There is NOTHING online that is so critical that I will knowingly give a consistent fingerprint of my browser to Cloudflare. They are not an ethical actor. They absolutely cannot be trusted.

#InfoSec #Sysadminnery #WebAdmin

Cloudflare Turnstile is garbage. If you use it, you will turn away Safari 26 users.

The troubleshooting process can get Safari users to a page where CF admits that they are requiring browsers to be fingerprintable.

FUCK THAT.

There is NOTHING online that is so critical that I will knowingly give a consistent fingerprint of my browser to Cloudflare. They are not an ethical actor. They absolutely cannot be trusted.

#InfoSec #Sysadminnery #WebAdmin

Some people might call it a company culture problem that all of our customers are used to just emailing my boss whenever they have issues. I call it a way for his vacations to also be effective time off for me…

Of course, what I do on those days is all the tech debt stuff that I can never get done while firefighting.

#Sysadminnery

A few times monthly we get alerts from a MDR provider of data flow ceasing from one of the dozen clients we have set up for them. It’s never clear what’s gone wrong: the tool is an effectively undocumented "install this & run it" blob. I wasn’t involved in selecting this. I'm extremely skeptical of the provider, who owe their independent existence to a massive public breach under a former parent some years ago: problem unit(s) were spun out to save the parent.
My reply:
#InfoSec #Sysadminnery

RE: https://infosec.exchange/@david_chisnall/116543566441127430

I cannot begin to count how many words I’ve written about SMTP (and email in general) being unfit for various purposes.

The only sysadmin subspecialty I am notably advanced in is email administration.

#Sysadminnery

Just ran out of brainpower for the day trying to fix WTF a dev w/root on a production box did to deploy a #uwsgi application. Dev in question: no longer available. Apparently he didn't know there was a EL package available or how the “Emperor" rig works, so the app needs to be started by hand on every reboot. This was not a documented fact.

I failed to hack up a fix. The Emperor kept cursing the application instance. I like the absurd terminology, hate the documentation...

#Sysadminnery

Basic networking lesson I have had to learn repeatedly, including today…

Client devicess and routers will conspire to make things kinda halfway work when there’s a missing static route.

I probably should have read @mwl’s #n4sa2e before starting this project.

#Sysadminnery

#FreeBSD #HotTake: Even though everyone with deeper #firewall juju than myself says #pf is better than #ipfw, so I guess it must be, I still like knowing my rules by numbers that don’t change. Plus I have tools written over many years around ipfw and would need to totally redesign them conceptually for pf. I don't have enough working years to do that.

#Sysadminnery

CONFESSION:
My ongoing hatred of git is almost entirely grounded in my inability to switch FreeBSD branches without wiping and recloning.

#Sysadminnery

The truth is, I’ve never trusted FileVault or really any whole-disk encryption.
I have a couple of FreeBSD machines using geli but not on their boot disk (is that even a thing?) and I’ve never been willing to jump into FileVault because it seems to have had a steady stream of “Oh, well, hope you have a backup” problems.

#Sysadminnery #IWeep6Colors

Good. Hopefully it means that they hired a few good admins.

The worst thing to happen to email was providers of other services bundling email at prices less than it should cost to provide high quality service.

It has deprofressionalized email. The pressure on email operations to cost nothing has squeezed out high competence admins and left the largest operations mysterious black boxes to the people who run them.
#Sysadminnery @arstechnica https://mastodon.social/@arstechnica/115907321311459073

Open X-Change making a *universally breaking* change in config format & removing features from #Dovecot in the 2.3->2.4 upgrade is telling. That they removed some core features (e.g. unix login auth) inks in the drawing. They are tired of having the world's de facto default open source mailstore server, they want the world's most lucrative mailstore server.

And the page explaining the changes at https://doc.dovecot.org/2.4.0/installation/upgrade/2.3-to-2.4.html is basically a big fat FUCK YOU to the Dovecot Community.
#Sysadminnery

@schwa @alexr No kinkshaming on the Fediverse!

If someone wants to run Real Sendmail, that’s just fine as long as they don’t harm others in doing so.
I’d even say that for QMail. None of my business how people abuse themselves.

Exchange is, of course, a whole different story. There are limits, after all.

#Sysadminnery

Why doesn’t logrotate work on Ubuntu?

#Sysadminnery

(half-serious... I support 1 Ubuntu 20 LTS machine with a high-abuse web server and periodically need to run logrotate -F /etc/logrotate.d/apache2 by hand because it really doesn't work… I’m likely to figurer it out... )

Of course I had already told him that my proxy wasn't seeing anything from his firewall, so the problem couldn't be the customer's janky IIS box on the backside. But I had to go probing there anyway. Ultimately he got into his Cisco ADSM console and saw the problem quickly.

Proof that even if you buy @mwl's #n4sa2e book and learn enough about networks to school your network-headed boss, it might not be enough.

#Sysadminnery

Had a major customer infra migration today, went perfectly fine until it didn't, right when boss/network guy was boarding a plane.
He spent most of his DC-Miami trip texting w/me getting me to look at everything except his firewalls. He even made me go digging into a Win22 IIS and on-host firewall config, which is an absolute fucking nightmare.

Guess where the problem was...

#Sysadminnery

What's that work thing that I keep forgetting about until I run into it while doing something else and realize that I really need to fix that?

Why don't you know?

#Sysadminnery #IAmOld

Incidentally, the suspicious TLD lists are public and here's the file from which it is drawn:

https://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/sandbox/pds/20_ntld.cf?revision=1928318&view=markup

(remove everything after the '?' to see a list of all revisions)

If you use SA you can find the current lists in the 72_active.cf file of the daily rules distribution.

#PSA #SpamAssassin #spam #Sysadminnery #domains

#PSA: BEFORE selecting a domain name which you want to use for email, you definitely should consult the #SpamAssassin list of "suspicious" gTLDs. Those are gTLDs which have been so badly run that the overwhelming majority (99%+ in most cases) of messages using them for email addresses or even in URIs in the body are #spam. This means that if you pick example.online or examp-le.pro or any other domain in a suspicious gTLD, your mail will have delivery problems.

#Sysadminnery #domains

Why is there no scanf(1)?

#Sysadminnery

Customer just asked us to upgrade-in-place their 12yo RoR-based PM app currently running on Ubuntu 12.
Not looking forward to this fun.

#Sysadminnery

Canonical is evil.

If I wanted to use nano, I suppose I might run something like "nanosudo"

BUT THAT’S NOT WHAT I RAN.

It only took me about 30 seconds of flailing before I noticed the garbage at the bottom of the screen.

#Ubuntu #sudo #Sysadminnery