#systemssafety — Public Fediverse posts on home.social

I wouldn't want to be among the people who now need to explain why there is a single-point-of-failure for critical infrastructure. It might have looked like an acceptable risk at the time. After the fact it looks like a foolish decision.

I'd expect that other subsystems will be scrutinized as well. This looks like a higher level problem.

#heathrow #SystemsSafety

https://www.theguardian.com/uk-news/live/2025/mar/22/heathrow-airport-closure-fire-reopens-london-travel-flight-information-latest-news-updates

#systemssafety #heathrow

Thomas @[email protected] · 2024-12-26 · 08:20 UTC

As a reminder: don't let LLMs handle anything in the political sphere unless you have RLHF (Reinforcement Learning from Human Feedback) active before you show the result to anyone*. Also think of automation risks and human factors (HF). That's "Good Old Systems Safety".

*) ... or unless your goal is to damage a 3rd party's reputation (fake news style).

#llm #ai #rlhf #automationrisks #SystemsSafety

https://www.theregister.com/2024/12/20/apple_ai_headline_summaries/?td=rt-3a

#systemssafety #automationrisks #rlhf #ai #llm

Adam Cook @[email protected] · 2023-07-24 · 15:43 UTC

Now, let’s look at the underlying #SystemsSafety part.

#Autopilot and #FSDBeta-equipped vehicles are Level 2-capable vehicles - with the exact same *limitations* as many other vehicles on the market today.

Namely, the key limitation is that the human driver must remain the fallback for any dynamic driving task or vehicle failures at all times and under all conditions.

Effectively, that means that the human has the exact same control responsibilities between the two vehicles shown below.

#autopilot #fsdbeta #systemssafety

Adam Cook @[email protected] · 2023-07-18 · 12:57 UTC

#SystemsSafety has an odd power.

People today cannot really remember a time when everyday products would readily kill or maim. So, having lost those experiences over the decades, Silicon Valley increasingly saw a business opportunity.

But modern society is grounded on the public’s trust and, given enough critical mass, trust can be virtually lost overnight.

Quite literally.

That is Great Depression stuff right there, folks.

#systemssafety

Adam Cook @[email protected] · 2023-07-07 · 08:41 UTC

@CrackedWindscreen Tragic.

Really.

Indeed. These incidents require an exhaustive root cause analysis... which is, as you know, very rarely performed.

But the knee-jerk "just sprinkle a little automation on it" is a absolute Cancer of Simplicity that has zero #SystemsSafety foundation.

There are immense downsides to automation that are very vague.

#systemssafety

Adam Cook @[email protected] · 2023-07-05 · 11:02 UTC

Let's talk about #Mercedes vehicles equipped with #DrivePilot a bit - a Level 3-capable vehicle that has been recently "approved" in a handful of US states.

This article almost entirely focuses on the legal dynamics of consumer liability should this vehicle create a direct (or, presumably, an indirect) incident.

But, as always, I want to talk about what I feel are the #SystemsSafety realities at work here and the many foot-guns that are associated with that.

#cars

https://www.autonews.com/mobility-report/mercedes-drive-pilot-automated-system-poses-legal-questions

🧵👇

#mercedes #drivepilot #systemssafety #cars

Adam Cook @[email protected] · 2023-07-04 · 03:05 UTC

@lolgop Constantly. Constantly this is done.

It is beyond exhausting.

And it has actually proved to be extraordinarily dangerous when #Musk lies about the capabilities and availability of #Tesla's #FSDBeta product, in particular.

The press often allows Wall Street analysts, that are not competent in #SafetyCritical systems, to advance Musk's dangerous lies.

The #SystemsSafety community has been battling this for years.

#musk #tesla #fsdbeta #safetycritical #systemssafety

Adam Cook @[email protected] · 2023-06-27 · 16:05 UTC

I suppose that I should also note that no system can ever be "perfectly safe".

That is not possible

And the concept of "perfection" is not relevant to systems safety.

Ross submits that "at no time was anybody at any risk of crashing".

No.

There is **always** risk!

#SystemsSafety is about maintaining processes such that always-present, finite risk is continuously and exhaustively identified and managed.

It is about appreciating that risk exists - the opposite of what Ross submits.

#Tesla

#systemssafety #tesla

Adam Cook @[email protected] · 2023-06-27 · 15:15 UTC

I was at #Chicago Pride all weekend while visiting with my wife (videos and photos soon!), so I missed this #Tesla Drama concerning #FSDBeta that erupted.

Ok.

Let us, again, all put on our #SystemsSafety hats and take a look at the situation here as I understand it.

Below is the video that kicked the beehive between Tesla defenders and detractors on "what really happened?".

This clearly chaotic video was taken from a larger drive sequence in which FSD Beta was active.

🧵👇

#chicago #tesla #fsdbeta #systemssafety

Adam Cook @[email protected] · 2023-06-19 · 09:43 UTC

@CrackedWindscreen Bingo. You nailed it!

**Anytime** someone or something uses the term "I think", "could be" or similar and then makes some sort of #SystemsSafety assessment... you can toss it right in the trash.

Systems safety is about constantly and exhaustively asking pointed questions and seeking quantifiable answers.

Academic and industry research can be used to be develop a safety case, but it is not at all complete by itself.

#systemssafety

Adam Cook @[email protected] · 2023-06-18 · 04:23 UTC

@BruceMirken This is the same Bad Faith shit that us in the #SystemsSafety community have been dealing with on the #Tesla #Autopilot and #FSDBeta wrongdoings for years.

I am not going to lecture Dr. Hotez on how to deal with this... but I can say this... the strategy on the #Musk-side and of his sycophants is to **passionately pretend** like they want to have a Good Faith debate... but they are just looking for "gotcha soundbites".

#systemssafety #tesla #autopilot #fsdbeta #musk

Adam Cook @[email protected] · 2023-06-15 · 10:28 UTC

@kentindell and @CrackedWindscreen,

Let me just say this for the record and for what it is worth, #Ford's Jim Farley is worrying the hell out of me lately in terms of #SystemsSafety.

And I think something really objectionable could be brewing there.

The way he is talking... I recognize this. This talk is familiar to me. I remember it at #Boeing when I was there many moons ago.

I am watching the layoff reporting at Ford very carefully, to the extent that I can.

#ford #systemssafety #boeing

Adam Cook @[email protected] · 2023-06-15 · 01:17 UTC

Efficiently and reliably extracting "safety data" from the roadway is a non-starter.

It simply cannot be done.

Why?

Because the only "safety data" that actually matters for this conversation is "data" that is **forensically** extracted from all interactions with a #FSDBeta-active vehicle - **both** direct and indirect interactions.

No one has that.

Not even close.

Not even #Tesla.

Zero chance.

Numbers on a page are not important to #SystemsSafety.

The root causes are!

#fsdbeta #tesla #systemssafety

Adam Cook @[email protected] · 2023-06-13 · 01:54 UTC

I am almost starting to get the feeling that we, the public and the #SystemsSafety community, are being trolled by these #Tesla #FSDBeta YouTube titles.

Perhaps gaslighted is a better term?

Anyways, yup, "safety" on display here all right...

https://www.youtube.com/watch?v=6QKklP91ITU

#systemssafety #tesla #fsdbeta

Adam Cook @[email protected] · 2023-06-10 · 17:12 UTC

@colburn Zero actual independent, oversight for the validation processes associated with these automated driving system programs.

Zero.

Not that it matter much, but the #SystemsSafety community has been pounding on this table for years.

When an Uber ATG automated vehicle killed a pedestrian in Arizona way back in 2018… absolutely nothing was done by federal regulators.

#systemssafety

Adam Cook @[email protected] · 2023-06-10 · 13:35 UTC

#SystemsSafety is not predicated upon statistics, data, videos on YouTube/Twitter and personal experiences.

It is **continuously** defined by a **process** of exhaustively asking questions about the robustness of the system in response to identified failure modes.

No engineered system can ever be "perfectly safe".

That is not the issue.

The issue is what does the initial and continuous (that is, never ended) validation process look like for any given system.

#systemssafety

Adam Cook @[email protected] · 2023-06-10 · 09:02 UTC

Really thinking of just dropping #Reddit, for what it is worth.

Dropped my profile off my Mastodon bio.

Reddit management is giving me serious #Musk vibes lately and I have been getting follow spammed 4-5 times daily for the last two weeks.

Met a lot of great technical and #SystemsSafety experts on there though - mostly through pushing back against #Tesla's #Autopilot and #FSDBeta wrongdoings.

I suppose that work continues here now...

#reddit #musk #systemssafety #tesla #autopilot #fsdbeta

Adam Cook @[email protected] · 2023-06-08 · 00:06 UTC

First off, #cars are **not** smartphones.

I cannot say that enough.

And if you hear anyone describing them as such, it almost certainly means that they are (knowingly or not) hand-waving away the incomparable #SystemsSafety differences between a consumer electronic device and a #SafetyCritical system.

That makes reports like this on #Tesla's hiring preferences **very** concerning: https://www.washingtonpost.com/technology/2021/05/14/tesla-apple-tech/

#cars #systemssafety #safetycritical #tesla

Adam Cook @[email protected] · 2023-06-07 · 23:59 UTC

Oh memories.

Taking a break from #Musk's Hate Train on the Hellsite to recall this series of Tweets from a few years ago.

While under-appreciated then and now, the Tweet thread by Musk posted below contains an extremely damning #SystemsSafety admission and it displays the considerable #PublicSafety blind spot associated with remotely updating #SafetyCritical systems without oversight.

Musk has no clue what he admitted to here, but systems safety experts do.

#Tesla

#musk #systemssafety #publicsafety #safetycritical #tesla

Adam Cook @[email protected] · 2023-06-07 · 08:34 UTC

@jbenjamint @ottocrat The other thing routinely left out of these conversations is lifecycle “emissions” (negative externalities) through the potential complication of #SystemsSafety via #EVs.

This “fact check” opinion article may be more on point than the opinion article it is responding to (and so believe that it is), but such pieces are, as usual forgetting of the larger, complex system at work here.

Systems atop systems. These pieces at least need to make a nod to it, in my view.

#systemssafety #evs

Adam Cook @[email protected] · 2023-06-05 · 08:52 UTC

@serichards @CrackedWindscreen @kentindell It is the "unseen" issues after an OTA update is applied (in a system as massively-complex as a car) that concern me.

And one of the chief avenues where unseen issues creep in is when Configuration Management (CM) becomes impractical.

We already know that #Tesla completely hand-waves away CM, because #Musk admitted it not too long ago.

When I saw this Tweet, as a #SystemsSafety engineer, my jaw hit the floor.

Musk has no idea what he admitted to.

#tesla #musk #systemssafety

Adam Cook @[email protected] · 2023-06-05 · 08:18 UTC

@CrackedWindscreen The problem is... and I am not at all convinced Farley recognizes it (or wishes to admit it based on, potentially, feedback from its internal teams)... that today's consumer software demands cannot have a safety foundation under it.

That is the "tricky" part.

That is the part that many automakers likely cannot square internally.

I would have to toss all of my #SystemsSafety experience in the garbage to satisfy today's auto consumer.

#automotive #ElectricVehicles #Ford

#systemssafety #automotive #electricvehicles #ford

Adam Cook @[email protected] · 2023-06-01 · 09:19 UTC

Have they "improved"?

How do we know?

How is "improved" defined?

See where I am going here?

#SystemsSafety is not about appearances or YouTube videos or press releases or expanded service areas or the lack of "bad news reports" or numbers on a page.

It is about constantly asking questions and receiving quantifiable answers of system reliability.

#systemssafety

Adam Cook @[email protected] · 2023-06-01 · 09:11 UTC

Hmm. Going to throw up some flags here. This needs a #SystemsSafety perspective.

Firstly, there is no #PublicSafetly value (ultimately, the only thing that matters) in "criticizing" #Waymo and #Cruise (or, at least without criticizing, say, the #NHTSA at the same time)...

They are operating under a complete and deliberate lack of regulatory scrutiny that **should be** in place, but is not in place.

#systemssafety #publicsafetly #waymo #cruise #nhtsa

Adam Cook @[email protected] · 2023-05-31 · 22:43 UTC

Competent #SystemsSafety experts knew better that they would not, though.

Because, exactly like I mentioned in the first Toot, no one and no firm has ever offered a remotely convincing safety case to realize a viable Level 5-capable vehicle.

A Level 5-capable vehicle has an effectively unbounded Operational Design Domain (ODD).

Such an expansive ODD would undoubtedly require a validation strategy that would rely on an unquantifiable number of breakthroughs.

#systemssafety

Adam Cook @[email protected] · 2023-05-31 · 22:34 UTC

No, because "flying cars" have never remotely made financial sense - particularly with the #SystemsSafety concerns which are almost always hand-waved away by the startups.

#Michigan #VTOL

https://www.crainsdetroit.com/manufacturing/does-detroit-city-airport-have-future-flying-cars

#systemssafety #michigan #vtol

Adam Cook @[email protected] · 2023-05-31 · 02:11 UTC

@PeceK All of this said, while the few names I highlighted do not seem to have the "technical chops" at understanding these technologies... almost every single signatory to these various letters stayed silent (or applauded) while:

1. #Musk and Andrej #Karpathy were engaged in these wrongdoings; and/or
2. While Musk sent his cult to attack Professor Missy Cummings, a top #SystemsSafety researcher, who first raised these issues years ago.

There is some real hypocrisy here.

#musk #karpathy #systemssafety

Adam Cook @[email protected] · 2023-05-29 · 15:56 UTC

Presumably, the individual in the article activated #FSDBeta.

Yes, I understand what the full name of that product is called, but there is no need to expand it.

Do not assist #Tesla and #Musk in their lies.

Additionally, I would argue that it is an ethical responsibility to note that these vehicles are not capable of “driving themselves”, at the very least.

The #SystemsSafety community has spent **years** publishing open content on these matters.

Gotta pay attention.

#fsdbeta #tesla #musk #systemssafety

Adam Cook @[email protected] · 2023-05-26 · 09:58 UTC

@CrackedWindscreen Honestly, I am not sure that I have ever seen any reporting (in a major publication) that really digs down deep into the core issue here - flimsy and non-existent safety **processes** at #Tesla.

Some competent #SystemsSafety experts tapped by some publications are given space to make nods to it, but really not enough space.

I mean… the lack of a process at Tesla is clear… that **definitively** means that Tesla is doing nothing to understand their own system.

#tesla #systemssafety

Adam Cook @[email protected] · 2023-03-14 · 15:11 UTC

#Safety is a word that is tossed around like a cheap suit these days - particularly in the context of #AutomatedDriving systems and, more typically, in white hot topics like #Tesla's #FSDBeta program.

But what is "safety" (or “#SystemsSafety”, terminology that I prefer to use)?

How is it defined?

This thread is probably overdue given the immense amount of incorrect and strawman arguments that I see out there on this topic - and that has real-world ramifications.

Let us explore very briefly.

#safety #automateddriving #tesla #fsdbeta #systemssafety

Adam Cook @[email protected] · 2023-03-03 · 16:15 UTC

This is an interesting development by #Ford - basically picking up the engineers from the now-defunct Argo.AI unit that was previously financed by #Ford and #VW.

Argo.AI had (now Latitude AI has) some of the finest #robotics and #ControlSystems and #SystemsSafety experts in the industry.

That said...

https://media.ford.com/content/fordmedia/fna/us/en/news/2023/03/02/ford-establishes-latitude-ai-to-develop-future-automated-driving.html

#ford #vw #robotics #controlsystems #systemssafety

Adam Cook @[email protected] · 2023-02-10 · 23:29 UTC

Gotta respect the hell out of @coachtony for being so open here on past #medium challenges and missteps.

Feels like a service that I can trust, which is a bit odd to say these days - and I will be looking hard at Medium this very weekend.

I have been meaning to start a #SystemsSafety, #Manufacturing, #Robotics and #EngineeringEthics publication...

https://news.ycombinator.com/item?id=34747015

#medium #systemssafety #manufacturing #robotics #engineeringethics

Adam Cook @[email protected] · 2022-12-17 · 22:05 UTC

This is an absolute tragedy for #RoadwaySafety, at a time when we can least afford it in the US.

Professor Missy Cummings is one of the foremost experts on automated systems, Human Factors and #SystemsSafety and her work at the #NHTSA was undercut from Day 1... undercut by multiple issues.

The NHTSA desperately needed her expertise during a time of very dangerous technological change in the auto industry.

And that is gone now.

Let me explain.

https://subscriber.politicopro.com/article/2022/12/missy-cummings-target-of-tesla-ire-to-depart-nhtsa-00074075

#VisionZero

🧵 🔽

1/6

#roadwaysafety #systemssafety #nhtsa #visionzero

Teixi @[email protected] · 2022-11-20 · 17:47 UTC

» But his advice is one I often repeat to my students:
take a sheet of paper, divide it into two columns, and as objectively as you can list the pluses and minuses of each place.
Then sleep on it and in the morning go with your gut feeling.«

In Memoriam: Frederick P. Brooks, Jr.
—Personal Recollections @SteveBellovin

https://www.cs.columbia.edu/~smb/blog/2022-11/2022-11-18.html

#TheMythicalManMonth
#RequirementsEngineering
#SoftwareEngineering
#DistributedSystems
#SystemsSafety
#SoftwareCraft

#themythicalmanmonth #requirementsengineering #softwareengineering #distributedsystems #systemssafety #softwarecraft

Teixi @[email protected] · 2022-11-20 · 17:47 UTC

» But his advice is one I often repeat to my students:
take a sheet of paper, divide it into two columns, and as objectively as you can list the pluses and minuses of each place.
Then sleep on it and in the morning go with your gut feeling.«

In Memoriam: Frederick P. Brooks, Jr.
—Personal Recollections @SteveBellovin

https://www.cs.columbia.edu/~smb/blog/2022-11/2022-11-18.html

#TheMythicalManMonth
#RequirementsEngineering
#SoftwareEngineering
#DistributedSystems
#SystemsSafety
#SoftwareCraft

#themythicalmanmonth #requirementsengineering #softwareengineering #distributedsystems #systemssafety #softwarecraft

Teixi @[email protected] · 2022-11-20 · 17:47 UTC

» But his advice is one I often repeat to my students:
take a sheet of paper, divide it into two columns, and as objectively as you can list the pluses and minuses of each place.
Then sleep on it and in the morning go with your gut feeling.«

In Memoriam: Frederick P. Brooks, Jr.
—Personal Recollections @SteveBellovin

https://www.cs.columbia.edu/~smb/blog/2022-11/2022-11-18.html

#TheMythicalManMonth
#RequirementsEngineering
#SoftwareEngineering
#DistributedSystems
#SystemsSafety
#SoftwareCraft

#themythicalmanmonth #requirementsengineering #softwareengineering #distributedsystems #systemssafety #softwarecraft

Teixi @[email protected] · 2022-11-20 · 17:47 UTC

» But his advice is one I often repeat to my students:
take a sheet of paper, divide it into two columns, and as objectively as you can list the pluses and minuses of each place.
Then sleep on it and in the morning go with your gut feeling.«

In Memoriam: Frederick P. Brooks, Jr.
—Personal Recollections @SteveBellovin

https://www.cs.columbia.edu/~smb/blog/2022-11/2022-11-18.html

#TheMythicalManMonth
#RequirementsEngineering
#SoftwareEngineering
#DistributedSystems
#SystemsSafety
#SoftwareCraft

#softwarecraft #systemssafety #distributedsystems #softwareengineering #requirementsengineering #themythicalmanmonth

Teixi @[email protected] · 2022-11-20 · 17:47 UTC

» But his advice is one I often repeat to my students:
take a sheet of paper, divide it into two columns, and as objectively as you can list the pluses and minuses of each place.
Then sleep on it and in the morning go with your gut feeling.«

In Memoriam: Frederick P. Brooks, Jr.
—Personal Recollections @SteveBellovin

https://www.cs.columbia.edu/~smb/blog/2022-11/2022-11-18.html

#TheMythicalManMonth
#RequirementsEngineering
#SoftwareEngineering
#DistributedSystems
#SystemsSafety
#SoftwareCraft

#themythicalmanmonth #requirementsengineering #softwareengineering #distributedsystems #systemssafety #softwarecraft

Adam Cook @[email protected] · 2022-11-14 · 13:25 UTC

Time for another #SystemsSafety Monday!

#OTA (Over-The-Air) software updates and remote #telemetry mechanisms for #SafetyCritical systems are an under-appreciated public safety hazard when unregulated - as they are today.

Some automakers already have these mechanisms available on existing vehicles and many more equipped vehicles are now coming to market.

Very soon, these mechanisms will be commonplace.

🧵 ⬇️

#Automotive #SystemsSafety #RoadwaySafety

1/6

#systemssafety #ota #telemetry #safetycritical #automotive #roadwaysafety