Lorin Hochstein

Student of complex systems failures, resilience engineering, cognitive systems engineering. Will talk your ear off about learning from incidents in software.

Public Key

npub1h2ga9usf27gyr2hgvzv5grmaw4x4dssz4e8753gjydc0p3jcd0yqxzw6q6

NIP-05 Address

norootcause_at_hachyderm.io@mostr.pub

Profile Code

nprofile1qqst4ywj7gy40yzp4t5xpx2ypa7h2n2kcgp2unl2g5fzxu8scevxhjqpz4mhxue69uhhyetvv9ujumt0wd68ytnsw43qz9thwden5te0v35hgar09ec82c30wfjkcctecyh3nu

Publishing to

relay.mostr.pub

ditto.pub/relay

Author Public Key

npub1h2ga9usf27gyr2hgvzv5grmaw4x4dssz4e8753gjydc0p3jcd0yqxzw6q6

Show more details

Last Notes

2024-07-30 20:47:31

“Deadlock” would be a pretty cool supervillain name.

2024-07-24 21:07:51

I don’t think the estimates of the cost of CrowdStrike outage takes into account the lost productivity of the ensuing post-incident commentary and discussion.

2024-07-20 05:25:04

I gotta admit, named pipes is not something that comes up often in incident write-ups. Didn’t even know that Windows supported them!
https://www.crowdstrike.com/blog/technical-details-on-todays-outage/

2024-07-19 22:53:18

Every SRE job posting should include the word “wrangling” in the description. Because, boy, is there ever a lot of wrangling!

2024-06-30 18:28:25

Complaining about things not scaling doesn’t scale

2024-06-29 14:11:49

Hypothesis: once a company reaches a certain size, to a first approximation, all of its technical problems are actually socio-technical ones.

2024-06-27 15:28:27

- reply

@npub1rml…xa6t I had to look this up to understand it, but it was worth it. https://yourdailygerman.com/meaning-german-feierabend/

2024-06-26 11:54:05

Whenever I visit the east coast, I forget how much I enjoy having breakfast at diners.

2024-06-25 18:27:51

Tired: software developers don’t know enough computer science
Wired: software developers don’t know enough social science

2024-06-20 22:30:07

- reply

@npub1nlt…t44t That was *amazing*! I can’t believe I haven’t seen it before.

2024-06-20 20:53:45

Still waiting for someone to invent work-that-best-advances-the-goals-of-the-org-driven-development

2024-06-19 23:37:44

Tired: expert systems
Wired: experts in the system

2024-06-16 21:08:11

Cook’s “How complex systems fail” is the most personally impactful paper I have ever read, and yet I’m convinced that it would never have been accepted for publication in a peer-reviewed journal.
https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf

2024-06-13 19:50:12

If your technical interview process selects for quick thinking over deep thinking, what does that say about your workforce?

2024-06-12 02:15:43

Not Invented Here
vs
No In-house expertise Here

2024-06-12 01:39:09

I know which teams own the various boxes, but who owns the arrows?

2024-06-08 05:32:17

It is so damned seductive to believe that you can automate the risk away.

2024-06-07 20:36:25

Charlton Heston running through the streets, screaming “Availability is made out of people! It’s *people*!”

2024-05-26 23:25:23

You get what you model.

2024-05-19 19:03:10

The hard thing about distributed systems is that "turn the whole thing off and on again" isn't an option.

2024-05-16 03:58:33

They should rename Chief Technology Officer to Chief Migration Officer.

2024-04-30 01:22:34

Whoever is responsible for redirecting https://wedontneedno.education to wikipedia: I salute you.

2024-04-24 19:22:59

- reply

@npub1nlt…t44t Wait, will you make customer service calls on my behalf???

2024-04-24 18:50:25

Greasing-the-wheels-as-a-service

2024-04-21 19:18:04

- reply

@npub1rml…xa6t I heard it often in my youth growing up in Quebec!

2024-04-21 19:10:14

- reply

@npub1rml…xa6t That is the only way I have heard it pronounced by English speakers!

2024-04-21 18:57:15

It's funny that we say "in lieu of" instead of "in place of" given that "lieu" is French for "place". I guess we're saving a letter?

2024-04-14 17:55:19

- reply

@npub1rml…xa6t 404? More like 410.

2024-04-10 02:30:55

You can’t win, but you’ve still gotta try

2024-04-01 21:10:54

Somebody should write a Lisp book titled “Prose and Cons”.

2024-03-29 16:24:14

Most overused word in my writing: actually

2024-03-29 01:42:59

"Human error is the attributed cause of large system accidents because human performance in these complex systems is so good. Failures of these systems are, by almost any measure, rare and unusual events. Most of the system operations go smoothly; incidents that occur do not usually lead to bad outcomes. These systems have come to be regarded as 'safe' by *design* rather than by *control*."
– R.I. Cook, D.D. Woods, Operating at the Sharp End: The Complexity of Human Error

2024-03-16 19:47:51

If you don’t know how the work actually gets done, your proposed improvements are unlikely to have the effects that you expect.

2024-03-12 20:02:14

Brb, typing “now make it even more reliable” into the LLM chat interface.

2024-03-11 23:33:17

How come nobody talks about the P99TTD or P99TTR?

2024-03-08 05:10:59

In a complex system, there isn’t a “safety” knob that you can just turn to the right to increase safety. Safety features increase complexity (new failure modes!) and have opportunity costs (finite resources!). Every intervention involves a tradeoff.

2024-03-07 03:54:34

One day I’d like to write a “Senior Software Engineer” chatbot that randomly responds to questions with answers like:
- “It depends…”
- “What are you trying to accomplish?”
- “You’re asking the wrong question.”

2024-03-06 03:10:10

Example of a technology that has been around my entire life, and yet I amazed every time I interact with it: aluminum foil.

2024-02-29 18:30:28

hot-potato-driven-development

2024-02-29 01:46:03

Two of the biggest sources of incidents I’ve seen are:
1. Legacy code
2. Migrating away from legacy code
The conclusion is clear: you should only write non-legacy code

2024-02-27 05:29:32

“A weakness in the [action item] list approach is a tendency to jump to countermeasures before we understand a situation.” – Mike Rother, Toyota Kata

2024-02-23 04:42:04

Every time I have to vote for a judge on the California ballot, I think “Judges really shouldn’t be selected by popular vote.”

2024-02-18 21:42:49

If you don’t know what the constraints are, you won’t be able to make sense of the system behavior.

2024-02-11 20:50:31

Fun line from Porter's book "Trust in Numbers":
> Only graduate students pay much attention to published papers, mature scientists interact mainly by talking, not writing.
(He's referring specifically to high-energy physics here)

2024-02-11 19:26:08

- reply

@npub1nlt…t44t It's like full-scale ping-pong!

2024-02-06 04:24:27

We should bring back cloaks. What a fantastic idea it is to treat a blanket as an article of clothing.

2024-01-31 01:10:23

A reminder that I don’t even know what to worry about: https://www.scientificamerican.com/article/sand-mafias-are-plundering-the-earth/?s=08

2024-01-24 01:43:39

There’s no potence like idempotence.

2024-01-14 23:42:09

- reply

@npub1nlt…t44t But you will learn the names of the dogs!

2024-01-14 23:04:49

If you are ever interested in practicing small talk, I recommend getting a dog. You will get many, many opportunities to strike up conversations with other dog owners on walks. It’s really remarkable.

2023-12-31 20:10:47

- reply

"Indeed, the balance between procedural compliance, production goals, and the reactors' inherent unpredictability may prove unstable. As one manager indicated, 'The next major accident in the nuclear industry (after TMI) will be caused by operators following procedures.' The very system created to contain and control risk may actually create danger."

2023-12-31 20:09:03

- reply

Another great quote from this paper:
"Feeling that they were likely candidates for punishment, they worked covertly rather than openly, 'doing what the boss wants, not what he says.' They subverted formal management process to meet management's expectations."

2023-12-31 19:55:37

- reply

And what is automation but executable procedures?

2023-12-31 19:54:35

"No procedures, however well written, can substitute for technical knowledge." – nuclear plant supervisor quoted by Larry Hirschhorn in "Hierarchy versus bureaucracy: the case of a nuclear reactor".

2023-09-19 04:13:47

Why LLMs are AI: https://existentialcomics.com/comic/516

2023-07-15 21:53:01

- reply

@npub1nlt…t44t How quaint!

2023-07-15 21:36:36

Why do we call it the command-line and not the request-line?

2023-06-19 05:01:47

If you claim the system failed because of human error, then you’re saying that your system requires error-free humans to function without failure.
If that’s the case, your problem isn’t the people, it’s that you have a fragile system.

2023-06-14 03:02:04

> If I told you that US venture capitalists promoted a Ponzi scheme that used a cartoon computer game to steal hundreds of millions of dollars from poor workers in the Philippines and send it to North Korea to fund a ballistic missile program, you probably wouldn’t believe me. Unless I said “… using crypto,” in which case you would probably say “oh yeah that sounds about right.”
I wish I could write like Matt Levine.

2023-03-22 16:30:27

Fantastic talk by #[0] at #[1] on cognitive apprenticeship
https://www.usenix.org/conference/srecon23americas/presentation/cruz
- maybe learning is a good enough goal by itself
- "I didn't know system worked like that"
- "I didn't know those things talked to each other"
- "Oh, *that's* how you query!"