When AI is used to perform high-velocity "decision support" that is supposed to inform a "human in the loop," it quickly overwhelms its human overseer, who takes on the role of "moral crumple zone," pressing the "OK" button as fast as they can. This is bad enough when the sacrificial victim is a human overseeing, say, proctoring software that accuses remote students of cheating on their tests:
https://pluralistic.net/2022/02/16/unauthorized-paper/#cheating-anticheat
5/
Cory Doctorow /
npub1yx…75qvv
2024-10-29 19:32:08
in reply to nevent1q…uq4y
Author Public Key
npub1yxzkmtuyctjw2pffp6e9uvyrkp29hrqra2tm3xp3z9707z06muxsg75qvvSeen on
wss://relay.mostr.pubPublished at
2024-10-29 19:32:08Event JSON
{
"id": "3e5cce469da3d7ac9e97d0690060a996eb2ae66c3a1f0fa2f876dbbf6da826cc",
"pubkey": "21856daf84c2e4e505290eb25e3083b0545b8c03ea97b89831117cff09fadf0d",
"created_at": 1730230328,
"kind": 1,
"tags": [
[
"e",
"ff14a8e777e1b62d8b7c26687902fca654d8cf929d722cd2112e2da64935779d",
"wss://relay.mostr.pub",
"reply"
],
[
"content-warning",
"Long thread/5"
],
[
"proxy",
"https://mamot.fr/users/pluralistic/statuses/113392374829977021",
"activitypub"
]
],
"content": "When AI is used to perform high-velocity \"decision support\" that is supposed to inform a \"human in the loop,\" it quickly overwhelms its human overseer, who takes on the role of \"moral crumple zone,\" pressing the \"OK\" button as fast as they can. This is bad enough when the sacrificial victim is a human overseeing, say, proctoring software that accuses remote students of cheating on their tests:\n\nhttps://pluralistic.net/2022/02/16/unauthorized-paper/#cheating-anticheat\n\n5/",
"sig": "9e83f32458b47b88c12f94e5c81602cd996681765da08afab9fcf466ba56181bf9606ddad1e096a0c43dd607b7b0b2eb727c6e2609a0fc7f796b0f519075602e"
}