Have you read anything by Subbarao Kambhampati? He has a lot of great posts that cut away at AI hype language, here’s a recent snippet about alignment:
“IMHO, the problem with that recent Anthropic study about "unfaithful chains of thought" is that they, like a big part of the alignment contingent, ascribe some sort of intentionality to the model. As I wrote two months back, there is actually no reason to expect semantics for the intermediate tokens, or any causal connection between the intermediate tokens and the result.
While conjuring up "deception" or "malice" is certainly good for AI Alignment business, it may not actually be needed when intermediate tokens are just intermediate mumbles that the model is trained to produce just to increase its chance of stumbling on more correct solution tokens.”
https://www.linkedin.com/posts/subbarao-kambhampati-3260708_sundayharangue-activity-7314399765448871937-Cdxt?utm_source=share&utm_medium=member_ios&rcm=ACoAAALDc-YBehJjDzNPgRqoDj_Pmj5ONH5lCXQ
Dustin / Dustin Dannenhauer
npub1mg…gpdjc
2025-04-08 15:33:46
in reply to nevent1q…0hup
Author Public Key
npub1mgvwnpsqgrem7jfcwm7pdvdfz2h95mm04r23t8pau2uzxwsdnpgs0gpdjcPublished at
2025-04-08 15:33:46Event JSON
{
"id": "a16bfebc552b07d8ce679ec01924f1f756eaed8c078c2377e146e82a428eabb8",
"pubkey": "da18e9860040f3bf493876fc16b1a912ae5a6f6fa8d5159c3de2b8233a0d9851",
"created_at": 1744126426,
"kind": 1,
"tags": [
[
"e",
"44503bef21fd81048133e4fb2a656a1b72f567f321c203900bfdf1ed2babc534",
"",
"root"
],
[
"e",
"11d4e349ae20b450e990fbfedac8fbb5d37f6f0f9a0b5afd4c2e20c17d13cdad",
"",
"reply"
],
[
"p",
"9fec72d579baaa772af9e71e638b529215721ace6e0f8320725ecbf9f77f85b1"
],
[
"p",
"c5fadeb5d90d68baffc631455a07ca340ccf1e31110955e16d45eb5f87147cd9"
],
[
"p",
"da18e9860040f3bf493876fc16b1a912ae5a6f6fa8d5159c3de2b8233a0d9851"
],
[
"client",
"Nostur",
"31990:9be0be0fc079548233231614e4e1efc9f28b0db398011efeecf05fe570e5dd33:1685868693432"
]
],
"content": "Have you read anything by Subbarao Kambhampati? He has a lot of great posts that cut away at AI hype language, here’s a recent snippet about alignment:\n\n“IMHO, the problem with that recent Anthropic study about \"unfaithful chains of thought\" is that they, like a big part of the alignment contingent, ascribe some sort of intentionality to the model. As I wrote two months back, there is actually no reason to expect semantics for the intermediate tokens, or any causal connection between the intermediate tokens and the result. \n While conjuring up \"deception\" or \"malice\" is certainly good for AI Alignment business, it may not actually be needed when intermediate tokens are just intermediate mumbles that the model is trained to produce just to increase its chance of stumbling on more correct solution tokens.”\n\nhttps://www.linkedin.com/posts/subbarao-kambhampati-3260708_sundayharangue-activity-7314399765448871937-Cdxt?utm_source=share\u0026utm_medium=member_ios\u0026rcm=ACoAAALDc-YBehJjDzNPgRqoDj_Pmj5ONH5lCXQ\n",
"sig": "4e159cdaa9873a2f2a7a4f0c54bfd650c3708df5dd851e0e2d311bc582b0a0f11e21be2529d94edb3811d0fa944c47a9338e7d4a5561380e3eaabab5f0764615"
}