https://www.crikey.com.au/2024/09/03/ai-worse-summarising-information-humans-government-trial/
> Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found. […]
> The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.
> Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.
> These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.
Flick 🇬🇧 /
npub1lm…n85vl
2024-09-04 09:24:30
Author Public Key
npub1lmksrgs7e5w5gz53fgq38py0dw3yfjdyur2nv0vg7rspce6sr4lqqn85vlPublished at
2024-09-04 09:24:30Event JSON
{
"id": "e7997b2962bafdb633a2cce083220195daa37a58c7ef3e193bbb19918d72fe61",
"pubkey": "feed01a21ecd1d440a914a0113848f6ba244c9a4e0d5363d88f0e01c67501d7e",
"created_at": 1725441870,
"kind": 1,
"tags": [
[
"proxy",
"https://spinster.xyz/objects/03204c75-013b-425f-b352-f6cd21800edc",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://spinster.xyz/objects/03204c75-013b-425f-b352-f6cd21800edc",
"pink.momostr"
],
[
"-"
]
],
"content": "https://www.crikey.com.au/2024/09/03/ai-worse-summarising-information-humans-government-trial/\n\n\u003e Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found. […]\n\n\u003e The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.\n\n\u003e Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.\n\n\u003e These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.",
"sig": "b25ce97fd6b31b9df1f786a6daecd6b3ed052148e60e92c6f76a839cd525206d82dfbab3d6de6628f3fc4e683872d54036734260781fda5dee56f53a716f8255"
}