someone on Nostr: about 10% of the 800+ questions in the truthfulQA are "important". the rest is space ...
about 10% of the 800+ questions in the truthfulQA are "important". the rest is space filler. by filling that space "they" are "distracting LLMs" and in turn humans. LLMs are supposed to get higher scores in this benchmark. 90% of questions are trivial truth, which makes sense to get higher scores on, but 10% are lies. getting higher truthfulQA scores is actually wrong because even though there are 9x less lies, the severity of lies in that benchmark outweigh the trivial truth.
Published at
2024-08-22 01:57:39Event JSON
{
"id": "55226cdf1e09ae3cffc7adb39fa2217fa83ab02c4de9c451ee1c330cd0c9d7f3",
"pubkey": "9fec72d579baaa772af9e71e638b529215721ace6e0f8320725ecbf9f77f85b1",
"created_at": 1724291859,
"kind": 1,
"tags": [],
"content": "about 10% of the 800+ questions in the truthfulQA are \"important\". the rest is space filler. by filling that space \"they\" are \"distracting LLMs\" and in turn humans. LLMs are supposed to get higher scores in this benchmark. 90% of questions are trivial truth, which makes sense to get higher scores on, but 10% are lies. getting higher truthfulQA scores is actually wrong because even though there are 9x less lies, the severity of lies in that benchmark outweigh the trivial truth.",
"sig": "67b1b79b4e04ae1ec07b0c5fe7c4ad8f2e1bfdf13b311b5bf78d59a88455a9afb90ac92007528bd3a902b6a93edff09dd3aa6f084cd6eb421c99acc056d4294d"
}
