john on Nostr: Im not an expert but the LLM analyzes the text broken down into little tokens like ...
Im not an expert but the LLM analyzes the text broken down into little tokens like words or parts of words or letters and then the llms are trained on statistical relationships between the tokens in the data it analyzes. This gives tokens a kind of preference in relation to other tokens (preference is just a word I used not a technical term) so when you use the llm you aren't seeing humanities knowledge youre seeing a statistical representation of humanities mode of expressing its knowledge.
Published at
2024-08-20 16:44:27Event JSON
{
"id": "c0c2f07a13df054ab83ef25250330020045954d9bc3a9dc95c8f3c5adb6b1046",
"pubkey": "67ada8e344532cbf82f0e702472e24c7896e0e1c96235eacbaaa4b8616052171",
"created_at": 1724172267,
"kind": 1,
"tags": [
[
"e",
"06557523384d69460b2f2a36de7dab6bd68c9d06d40b37edd0246c18820bb76a",
"wss://nos.lol",
"root"
],
[
"e",
"114761536fa075978206b1b8f4011486eb48989f9e6c90bcce3986205f7703c5",
"wss://nos.lol",
"reply"
],
[
"p",
"82341f882b6eabcd2ba7f1ef90aad961cf074af15b9ef44a09f9d2a8fbfbe6a2"
],
[
"p",
"af9fa70cd13b02b3712106aa271cb42c7075d7c489de5da353093e759745ef73"
],
[
"p",
"67ada8e344532cbf82f0e702472e24c7896e0e1c96235eacbaaa4b8616052171"
],
[
"p",
"6c6e3e05e1c9d2aae0ed2431544aea411771dd9d81017539af0fd818b2389f28"
]
],
"content": "Im not an expert but the LLM analyzes the text broken down into little tokens like words or parts of words or letters and then the llms are trained on statistical relationships between the tokens in the data it analyzes. This gives tokens a kind of preference in relation to other tokens (preference is just a word I used not a technical term) so when you use the llm you aren't seeing humanities knowledge youre seeing a statistical representation of humanities mode of expressing its knowledge.",
"sig": "88c5b84c2fd76d3cbfb9b20459db6c0a284464107a1b2231bc15d5152731b6b6bd398758bf5b2589beec8cb240b83ad3fae719b5c96e183bd66546da4229f31d"
}
