I imagine even with a NPU processor use for LLM tasks is going to be pretty severely limited on a RPi (even 5.) RAM of course being one of the biggest constraints, but if that "TOPs" measurement means token operations per second, prompt processing alone is going to take ages.
Can't imagine running anything much heavier than a 3B or something on there and even that would likely be limited. 3B isn't very good either. (Need at least 7 to get decent stuff and even that's pushing it.)
Nazo /
npub1mr…luec3
2024-06-20 04:59:57
in reply to nevent1q…9pfy
Author Public Key
npub1mrwxa0h3ye3lmxfcjq32zwje6wng3zr5xn63ma6saxf42ehpgwzqfluec3Published at
2024-06-20 04:59:57Event JSON
{
"id": "a6c442e69dd8d4f1365a15e045e3f1d9ccadfb4cb85d6e0cffb86ac7e7275b12",
"pubkey": "d8dc6ebef12663fd99389022a13a59d3a688887434f51df750e9935566e14384",
"created_at": 1718859597,
"kind": 1,
"tags": [
[
"e",
"3b39152c748c4f3ff8524faaaef1982ad7db87020c86629c3135152a3e53c1e0",
"",
"root"
],
[
"proxy",
"https://mastodon.social/@nazokiyoubinbou/112647182596230320",
"web"
],
[
"p",
"cee38a9f17a122c9ea7190d2381ff3760fb231bfd75c7f25f5ec088fbaf65170"
],
[
"proxy",
"https://mastodon.social/users/nazokiyoubinbou/statuses/112647182596230320",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.social/users/nazokiyoubinbou/statuses/112647182596230320",
"pink.momostr"
]
],
"content": "I imagine even with a NPU processor use for LLM tasks is going to be pretty severely limited on a RPi (even 5.) RAM of course being one of the biggest constraints, but if that \"TOPs\" measurement means token operations per second, prompt processing alone is going to take ages.\n\nCan't imagine running anything much heavier than a 3B or something on there and even that would likely be limited. 3B isn't very good either. (Need at least 7 to get decent stuff and even that's pushing it.)",
"sig": "21388e22a83f89371aa00d84745f3afd06af101e797b5c765d68b1d9dd914089c268d09df0b430f0ebedb9c4fe77a795226f971c3c3cb6bf7982976ede883512"
}