nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqc9m22hkc5h6zgrwkz48crhcpw6vch2rf6j97746ugl3neys86jeqcr59k6 (nprofile…59k6) nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqjngmqxxnn9az3r8j257aag2e5vmxtqv9j3nmf8sugsv3us0dq0pqlw4xpp (nprofile…4xpp)
Ah - I think I understand your point now.
If in practice tokens only ever pay attention to recent tokens, then full attention matrices are very inefficient. I believe some implementations have done lower band diagonal implementations (ie rolling context windows) to address this - but it would eliminate the occasional long distance reference that a full-fat transformer would permit - maybe this matters…?
Tom_Drummond /
npub1g4…htq97
2024-11-19 02:53:22
in reply to nevent1q…ht62
Author Public Key
npub1g4ss3v8573z4aus0ytsq3ewah9cphk5p48ffnfz5hp8htsc9ndjqphtq97Published at
2024-11-19 02:53:22Event JSON
{
"id": "2de13bbfee977672fdade54948e37550f543947c30a60529c00946ac1fd45f38",
"pubkey": "456108b0f4f4455ef20f22e008e5ddb9701bda81a9d299a454b84f75c3059b64",
"created_at": 1731984802,
"kind": 1,
"tags": [
[
"p",
"c176a55ed8a5f4240dd6154f81df0176998ba869d48bef575c47e33c9207d4b2",
"wss://relay.mostr.pub"
],
[
"p",
"94d1b018d3997a288cf2553ddea159a3366581859467b49e1c44191e41ed03c2",
"wss://relay.mostr.pub"
],
[
"p",
"9be8a492bf1f95972c6c6789339a72a0f2533111690c83e808e740b198086c4a",
"wss://relay.mostr.pub"
],
[
"p",
"3422fcbc32f333fb2d3481b2e981258af8a0b571869cbfe93c42962410e232ef",
"wss://relay.mostr.pub"
],
[
"e",
"e773ec19fb257484ea75cbfcca47e6b4e1d5c680afa64c66b77cb9fcf09a7964",
"wss://relay.mostr.pub",
"reply"
],
[
"proxy",
"https://mathstodon.xyz/users/Tom_Drummond/statuses/113507356024662379",
"activitypub"
]
],
"content": "nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqc9m22hkc5h6zgrwkz48crhcpw6vch2rf6j97746ugl3neys86jeqcr59k6 nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqjngmqxxnn9az3r8j257aag2e5vmxtqv9j3nmf8sugsv3us0dq0pqlw4xpp \nAh - I think I understand your point now.\n\nIf in practice tokens only ever pay attention to recent tokens, then full attention matrices are very inefficient. I believe some implementations have done lower band diagonal implementations (ie rolling context windows) to address this - but it would eliminate the occasional long distance reference that a full-fat transformer would permit - maybe this matters…?",
"sig": "da2f5f721d0a03e5942fa376ead66920fbec6775c38b491edf8aab627cbb257009a14ee761b3cf88a3ca582fda4ed183c818a928b1b6ed06686843e7f191a7d3"
}