I tried to tell everyone that the robots.txt thing wouldn't be respected back when they first started trying to block LLM data crawlers and I got ripped into.
I don't know if there is any truly great solution, but yeah, companies like those are going to just ignore them.
Nazo /
npub1mr…luec3
2024-06-15 21:33:46
in reply to nevent1q…7skf
Author Public Key
npub1mrwxa0h3ye3lmxfcjq32zwje6wng3zr5xn63ma6saxf42ehpgwzqfluec3Published at
2024-06-15 21:33:46Event JSON
{
"id": "7f812a0fc9480bed803b4fa77113d9095566b419691eec52a8c1d7661c287b25",
"pubkey": "d8dc6ebef12663fd99389022a13a59d3a688887434f51df750e9935566e14384",
"created_at": 1718487226,
"kind": 1,
"tags": [
[
"p",
"9d6742ff43235eb568c0bc55fa8acf7e004ca06a7c9e0c3da2b089a098b769b9"
],
[
"proxy",
"https://mastodon.social/@nazokiyoubinbou/112622778873667549",
"web"
],
[
"e",
"16708908811ca84fad9de5e707cdaa48776201800b161fd7999970e9804e661f",
"",
"root"
],
[
"proxy",
"https://mastodon.social/users/nazokiyoubinbou/statuses/112622778873667549",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.social/users/nazokiyoubinbou/statuses/112622778873667549",
"pink.momostr"
]
],
"content": "I tried to tell everyone that the robots.txt thing wouldn't be respected back when they first started trying to block LLM data crawlers and I got ripped into.\n\nI don't know if there is any truly great solution, but yeah, companies like those are going to just ignore them.",
"sig": "3bdfc9ce8867b31a969b34b81bc340ca6849176cb8723a4310d8325b53e25f2c90169db160eca9c42a5f5cf9fd41251d45f441dea1d7afabda897ed59a5a2012"
}