Instead of robots.txt targetting every new tool we should have opt-in.txt that gives permissions per type of use. These uses should be legally binding especially for companies.
Companies have resources to do it right. Each site owner having to keep up with robots and blocks is insanity.
I'm just tired of the constant opt-out mentality. And the need to block. Crawling and indexing is not free you know, especially the AI players can cause a LOT of unnecessary traffic these days.
Vesa Piittinen /
npub17q…prtth
2024-05-17 09:33:37
Author Public Key
npub17qjdea6xt7pgzm6cnu4tevprq9gqn6rf94g83mfn8h8fjpa0r33s4prtthPublished at
2024-05-17 09:33:37Event JSON
{
"id": "8d34cf326842a4fcc9a0bdfbe01413a01923e18407f714c691ca1b61550ac8c9",
"pubkey": "f024dcf7465f82816f589f2abcb023015009e8692d5078ed333dce9907af1c63",
"created_at": 1715938417,
"kind": 1,
"tags": [
[
"proxy",
"https://mastodon.social/@MerriNet/112455740130956617",
"web"
],
[
"proxy",
"https://mastodon.social/users/MerriNet/statuses/112455740130956617",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.social/users/MerriNet/statuses/112455740130956617",
"pink.momostr"
]
],
"content": "Instead of robots.txt targetting every new tool we should have opt-in.txt that gives permissions per type of use. These uses should be legally binding especially for companies.\n\nCompanies have resources to do it right. Each site owner having to keep up with robots and blocks is insanity.\n\nI'm just tired of the constant opt-out mentality. And the need to block. Crawling and indexing is not free you know, especially the AI players can cause a LOT of unnecessary traffic these days.",
"sig": "b21c3c2950518f9d062045b2019dd0e87d123529bd410d8de1b60d2f57f4a951ac77220ae1e0a01b233e59a2bbece8a2ba947f7bbbffbaf44506f82c75022173"
}