Does anyone out there have a favorite, kept-up-to-date robots.txt for rejecting AI scraper bots and the like? I'm certain as I move web hosts that it's time to update mine.
(And yes, I know the bots have a maddening tendency to ignore robots.txt but at least having an updated list will repulse *some* and therefore worth having)
Boosts welcome for a more complete list of options
Michael J. Coffey /
npub187…9ru75
2024-10-16 15:50:10
Author Public Key
npub187yk6xlgw2afzytspc5v0w7a0wv6j2cc5tfsmfum4lxsw40npl5ss9ru75Published at
2024-10-16 15:50:10Event JSON
{
"id": "6062e77066309ecd6df1956c7fa212cc45b501ba41face3f30bf2b1d9358cf57",
"pubkey": "3f896d1be872ba9111700e28c7bbdd7b99a92b18a2d30da79bafcd0755f30fe9",
"created_at": 1729093810,
"kind": 1,
"tags": [
[
"proxy",
"https://scholar.social/users/michaelc/statuses/113317891957085895",
"activitypub"
]
],
"content": "Does anyone out there have a favorite, kept-up-to-date robots.txt for rejecting AI scraper bots and the like? I'm certain as I move web hosts that it's time to update mine. \n\n(And yes, I know the bots have a maddening tendency to ignore robots.txt but at least having an updated list will repulse *some* and therefore worth having)\n\nBoosts welcome for a more complete list of options",
"sig": "34f50e607b2f6056ae853c066eb99d00145d553eb6efc47cfd75f225330da322c3f5047960371048ee4b380529d5d5cd49d56086484c10df69c7e5ff387dadfe"
}