I've been helping some friends and colleagues block some of the site scraping bots that are feeding "AI" models. Decided to take some of my notes and make something others could use too. It's a work-in-progress. Happy to add to or correct things.
https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
Clarkesworld /
npub1la…z48hr
2023-08-24 12:18:53
Author Public Key
npub1lat6krv8e3qfqzau0cyw2s6e3lf2q8pxkh7yt6sk0k7mtq53kgcqdz48hrPublished at
2023-08-24 12:18:53Event JSON
{
"id": "61693f68751fc81db968c43699b4a8b9f5eafa8acb9fe21ce739a1bd16cefa14",
"pubkey": "ff57ab0d87cc40900bbc7e08e543598fd2a01c26b5fc45ea167dbdb58291b230",
"created_at": 1692879533,
"kind": 1,
"tags": [
[
"proxy",
"https://mastodon.online/users/clarkesworld/statuses/110944553128570356",
"activitypub"
]
],
"content": "I've been helping some friends and colleagues block some of the site scraping bots that are feeding \"AI\" models. Decided to take some of my notes and make something others could use too. It's a work-in-progress. Happy to add to or correct things.\nhttps://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/",
"sig": "c9d42632c3afe5226c7c240dda60367c6018bb17879c59fc2c0fefada5569800411861cbf7ccdbfa509adb446ec06da53a02c2a402aec555c3acb1b0b0bd0394"
}