There is a real, measurable backlash against indiscriminate AI scraping that can be seen in the robots.txt files of websites.
~28% of the most actively updated sites now restrict scraping. 5% of all sites block scraping, up from ~1% last year
https://www.404media.co/the-backlash-against-ai-scraping-is-real-and-measurable/
404 Media /
npub14a…272mq
2024-07-23 13:36:18
Author Public Key
npub14akyzfynsanvcjlevy4ls936cuzz224puykv306tl6frl4kv8jsq0272mqPublished at
2024-07-23 13:36:18Event JSON
{
"id": "12906905af02c82f494bff75d42a4609466477d175060a59554aae73a5c7956c",
"pubkey": "af6c4124938766cc4bf9612bf8163ac704252aa1e12cc8bf4bfe923fd6cc3ca0",
"created_at": 1721741778,
"kind": 1,
"tags": [
[
"proxy",
"https://mastodon.social/users/404mediaco/statuses/112836069224272508",
"activitypub"
]
],
"content": "There is a real, measurable backlash against indiscriminate AI scraping that can be seen in the robots.txt files of websites.\n\n~28% of the most actively updated sites now restrict scraping. 5% of all sites block scraping, up from ~1% last year\n\nhttps://www.404media.co/the-backlash-against-ai-scraping-is-real-and-measurable/",
"sig": "f1cb09f3e78500b72ef5d5db6fd3b60d294f5d47cb7eb1811c4611e7a082b64424b632eb38b1d76b48acff5424604929a76cbb740662777bbccf8e62fb3d7ccc"
}