The unwritten rules of respecting robots.txt and not scooping up everything have already crumpled due to AI scraping.
https://www.businessinsider.com/meta-web-crawler-bots-robots-txt-ai-2024-8
Seems to be the trend that if you can access it it will be used. Encrypt and deny access might become the new reality, unfortunately.
petri / Petri Kajander
npub1us…nn3mj
2024-08-23 18:03:22
in reply to nevent1q…yq7v
Author Public Key
npub1ust7u0v3qffejwhqee45r49zgcyewrcn99vdwkednd356c9resyqtnn3mjPublished at
2024-08-23 18:03:22Event JSON
{
"id": "d93d57f95fdee78f8ed8856e906a92835dc87ac9c35541f281bfaa7f0b286e2d",
"pubkey": "e417ee3d910253993ae0ce6b41d4a24609970f132958d75b2d9b634d60a3cc08",
"created_at": 1724436202,
"kind": 1,
"tags": [
[
"e",
"f480c8ac815b14cebd003c146ff9333e995151653c0c4275c3fe416e03b93960",
"",
"root"
],
[
"e",
"610b8ba4de8892f2fea6bccb01a7d461bd64543a8370070b1a9000c7af63803a",
"",
"reply"
],
[
"p",
"de7ecd1e2976a6adb2ffa5f4db81a7d812c8bb6698aa00dcf1e76adb55efd645"
],
[
"p",
"ccaa58e37c99c85bc5e754028a718bd46485e5d3cb3345691ecab83c755d48cc"
],
[
"p",
"d49a9023a21dba1b3c8306ca369bf3243d8b44b8f0b6d1196607f7b0990fa8df"
],
[
"client",
"Nostur",
"31990:9be0be0fc079548233231614e4e1efc9f28b0db398011efeecf05fe570e5dd33:1685868693432"
]
],
"content": "The unwritten rules of respecting robots.txt and not scooping up everything have already crumpled due to AI scraping.\n\nhttps://www.businessinsider.com/meta-web-crawler-bots-robots-txt-ai-2024-8\n\nSeems to be the trend that if you can access it it will be used. Encrypt and deny access might become the new reality, unfortunately.",
"sig": "314d945373b627c13575c043085d46b229a6f59fbce24747a4566b85f93d42a29c79ef7ae797aaca307ce85401e07f0f866a5c19dee877869ba4394ecd321ec3"
}