npub175d03ews4x6at9pm466wu8pm6l54gxddqfkcxcwmezxy2evretzqde0wl8 (npub175d…0wl8) I’m afraid that robots dot txt isn’t going to do anything for you at this point. This is because the LLM companies regularly disguise the true origin of their scraping by routing their traffic through hosting companies, VPNs / proxies, TOR, & other similar methods.
So, 1) use a password and username on things you want to keep away from the bots (simple HTTP auth works well), and 2) deploy IP blocks on the LLM companies & their intermediaries.
Denis Warburton /
npub1gw…hfu48
2024-10-16 21:57:26
in reply to nevent1q…uayn
Author Public Key
npub1gw6zu5t0csg4z6aj58urm7h9q529ehnav3zp3nplq8j6fhcj6lzsuhfu48Published at
2024-10-16 21:57:26Event JSON
{
"id": "b6a8167e273d9f55b046b2f840be45ba80ea523febb3e1b9e2b2a42652f623c9",
"pubkey": "43b42e516fc411516bb2a1f83dfae505145cde7d644418cc3f01e5a4df12d7c5",
"created_at": 1729115846,
"kind": 1,
"tags": [
[
"p",
"f51af8e5d0a9b5d5943baeb4ee1c3bd7e95419ad026d8361dbc88c456583cac4",
"wss://relay.mostr.pub"
],
[
"p",
"3f896d1be872ba9111700e28c7bbdd7b99a92b18a2d30da79bafcd0755f30fe9",
"wss://relay.mostr.pub"
],
[
"e",
"6062e77066309ecd6df1956c7fa212cc45b501ba41face3f30bf2b1d9358cf57",
"wss://relay.mostr.pub",
"reply"
],
[
"proxy",
"https://digipres.club/users/artandtechnic/statuses/113319336109195527",
"activitypub"
]
],
"content": "nostr:npub175d03ews4x6at9pm466wu8pm6l54gxddqfkcxcwmezxy2evretzqde0wl8 I’m afraid that robots dot txt isn’t going to do anything for you at this point. This is because the LLM companies regularly disguise the true origin of their scraping by routing their traffic through hosting companies, VPNs / proxies, TOR, \u0026 other similar methods.\n\nSo, 1) use a password and username on things you want to keep away from the bots (simple HTTP auth works well), and 2) deploy IP blocks on the LLM companies \u0026 their intermediaries.",
"sig": "d5d7d798414e4560df510d8565c7f445f3b83e8943d3f4a6f63a0c57edf97cc7cb4d0a2ea4cc9b2af78140a6b7c4d854a6068114129a6e5ba6ba7033a2d173ab"
}