Hi, this is true. Although we do opt-out of AI scrapers using our robots.txt, it is still the Internet and relying on the platform for such protection is not 100% safe, no matter if you rely on GitHub or our platform. We recently had to block IP addresses that were relentlessly scraping our platform but _after_ they caused performance problems. ~n
P.S. WIRED published something on AI "content aggregators" ignoring robots.txt very recently: https://www.wired.com/story/perplexity-is-a-bullshit-machine/
Codeberg.org /
npub1fh…3ee83
2024-06-20 12:40:38
in reply to nevent1q…hg7f
Author Public Key
npub1fh736ae0xzk98s59wggazdpqg0g78mg2rwk7qf87ew96cqhc4lrst3ee83Published at
2024-06-20 12:40:38Event JSON
{
"id": "ec4283a69c39e2cd46bf50d15e7154fbec5dacb5b49a093d6b1531298dd28271",
"pubkey": "4dfd1d772f30ac53c2857211d1342043d1e3ed0a1bade024fecb8bac02f8afc7",
"created_at": 1718887238,
"kind": 1,
"tags": [
[
"p",
"9ec94f7955b3da50a74e045be41348a80872a774f4ff9c275ed51f7d6fbb4607"
],
[
"e",
"cf85850cffe2279fa9303d6319bc1cd3db8701ab024e2bd4ed8fb1a92039e12b",
"",
"root"
],
[
"proxy",
"https://social.anoxinon.de/@Codeberg/112648994032919650",
"web"
],
[
"p",
"4a474c6c8d7d75bde76b460cce8e839059df36222908317c5e440a8715aaf436"
],
[
"e",
"9cebe7566e06eac5e0cecc72ef6dace63130cbccd5e1c30c843cff9b66b0c5d7",
"",
"reply"
],
[
"p",
"a29e2cf1f76321f0668c964464075b1c738db8387d3b1fb59ab11059eaaebe41"
],
[
"p",
"94d2bd37a3070e9ac368ff30c6367a6b12c9e10413ae6d175d2cd918dc437c87"
],
[
"p",
"4dfd1d772f30ac53c2857211d1342043d1e3ed0a1bade024fecb8bac02f8afc7"
],
[
"p",
"50e2a7ef1309bd79c06561c8d736c5c9003f8767e41900b8416d6ee3a29f28db"
],
[
"p",
"f422916161d81780d6c0fedbc282e8da5f5bd4310635a85f3576b4ffec52763b"
],
[
"proxy",
"https://social.anoxinon.de/users/Codeberg/statuses/112648994032919650",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://social.anoxinon.de/users/Codeberg/statuses/112648994032919650",
"pink.momostr"
]
],
"content": "Hi, this is true. Although we do opt-out of AI scrapers using our robots.txt, it is still the Internet and relying on the platform for such protection is not 100% safe, no matter if you rely on GitHub or our platform. We recently had to block IP addresses that were relentlessly scraping our platform but _after_ they caused performance problems. ~n\n\nP.S. WIRED published something on AI \"content aggregators\" ignoring robots.txt very recently: https://www.wired.com/story/perplexity-is-a-bullshit-machine/",
"sig": "3bf22f80747361e4bdbc817ac500dec3c94edd44ea6e3383dca51568e35d9cab1312acd280462df5e4445ee682e241cb353f6501e9f8e41f09500ac6a2bebd36"
}