I think they need to.
If they refuse to respect robots.txt then if an instance goes to court then that should be used as evidence against them as it clearly indicates the publisher doesn't want their content used in that way rather than it'd publicly available which it isn't
Peter Mount /
npub1hh…fwhy6
2024-07-31 13:54:03
in reply to nevent1q…3vcr
Author Public Key
npub1hhn2u26m72u232jj3rzcu5vdnqdhk0xp8prwmut3596au93mc5zsgfwhy6Published at
2024-07-31 13:54:03Event JSON
{
"id": "1f6927a73b8b3ca67761afbd57539835d63c49978c7a6b46ce3cc10bd1c331a9",
"pubkey": "bde6ae2b5bf2b8a8aa5288c58e518d981b7b3cc13846edf171a175de163bc505",
"created_at": 1722434043,
"kind": 1,
"tags": [
[
"p",
"02b82140c344de6e1ced44237eca488cecec57453c22206e52b15d812b93042e"
],
[
"e",
"b3386eb7abc70fa1475adb112da67b56a038f6ae2b6165f0d4bce549d8d3a957",
"",
"root",
"02b82140c344de6e1ced44237eca488cecec57453c22206e52b15d812b93042e"
],
[
"proxy",
"https://area51.social/@peter/112881437480357081",
"web"
],
[
"e",
"9756a1601fb1047c24c846d20bf7e8fb4a67132d444420a42cc824fd6bfcee87",
"",
"reply",
"02b82140c344de6e1ced44237eca488cecec57453c22206e52b15d812b93042e"
],
[
"proxy",
"https://area51.social/users/peter/statuses/112881437480357081",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://area51.social/users/peter/statuses/112881437480357081",
"pink.momostr"
],
[
"-"
]
],
"content": "I think they need to.\n\n If they refuse to respect robots.txt then if an instance goes to court then that should be used as evidence against them as it clearly indicates the publisher doesn't want their content used in that way rather than it'd publicly available which it isn't",
"sig": "8c9f843919e1659af354c27505d45a92f4fa3ccb3906ba3074435aa585077269e7139db1e6a0afdc1f2c725789466da9bc02735caaca978d8871b87458c4943b"
}