yeah, they are only reading, not writing, so all that would really be required is to slow them down
probably could split the direct followed vs follows of follows into two tiers also so that the second level of depth get less service quality
probably maybe should work on it today because this problem with scrapers is fairly annoying
it's only nostr WoT spiders at this point, i wish i could just suggest to the spider people to just open a subscription for these and catch all the new stuff live, that would be more efficient for them and less expensive for relay operators
at this point it's fairly early in the game for that stuff so maybe some education of WoT devs might help...
cc: pippellia (npub176p…vgup) there is no point in doing sliding window spidering on events once you have the history, from then on just open a subscriptiion
this would be more friendly from the AI spiders if they had a way to just catch new updates, but unless dey pay me! i'm just gonna tarpit them
mleku
npub1fj…mleku
2025-03-28 06:25:12
in reply to nevent1q…tzvh
Author Public Key
npub1fjqqy4a93z5zsjwsfxqhc2764kvykfdyttvldkkkdera8dr78vhsmmlekuPublished at
2025-03-28 06:25:12Event JSON
{
"id": "18588fbc8880edbe4b8d2e5bb59d3dde30ffec44e545c4ba7b0018544689b840",
"pubkey": "4c800257a588a82849d049817c2bdaad984b25a45ad9f6dad66e47d3b47e3b2f",
"created_at": 1743143112,
"kind": 1,
"tags": [
[
"e",
"7bb0400c790f4ac01df2e8f30d8b379d39d976fdb9e98b66426616f0a26e3db6",
"",
"root",
"846ebf79a0a8813274ec9727490621ad423f16a3e474d7fd66e6a98bfe4e39a4"
],
[
"e",
"3abd9f77f8dcab4e05053f7d4843ae509f73355cf4e2dfda2eac7552a7187a93",
"wss://theforest.nostr1.com/",
"reply",
"fd208ee8c8f283780a9552896e4823cc9dc6bfd442063889577106940fd927c1"
],
[
"p",
"f683e87035f7ad4f44e0b98cfbd9537e16455a92cd38cefc4cb31db7557f5ef2"
],
[
"p",
"ee9d979e7e1418a9f7472baf82dc96082a0d4251b77bcb170a6ecec93fadff64"
],
[
"p",
"036533caa872376946d4e4fdea4c1a0441eda38ca2d9d9417bb36006cbaabf58"
],
[
"p",
"fd208ee8c8f283780a9552896e4823cc9dc6bfd442063889577106940fd927c1"
]
],
"content": "yeah, they are only reading, not writing, so all that would really be required is to slow them down\n\nprobably could split the direct followed vs follows of follows into two tiers also so that the second level of depth get less service quality\n\nprobably maybe should work on it today because this problem with scrapers is fairly annoying\n\nit's only nostr WoT spiders at this point, i wish i could just suggest to the spider people to just open a subscription for these and catch all the new stuff live, that would be more efficient for them and less expensive for relay operators\n\nat this point it's fairly early in the game for that stuff so maybe some education of WoT devs might help... \n\ncc: nostr:npub176p7sup477k5738qhxx0hk2n0cty2k5je5uvalzvkvwmw4tltmeqw7vgup there is no point in doing sliding window spidering on events once you have the history, from then on just open a subscriptiion\n\nthis would be more friendly from the AI spiders if they had a way to just catch new updates, but unless dey pay me! i'm just gonna tarpit them",
"sig": "87bab33b936187d3aa866d56a29e49258eea97a5b8156f7050f279b4e336b96b8ff5baa2da3f87c2fd0e9af3e6f8fc758104f3a02afc7e4ec03cae9a79d4588a"
}