> there is no point in doing sliding window spidering on events once you have the history, from then on just open a subscriptiion
I am not following sorry. My crawler does this:
- Firehose: subscribes for new kind3s and kind0s
When a pubkey that is in the database is "promoted" (has acquired enough reputation)
- Query: query directly for its own kind3s and kind0s.
The query starts when one of these two conditions is met:
- One minute has passed since the last (with at least one pubkey)
- There are 100 pubkeys that have been promoted
Normally the crawler in production promoted ~100 pubkeys/day.
However, when I do tests locally I re-do the entire crawl which obviously is much more intensive, but I'll make sure I'm not using your relay for that sir.
pippellia / Pip the social graph guy
npub176…7vgup
2025-03-28 08:13:38
in reply to nevent1q…vtsx
Author Public Key
npub176p7sup477k5738qhxx0hk2n0cty2k5je5uvalzvkvwmw4tltmeqw7vgupPublished at
2025-03-28 08:13:38Event JSON
{
"id": "1bafc7c07b2e99f4bc8be29173cac0ec3fa646ab447521a650a160987f3c2578",
"pubkey": "f683e87035f7ad4f44e0b98cfbd9537e16455a92cd38cefc4cb31db7557f5ef2",
"created_at": 1743149618,
"kind": 1,
"tags": [
[
"p",
"f683e87035f7ad4f44e0b98cfbd9537e16455a92cd38cefc4cb31db7557f5ef2"
],
[
"p",
"ee9d979e7e1418a9f7472baf82dc96082a0d4251b77bcb170a6ecec93fadff64"
],
[
"p",
"036533caa872376946d4e4fdea4c1a0441eda38ca2d9d9417bb36006cbaabf58"
],
[
"p",
"fd208ee8c8f283780a9552896e4823cc9dc6bfd442063889577106940fd927c1"
],
[
"p",
"4c800257a588a82849d049817c2bdaad984b25a45ad9f6dad66e47d3b47e3b2f",
"wss://nostr.oxtr.dev"
],
[
"p",
"846ebf79a0a8813274ec9727490621ad423f16a3e474d7fd66e6a98bfe4e39a4",
"wss://theforest.nostr1.com"
],
[
"e",
"18588fbc8880edbe4b8d2e5bb59d3dde30ffec44e545c4ba7b0018544689b840",
"wss://nostr.oxtr.dev",
"reply",
"4c800257a588a82849d049817c2bdaad984b25a45ad9f6dad66e47d3b47e3b2f"
],
[
"e",
"7bb0400c790f4ac01df2e8f30d8b379d39d976fdb9e98b66426616f0a26e3db6",
"wss://theforest.nostr1.com",
"root",
"846ebf79a0a8813274ec9727490621ad423f16a3e474d7fd66e6a98bfe4e39a4"
]
],
"content": "\u003e there is no point in doing sliding window spidering on events once you have the history, from then on just open a subscriptiion\n\nI am not following sorry. My crawler does this:\n\n- Firehose: subscribes for new kind3s and kind0s\n\nWhen a pubkey that is in the database is \"promoted\" (has acquired enough reputation)\n\n- Query: query directly for its own kind3s and kind0s.\n\nThe query starts when one of these two conditions is met:\n- One minute has passed since the last (with at least one pubkey)\n- There are 100 pubkeys that have been promoted \n\nNormally the crawler in production promoted ~100 pubkeys/day.\n\nHowever, when I do tests locally I re-do the entire crawl which obviously is much more intensive, but I'll make sure I'm not using your relay for that sir.",
"sig": "aac464a5e0ea6dbb9f92494b00425c33c8ab7ef8206fb765dfad7715cc2bc061274b8a1b912f1f3e76549b7ceb8b58f5a84fe62f9260b117327f2b58de8940c2"
}