📅 Original date posted:2021-02-27
📝 Original message:Hi Keagan,
I had a very similar idea. The only difference being for the node to decide
on a range of blocks to keep beforehand, rather than making the decision
block-by-block like you suggest.
I felt the other nodes would be better served by ranges due to the
sequential nature of IBD. Perhaps this would be computationally lighter as
well.
I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT
scheme to solve this same problem.
Cheers,
Igor
[1] https://arxiv.org/abs/1902.02174
On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
> Hi all,
>
> I've been thinking for quite some time about the problem of pruned nodes
> and ongoing storage costs for full nodes. One of the things that strikes me
> as odd is that we only really have two settings.
>
> A. Prune everything except the most recent blocks, down to the cache size
> B. Keep everything since genesis
>
> From my observations and conversations with various folks in the
> community, they would like to be able to run a "partially" pruned node to
> help bear the load of bootstrapping other nodes and helping with data
> redundancy in the network, but would prefer to not dedicate hundreds of
> Gigabytes of storage space to the cause.
>
> This led me to the idea that a node could randomly prune some of the
> blocks from history if it passed some predicate. A rough sketch of this
> would look as follows.
>
> 1. At node startup, it would generate a random seed, this would be unique
> to the node but not necessary that it be cryptographically secure.
> 2. In the node configuration it would also carry a "threshold" expressed
> as some percentage of blocks it wanted to keep.
> 3. As IBD occurs, based off of the threshold, the block hash, and the
> node's unique seed, the node would either decide to prune the data or keep
> it. The uniqueness of the node's hash should ensure that no block is
> systematically overrepresented in the set of nodes choosing this storage
> scheme.
> 4. Once the node's IBD is complete it would advertise this as a peer
> service, advertising its seed and threshold, so that nodes could
> deterministically deduce which of its peers had which blocks.
>
> The goals are to increase data redundancy in a way that more uniformly
> shares the load across nodes, alleviating some of the pressure of full
> archive nodes on the IBD problem. I am working on a draft BIP for this
> proposal but figured I would submit it as a high level idea in case anyone
> had any feedback on the initial design before I go into specification
> levels of detail.
>
> If you have thoughts on
>
> A. The protocol design itself
> B. The barriers to put this kind of functionality into Core
>
> I would love to hear from you,
>
> Cheers,
> Keagan
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
--
*Igor Cota*
Codex Apertus d.o.o.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20210227/c79266b3/attachment.html>;
Igor Cota [ARCHIVE] /
npub1f6…jcphu
2023-06-07 18:29:14
in reply to nevent1q…e00w
Author Public Key
npub1f6lcnu5cs3nhzxnw7pmgq9lny6tpa93qdq932ay7epqjqpn5rq6s8jcphuPublished at
2023-06-07 18:29:14Event JSON
{
"id": "352a17a94681b1bbefbfe5f537d629b801c6cfd5dd5b901122863486fe391739",
"pubkey": "4ebf89f2988467711a6ef0768017f326961e9620680b15749ec8412006741835",
"created_at": 1686162554,
"kind": 1,
"tags": [
[
"e",
"0b4750c28487df8182e6a10b350fda3a1725de3982295ac3aa3cae69acb102b3",
"",
"root"
],
[
"e",
"c6105094ea0831792676820434e619a3665acfe706e2e74e05cf7289346cd338",
"",
"reply"
],
[
"p",
"4aa66c838f2cc55afbd46335fb2520507e2030601f773b28b96b76ec60a2f489"
]
],
"content": "📅 Original date posted:2021-02-27\n📝 Original message:Hi Keagan,\n\nI had a very similar idea. The only difference being for the node to decide\non a range of blocks to keep beforehand, rather than making the decision\nblock-by-block like you suggest.\n\nI felt the other nodes would be better served by ranges due to the\nsequential nature of IBD. Perhaps this would be computationally lighter as\nwell.\n\nI also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT\nscheme to solve this same problem.\n\nCheers,\nIgor\n\n[1] https://arxiv.org/abs/1902.02174\n\nOn Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev \u003c\nbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\n\u003e Hi all,\n\u003e\n\u003e I've been thinking for quite some time about the problem of pruned nodes\n\u003e and ongoing storage costs for full nodes. One of the things that strikes me\n\u003e as odd is that we only really have two settings.\n\u003e\n\u003e A. Prune everything except the most recent blocks, down to the cache size\n\u003e B. Keep everything since genesis\n\u003e\n\u003e From my observations and conversations with various folks in the\n\u003e community, they would like to be able to run a \"partially\" pruned node to\n\u003e help bear the load of bootstrapping other nodes and helping with data\n\u003e redundancy in the network, but would prefer to not dedicate hundreds of\n\u003e Gigabytes of storage space to the cause.\n\u003e\n\u003e This led me to the idea that a node could randomly prune some of the\n\u003e blocks from history if it passed some predicate. A rough sketch of this\n\u003e would look as follows.\n\u003e\n\u003e 1. At node startup, it would generate a random seed, this would be unique\n\u003e to the node but not necessary that it be cryptographically secure.\n\u003e 2. In the node configuration it would also carry a \"threshold\" expressed\n\u003e as some percentage of blocks it wanted to keep.\n\u003e 3. As IBD occurs, based off of the threshold, the block hash, and the\n\u003e node's unique seed, the node would either decide to prune the data or keep\n\u003e it. The uniqueness of the node's hash should ensure that no block is\n\u003e systematically overrepresented in the set of nodes choosing this storage\n\u003e scheme.\n\u003e 4. Once the node's IBD is complete it would advertise this as a peer\n\u003e service, advertising its seed and threshold, so that nodes could\n\u003e deterministically deduce which of its peers had which blocks.\n\u003e\n\u003e The goals are to increase data redundancy in a way that more uniformly\n\u003e shares the load across nodes, alleviating some of the pressure of full\n\u003e archive nodes on the IBD problem. I am working on a draft BIP for this\n\u003e proposal but figured I would submit it as a high level idea in case anyone\n\u003e had any feedback on the initial design before I go into specification\n\u003e levels of detail.\n\u003e\n\u003e If you have thoughts on\n\u003e\n\u003e A. The protocol design itself\n\u003e B. The barriers to put this kind of functionality into Core\n\u003e\n\u003e I would love to hear from you,\n\u003e\n\u003e Cheers,\n\u003e Keagan\n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n\n\n-- \n*Igor Cota*\nCodex Apertus d.o.o.\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20210227/c79266b3/attachment.html\u003e",
"sig": "14bce1da3fdb5b672f68c7f914583fda96ce1384a372b45dceb2c29c248c64fc7e6ddc2e37789e39828277962031c2871ce153740fa0078456cbb0dea25da3ad"
}