š
Original date posted:2021-03-01
š Original message:Something to consider adding to this proposal is to keep the idea of
pruning - i.e. retain a sequentially uninterrupted number of the most
recent blocks.
Many users do not run a node for entirely altruistic reasons - they do so,
at least in part, because it allows them to use their wallets privately.
Without this ability, I think the number of users willing to run their node
in this configuration might be reduced.
Another related thought is to have a decreasing density over blocks over
time as you go backwards towards genesis, in order for the data density of
the storage to match the actual usage of the network, in which (I would
imagine) more recent blocks are more heavily requested than early ones.
Craig
On Sun, Feb 28, 2021 at 10:18 AM Leo Wandersleb via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
> Only headers need to be downloaded sequentially so downloading relevant
> blocks
> from one node is totally possible with gaps in between.
>
> On 2/27/21 4:10 AM, Igor Cota via bitcoin-dev wrote:
> > Hi Keagan,
> >
> > I had a very similar idea. The only difference being for the node to
> decide on
> > a range of blocks to keep beforehand, rather than making the decision
> > block-by-block like you suggest.
> >
> > I felt the other nodes would be better served by ranges due to the
> sequential
> > nature of IBD. Perhaps this would be computationally lighter as well.
> >
> > I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT
> > scheme to solve this same problem.
> >
> > Cheers,
> > Igor
> >
> > [1] https://arxiv.org/abs/1902.02174
> >
> > On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev
> > <bitcoin-dev at lists.linuxfoundation.org
> > <mailto:bitcoin-dev at lists.linuxfoundation.org>> wrote:
> >
> > Hi all,
> >
> > I've been thinking for quite some time about the problem of pruned
> nodes
> > and ongoing storage costs for full nodes. One of the things that
> strikes
> > me as odd is that we only really have two settings.
> >
> > A. Prune everything except the most recent blocks, down to the cache
> size
> > B. Keep everything since genesis
> >
> > From my observations and conversations with various folks in the
> > community, they would like to be able to run a "partially" pruned
> node to
> > help bear the load of bootstrapping other nodes and helping with data
> > redundancy in the network, but would prefer to not dedicate hundreds
> of
> > Gigabytes of storage space to the cause.
> >
> > This led me to the idea that a node could randomly prune some of the
> > blocks from history if it passed some predicate. A rough sketch of
> this
> > would look as follows.
> >
> > 1. At node startup, it would generate a random seed, this would be
> unique
> > to the node but not necessary that it be cryptographically secure.
> > 2. In the node configuration it would also carry a "threshold"
> expressed
> > as some percentage of blocks it wanted to keep.
> > 3. As IBD occurs, based off of the threshold, the block hash, and the
> > node's unique seed, the node would either decide to prune the data
> or keep
> > it. The uniqueness of the node's hash should ensure that no block is
> > systematically overrepresented in the set of nodes choosing this
> storage
> > scheme.
> > 4. Once the node's IBD is complete it would advertise this as a peer
> > service, advertising its seed and threshold, so that nodes could
> > deterministically deduce which of its peers had which blocks.
> >
> > The goals are to increase data redundancy in a way that more
> uniformly
> > shares the load across nodes, alleviating some of the pressure of
> full
> > archive nodes on the IBD problem. I am working on a draft BIP for
> this
> > proposal but figured I would submit it as a high level idea in case
> anyone
> > had any feedback on the initial design before I go into specification
> > levels of detail.
> >
> > If you have thoughts on
> >
> > A. The protocol design itself
> > B. The barriers to put this kind of functionality into Core
> >
> > I would love to hear from you,
> >
> > Cheers,
> > Keagan
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev at lists.linuxfoundation.org
> > <mailto:bitcoin-dev at lists.linuxfoundation.org>
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> >
> >
> >
> > --
> > *Igor Cota*
> > Codex Apertus d.o.o.
> >
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev at lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20210301/57cba055/attachment-0001.html>;
Craig Raw [ARCHIVE] /
npub1qvā¦9t9zs
2023-06-07 18:29:39
in reply to nevent1qā¦2u24
Author Public Key
npub1qvg3k0w2c3ggh4wumg3pkzetfeqe9y4wgq7x5aaxz4ypjvap9vksr9t9zsPublished at
2023-06-07 18:29:39Event JSON
{
"id": "ffac27804b0c4488ee5a60581c8a4f81765341c0196a950c7382b4b42ca6d972",
"pubkey": "03111b3dcac4508bd5dcda221b0b2b4e419292ae403c6a77a615481933a12b2d",
"created_at": 1686162579,
"kind": 1,
"tags": [
[
"e",
"d10f4fcf03fd27c60326979d0ee1ad55ef89ead7d2ab3dbef2171c629eed1636",
"",
"reply"
],
[
"p",
"a23dbf6c6cc83e14cc3df4e56cc71845f611908084cfe620e83e40c06ccdd3d0"
]
],
"content": "š
Original date posted:2021-03-01\nš Original message:Something to consider adding to this proposal is to keep the idea of\npruning - i.e. retain a sequentially uninterrupted number of the most\nrecent blocks.\n\nMany users do not run a node for entirely altruistic reasons - they do so,\nat least in part, because it allows them to use their wallets privately.\nWithout this ability, I think the number of users willing to run their node\nin this configuration might be reduced.\n\nAnother related thought is to have a decreasing density over blocks over\ntime as you go backwards towards genesis, in order for the data density of\nthe storage to match the actual usage of the network, in which (I would\nimagine) more recent blocks are more heavily requested than early ones.\n\nCraig\n\nOn Sun, Feb 28, 2021 at 10:18 AM Leo Wandersleb via bitcoin-dev \u003c\nbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\n\u003e Only headers need to be downloaded sequentially so downloading relevant\n\u003e blocks\n\u003e from one node is totally possible with gaps in between.\n\u003e\n\u003e On 2/27/21 4:10 AM, Igor Cota via bitcoin-dev wrote:\n\u003e \u003e Hi Keagan,\n\u003e \u003e\n\u003e \u003e I had a very similar idea. The only difference being for the node to\n\u003e decide on\n\u003e \u003e a range of blocks to keep beforehand, rather than making the decision\n\u003e \u003e block-by-block like you suggest.\n\u003e \u003e\n\u003e \u003e I felt the other nodes would be better served by ranges due to the\n\u003e sequential\n\u003e \u003e nature of IBD. Perhaps this would be computationally lighter as well.\n\u003e \u003e\n\u003e \u003e I also encourage you to read Ryosuke Abe's paper [1] that proposes a DHT\n\u003e \u003e scheme to solve this same problem.\n\u003e \u003e\n\u003e \u003e Cheers,\n\u003e \u003e Igor\n\u003e \u003e\n\u003e \u003e [1] https://arxiv.org/abs/1902.02174\n\u003e \u003e\n\u003e \u003e On Fri, 26 Feb 2021 at 21:57, Keagan McClelland via bitcoin-dev\n\u003e \u003e \u003cbitcoin-dev at lists.linuxfoundation.org\n\u003e \u003e \u003cmailto:bitcoin-dev at lists.linuxfoundation.org\u003e\u003e wrote:\n\u003e \u003e\n\u003e \u003e Hi all,\n\u003e \u003e\n\u003e \u003e I've been thinking for quite some time about the problem of pruned\n\u003e nodes\n\u003e \u003e and ongoing storage costs for full nodes. One of the things that\n\u003e strikes\n\u003e \u003e me as odd is that we only really have two settings.\n\u003e \u003e\n\u003e \u003e A. Prune everything except the most recent blocks, down to the cache\n\u003e size\n\u003e \u003e B. Keep everything since genesis\n\u003e \u003e\n\u003e \u003e From my observations and conversations with various folks in the\n\u003e \u003e community, they would like to be able to run a \"partially\" pruned\n\u003e node to\n\u003e \u003e help bear the load of bootstrapping other nodes and helping with data\n\u003e \u003e redundancy in the network, but would prefer to not dedicate hundreds\n\u003e of\n\u003e \u003e Gigabytes of storage space to the cause.\n\u003e \u003e\n\u003e \u003e This led me to the idea that a node could randomly prune some of the\n\u003e \u003e blocks from history if it passed some predicate. A rough sketch of\n\u003e this\n\u003e \u003e would look as follows.\n\u003e \u003e\n\u003e \u003e 1. At node startup, it would generate a random seed, this would be\n\u003e unique\n\u003e \u003e to the node but not necessary that it be cryptographically secure.\n\u003e \u003e 2. In the node configuration it would also carry a \"threshold\"\n\u003e expressed\n\u003e \u003e as some percentage of blocks it wanted to keep.\n\u003e \u003e 3. As IBD occurs, based off of the threshold, the block hash, and the\n\u003e \u003e node's unique seed, the node would either decide to prune the data\n\u003e or keep\n\u003e \u003e it. The uniqueness of the node's hash should ensure that no block is\n\u003e \u003e systematically overrepresented in the set of nodes choosing this\n\u003e storage\n\u003e \u003e scheme.\n\u003e \u003e 4. Once the node's IBD is complete it would advertise this as a peer\n\u003e \u003e service, advertising its seed and threshold, so that nodes could\n\u003e \u003e deterministically deduce which of its peers had which blocks.\n\u003e \u003e\n\u003e \u003e The goals are to increase data redundancy in a way that more\n\u003e uniformly\n\u003e \u003e shares the load across nodes, alleviating some of the pressure of\n\u003e full\n\u003e \u003e archive nodes on the IBD problem. I am working on a draft BIP for\n\u003e this\n\u003e \u003e proposal but figured I would submit it as a high level idea in case\n\u003e anyone\n\u003e \u003e had any feedback on the initial design before I go into specification\n\u003e \u003e levels of detail.\n\u003e \u003e\n\u003e \u003e If you have thoughts on\n\u003e \u003e\n\u003e \u003e A. The protocol design itself\n\u003e \u003e B. The barriers to put this kind of functionality into Core\n\u003e \u003e\n\u003e \u003e I would love to hear from you,\n\u003e \u003e\n\u003e \u003e Cheers,\n\u003e \u003e Keagan\n\u003e \u003e _______________________________________________\n\u003e \u003e bitcoin-dev mailing list\n\u003e \u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e \u003e \u003cmailto:bitcoin-dev at lists.linuxfoundation.org\u003e\n\u003e \u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e \u003e\n\u003e \u003e\n\u003e \u003e\n\u003e \u003e --\n\u003e \u003e *Igor Cota*\n\u003e \u003e Codex Apertus d.o.o.\n\u003e \u003e\n\u003e \u003e _______________________________________________\n\u003e \u003e bitcoin-dev mailing list\n\u003e \u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e \u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20210301/57cba055/attachment-0001.html\u003e",
"sig": "bf5c9d682aa48f01b443d2e5fa58712604043136c461e997f8b126d43e0b51fdd57ac9d0ff844216e1cf9e2d66214a933060d27eeecf713514d595bdb94eefe1"
}