📅 Original date posted:2017-04-17
📝 Original message:While I fully agree with the intent (increasing full nodes so a big
miner waking up in a bad mood can't threaten the world any longer every
day as it is now) I am not sure to get the interest of this proposal,
because:
- it's probably not a good idea to encourage the home users to run full
nodes, there are many people running servers far from their capacity
that could easily run efficient full nodes
- if someone can't allocate 100 GB today to run a full node, then we
can't expect him to allocate more in the future
- the download time is a real concern
- this proposal is a kind of reinventing torrents, while limiting the
number of connections to something not efficient at all, I don't see why
something that is proven to be super efficient (torrents) would be
needed to be reinvented, I am not saying that it should be used as the
bittorrent network is doing but the concepts can be reused
- I don't get at all the concept of "archival" nodes since it's another
useless step toward centralization
I think the only way to increase full nodes it to design an incentive
for people to run them
Le 17/04/2017 à 08:54, David Vorick via bitcoin-dev a écrit :
> *Rationale:*
>
> A node that stores the full blockchain (I will use the term archival
> node) requires over 100GB of disk space, which I believe is one of the
> most significant barriers to more people running full nodes. And I
> believe the ecosystem would benefit substantially if more users were
> running full nodes.
>
> The best alternative today to storing the full blockchain is to run a
> pruned node, which keeps only the UTXO set and throws away already
> verified blocks. The operator of the pruned node is able to enjoy the
> full security benefits of a full node, but is essentially leeching the
> network, as they performed a large download likely without
> contributing anything back.
>
> This puts more pressure on the archival nodes, as the archival nodes
> need to pick up the slack and help new nodes bootstrap to the network.
> As the pressure on archival nodes grows, fewer people will be able to
> actually run archival nodes, and the situation will degrade. The
> situation would likely become problematic quickly if bitcoin-core were
> to ship with the defaults set to a pruned node.
>
> Even further, the people most likely to care about saving 100GB of
> disk space are also the people least likely to care about some extra
> bandwidth usage. For datacenter nodes, and for nodes doing lots of
> bandwidth, the bandwidth is usually the biggest cost of running the
> node. For home users however, as long as they stay under their
> bandwidth cap, the bandwidth is actually free. Ideally, new nodes
> would be able to bootstrap from nodes that do not have to pay for
> their bandwidth, instead of needing to rely on a decreasing percentage
> of heavy-duty archival nodes.
>
> I have (perhaps incorrectly) identified disk space consumption as the
> most significant factor in your average user choosing to run a pruned
> node or a lite client instead of a full node. The average user is not
> typically too worried about bandwidth, and is also not typically too
> worried about initial blockchain download time. But the 100GB hit to
> your disk space can be a huge psychological factor, especially if your
> hard drive only has 500GB available in the first place, and 250+ GB is
> already consumed by other files you have.
>
> I believe that improving the disk usage situation would greatly
> benefit decentralization, especially if it could be done without
> putting pressure on archival nodes.
>
> *Small Nodes Proposal:*
>
> I propose an alternative to the pruned node that does not put undue
> pressure on archival nodes, and would be acceptable and non-risky to
> ship as a default in bitcoin-core. For lack of a better name, I'll
> call this new type of node a 'small node'. The intention is that
> bitcoin-core would eventually ship 'small nodes' by default, such that
> the expected amount of disk consumption drops from today's 100+ GB to
> less than 30 GB.
>
> My alternative proposal has the following properties:
>
> + Full nodes only need to store ~20% of the blockchain
> + With very high probability, a new node will be able to recover the
> entire blockchain by connecting to 6 random small node peers.
> + An attacker that can eliminate a chosen+ 95% of the full nodes
> running today will be unable to prevent new nodes from downloading the
> full blockchain, even if the attacker is also able to eliminate all
> archival nodes. (assuming all nodes today were small nodes instead of
> archival nodes)
>
> Method:
>
> A small node will pick an index [5, 256). This index is that node's
> permanent index. When storing a block, instead of storing the full
> block, the node will use Reed-Solomon coding to erasure code the block
> using a 5-of-256 scheme. The result will be 256 pieces that are 20% of
> the size of the block each. The node picks the piece that corresponds
> to its index, and stores that instead. (Indexes 0-4 are reserved for
> archival nodes - explained later)
>
> The node is now storing a fragment of every block. Alone, this
> fragment cannot be used to recover any piece of the blockchain.
> However, when paired with any 5 unique fragments (fragments of the
> same index will not be unique), the full block can be recovered.
>
> Nodes can optionally store more than 1 fragment each. At 5 fragments,
> the node becomes a full archival node, and the chosen indexes should
> be 0-4. This is advantageous for the archival node as the encoded data
> for the first 5 indexes will actually be identical to the block itself
> - there is no computational overhead for selecting the first indexes.
> There is also no need to choose random indexes, because the full block
> can be recovered no matter which indexes are chosen.
>
> When connecting to new peers, the indexes of each peer needs to be
> known. Once peers totaling 5 unique indexes are discovered, blockchain
> download can begin. Connecting to just 5 small node peers provides a
> >95% chance of getting 5 uniques, with exponentially improving odds of
> success as you connect to more peers. Connecting to a single archive
> node guarantees that any gaps can be filled.
>
> A good encoder should be able to turn a block into a 5-of-256 piece
> set in under 10 milliseconds using a single core on a standard
> consumer desktop. This should not slow down initial blockchain
> download substantially, though the overhead is more than a rounding error.
>
> *DoS Prevention:*
>
> A malicious node may provide garbage data instead of the actual piece.
> Given just the garbage data and 4 other correct pieces, it is
> impossible (best I know anyway) to tell which piece is the garbage piece.
>
> One option in this case would be to seek out an archival node that
> could verify the correctness of the pieces, and identify the malicious
> node.
>
> Another option would be to have the small nodes store a cryptographic
> checksum of each piece. Obtaining the cryptographic checksum for all
> 256 pieces would incur a nontrivial amount of hashing (post segwit, as
> much as 100MB of extra hashing per block), and would require an
> additional ~4kb of storage per block. The hashing overhead here may be
> prohibitive.
>
> Another solution would be to find additional pieces and brute-force
> combinations of 5 until a working combination was discovered. Though
> this sounds nasty, it should take less than five seconds of
> computation to find the working combination given 5 correct pieces and
> 2 incorrect pieces. This computation only needs to be performed once
> to identify the malicious peers.
>
> I also believe that alternative erasure coding schemes exist which
> actually are able to identify the bad pieces given sufficient good
> pieces, however I don't know if they have the same computational
> performance as the best Reed-Solomon coding implementations.
>
> *Deployment:*
>
> Small nodes are completely useless unless the critical mass of 5
> pieces can be obtained. The first version that supports small node
> block downloads should default everyone to an archival node (meaning
> indexes 0-4 are used)
>
> Once there are enough small-node-enabled archive nodes, the default
> can be switched so that nodes only have a single index by default. In
> the first few days, when there are only a few small nodes, the
> previously-deployed archival nodes can help fill in the gaps, and the
> small nodes can be useful for blockchain download right away.
>
> ----------------------------------
>
> This represents a non-trivial amount of code, but I believe that the
> result would be a non-trivial increase in the percentage of users
> running full nodes, and a healthier overall network.
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
--
Zcash wallets made simple: https://github.com/Ayms/zcash-wallets
Bitcoin wallets made simple: https://github.com/Ayms/bitcoin-wallets
Get the torrent dynamic blocklist: http://peersm.com/getblocklist
Check the 10 M passwords list: http://peersm.com/findmyass
Anti-spies and private torrents, dynamic blocklist: http://torrent-live.org
Peersm : http://www.peersm.com
torrent-live: https://github.com/Ayms/torrent-live
node-Tor : https://www.github.com/Ayms/node-Tor
GitHub : https://www.github.com/Ayms
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170417/90387873/attachment-0001.html>;
Aymeric Vitte [ARCHIVE] /
npub15f…g5grz
2023-06-07 18:00:20
in reply to nevent1q…dlr8
Author Public Key
npub15fc36esk6dy2x4ptk2ner209rl9u0d736gr99edtu9knu9uny80s4g5grzPublished at
2023-06-07 18:00:20Event JSON
{
"id": "4b83e6f5e515504c22a268233bb6cf8d6cc55853db52989fbcaed7fa5655815c",
"pubkey": "a2711d6616d348a3542bb2a791a9e51fcbc7b7d1d20652e5abe16d3e179321df",
"created_at": 1686160820,
"kind": 1,
"tags": [
[
"e",
"9e0bce6c49b056a94ef9193a53941e860a13142ae7014703964955e2e8de52b1",
"",
"root"
],
[
"e",
"8cab176e14c03adbffa20e7fad5e990c719eba874cdf6f22e383c177ed899416",
"",
"reply"
],
[
"p",
"2c450c87d1e160ec6a769c39cfb12a3c52dac051b06e101e1125095ce50f31d7"
]
],
"content": "📅 Original date posted:2017-04-17\n📝 Original message:While I fully agree with the intent (increasing full nodes so a big\nminer waking up in a bad mood can't threaten the world any longer every\nday as it is now) I am not sure to get the interest of this proposal,\nbecause:\n\n- it's probably not a good idea to encourage the home users to run full\nnodes, there are many people running servers far from their capacity\nthat could easily run efficient full nodes\n\n- if someone can't allocate 100 GB today to run a full node, then we\ncan't expect him to allocate more in the future\n\n- the download time is a real concern\n\n- this proposal is a kind of reinventing torrents, while limiting the\nnumber of connections to something not efficient at all, I don't see why\nsomething that is proven to be super efficient (torrents) would be\nneeded to be reinvented, I am not saying that it should be used as the\nbittorrent network is doing but the concepts can be reused\n\n- I don't get at all the concept of \"archival\" nodes since it's another\nuseless step toward centralization\n\nI think the only way to increase full nodes it to design an incentive\nfor people to run them\n\n\nLe 17/04/2017 à 08:54, David Vorick via bitcoin-dev a écrit :\n\u003e *Rationale:*\n\u003e\n\u003e A node that stores the full blockchain (I will use the term archival\n\u003e node) requires over 100GB of disk space, which I believe is one of the\n\u003e most significant barriers to more people running full nodes. And I\n\u003e believe the ecosystem would benefit substantially if more users were\n\u003e running full nodes.\n\u003e\n\u003e The best alternative today to storing the full blockchain is to run a\n\u003e pruned node, which keeps only the UTXO set and throws away already\n\u003e verified blocks. The operator of the pruned node is able to enjoy the\n\u003e full security benefits of a full node, but is essentially leeching the\n\u003e network, as they performed a large download likely without\n\u003e contributing anything back.\n\u003e\n\u003e This puts more pressure on the archival nodes, as the archival nodes\n\u003e need to pick up the slack and help new nodes bootstrap to the network.\n\u003e As the pressure on archival nodes grows, fewer people will be able to\n\u003e actually run archival nodes, and the situation will degrade. The\n\u003e situation would likely become problematic quickly if bitcoin-core were\n\u003e to ship with the defaults set to a pruned node.\n\u003e\n\u003e Even further, the people most likely to care about saving 100GB of\n\u003e disk space are also the people least likely to care about some extra\n\u003e bandwidth usage. For datacenter nodes, and for nodes doing lots of\n\u003e bandwidth, the bandwidth is usually the biggest cost of running the\n\u003e node. For home users however, as long as they stay under their\n\u003e bandwidth cap, the bandwidth is actually free. Ideally, new nodes\n\u003e would be able to bootstrap from nodes that do not have to pay for\n\u003e their bandwidth, instead of needing to rely on a decreasing percentage\n\u003e of heavy-duty archival nodes.\n\u003e\n\u003e I have (perhaps incorrectly) identified disk space consumption as the\n\u003e most significant factor in your average user choosing to run a pruned\n\u003e node or a lite client instead of a full node. The average user is not\n\u003e typically too worried about bandwidth, and is also not typically too\n\u003e worried about initial blockchain download time. But the 100GB hit to\n\u003e your disk space can be a huge psychological factor, especially if your\n\u003e hard drive only has 500GB available in the first place, and 250+ GB is\n\u003e already consumed by other files you have.\n\u003e\n\u003e I believe that improving the disk usage situation would greatly\n\u003e benefit decentralization, especially if it could be done without\n\u003e putting pressure on archival nodes.\n\u003e\n\u003e *Small Nodes Proposal:*\n\u003e\n\u003e I propose an alternative to the pruned node that does not put undue\n\u003e pressure on archival nodes, and would be acceptable and non-risky to\n\u003e ship as a default in bitcoin-core. For lack of a better name, I'll\n\u003e call this new type of node a 'small node'. The intention is that\n\u003e bitcoin-core would eventually ship 'small nodes' by default, such that\n\u003e the expected amount of disk consumption drops from today's 100+ GB to\n\u003e less than 30 GB.\n\u003e\n\u003e My alternative proposal has the following properties:\n\u003e\n\u003e + Full nodes only need to store ~20% of the blockchain\n\u003e + With very high probability, a new node will be able to recover the\n\u003e entire blockchain by connecting to 6 random small node peers.\n\u003e + An attacker that can eliminate a chosen+ 95% of the full nodes\n\u003e running today will be unable to prevent new nodes from downloading the\n\u003e full blockchain, even if the attacker is also able to eliminate all\n\u003e archival nodes. (assuming all nodes today were small nodes instead of\n\u003e archival nodes)\n\u003e\n\u003e Method:\n\u003e\n\u003e A small node will pick an index [5, 256). This index is that node's\n\u003e permanent index. When storing a block, instead of storing the full\n\u003e block, the node will use Reed-Solomon coding to erasure code the block\n\u003e using a 5-of-256 scheme. The result will be 256 pieces that are 20% of\n\u003e the size of the block each. The node picks the piece that corresponds\n\u003e to its index, and stores that instead. (Indexes 0-4 are reserved for\n\u003e archival nodes - explained later)\n\u003e\n\u003e The node is now storing a fragment of every block. Alone, this\n\u003e fragment cannot be used to recover any piece of the blockchain.\n\u003e However, when paired with any 5 unique fragments (fragments of the\n\u003e same index will not be unique), the full block can be recovered.\n\u003e\n\u003e Nodes can optionally store more than 1 fragment each. At 5 fragments,\n\u003e the node becomes a full archival node, and the chosen indexes should\n\u003e be 0-4. This is advantageous for the archival node as the encoded data\n\u003e for the first 5 indexes will actually be identical to the block itself\n\u003e - there is no computational overhead for selecting the first indexes.\n\u003e There is also no need to choose random indexes, because the full block\n\u003e can be recovered no matter which indexes are chosen.\n\u003e\n\u003e When connecting to new peers, the indexes of each peer needs to be\n\u003e known. Once peers totaling 5 unique indexes are discovered, blockchain\n\u003e download can begin. Connecting to just 5 small node peers provides a\n\u003e \u003e95% chance of getting 5 uniques, with exponentially improving odds of\n\u003e success as you connect to more peers. Connecting to a single archive\n\u003e node guarantees that any gaps can be filled.\n\u003e\n\u003e A good encoder should be able to turn a block into a 5-of-256 piece\n\u003e set in under 10 milliseconds using a single core on a standard\n\u003e consumer desktop. This should not slow down initial blockchain\n\u003e download substantially, though the overhead is more than a rounding error.\n\u003e\n\u003e *DoS Prevention:*\n\u003e\n\u003e A malicious node may provide garbage data instead of the actual piece.\n\u003e Given just the garbage data and 4 other correct pieces, it is\n\u003e impossible (best I know anyway) to tell which piece is the garbage piece.\n\u003e\n\u003e One option in this case would be to seek out an archival node that\n\u003e could verify the correctness of the pieces, and identify the malicious\n\u003e node.\n\u003e\n\u003e Another option would be to have the small nodes store a cryptographic\n\u003e checksum of each piece. Obtaining the cryptographic checksum for all\n\u003e 256 pieces would incur a nontrivial amount of hashing (post segwit, as\n\u003e much as 100MB of extra hashing per block), and would require an\n\u003e additional ~4kb of storage per block. The hashing overhead here may be\n\u003e prohibitive.\n\u003e\n\u003e Another solution would be to find additional pieces and brute-force\n\u003e combinations of 5 until a working combination was discovered. Though\n\u003e this sounds nasty, it should take less than five seconds of\n\u003e computation to find the working combination given 5 correct pieces and\n\u003e 2 incorrect pieces. This computation only needs to be performed once\n\u003e to identify the malicious peers.\n\u003e\n\u003e I also believe that alternative erasure coding schemes exist which\n\u003e actually are able to identify the bad pieces given sufficient good\n\u003e pieces, however I don't know if they have the same computational\n\u003e performance as the best Reed-Solomon coding implementations.\n\u003e\n\u003e *Deployment:*\n\u003e\n\u003e Small nodes are completely useless unless the critical mass of 5\n\u003e pieces can be obtained. The first version that supports small node\n\u003e block downloads should default everyone to an archival node (meaning\n\u003e indexes 0-4 are used)\n\u003e\n\u003e Once there are enough small-node-enabled archive nodes, the default\n\u003e can be switched so that nodes only have a single index by default. In\n\u003e the first few days, when there are only a few small nodes, the\n\u003e previously-deployed archival nodes can help fill in the gaps, and the\n\u003e small nodes can be useful for blockchain download right away.\n\u003e\n\u003e ----------------------------------\n\u003e\n\u003e This represents a non-trivial amount of code, but I believe that the\n\u003e result would be a non-trivial increase in the percentage of users\n\u003e running full nodes, and a healthier overall network.\n\u003e\n\u003e\n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\n-- \nZcash wallets made simple: https://github.com/Ayms/zcash-wallets\nBitcoin wallets made simple: https://github.com/Ayms/bitcoin-wallets\nGet the torrent dynamic blocklist: http://peersm.com/getblocklist\nCheck the 10 M passwords list: http://peersm.com/findmyass\nAnti-spies and private torrents, dynamic blocklist: http://torrent-live.org\nPeersm : http://www.peersm.com\ntorrent-live: https://github.com/Ayms/torrent-live\nnode-Tor : https://www.github.com/Ayms/node-Tor\nGitHub : https://www.github.com/Ayms\n\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170417/90387873/attachment-0001.html\u003e",
"sig": "02bd26ce4468cf8ffbc8e5ff3b5f05bfe52a64502474e37e6068306554414d0f6f0d8c7bb02a325782c345e7fe6fada2d31afeb7973f26f48bcae1d55bcc20a3"
}