š
Original date posted:2017-04-20
š Original message:Try to find 1TB dedicated server hosting ...
If you want to set up an ecommerce site somewhere besides your living room,
storage costs are still a concern.
On Mon, Apr 17, 2017 at 3:11 AM, Danny Thorpe via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
> 1TB HDD is now available for under $40 USD. How is the 100GB storage
> requirement preventing anyone from setting up full nodes?
>
> On Apr 16, 2017 11:55 PM, "David Vorick via bitcoin-dev" <
> bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>> *Rationale:*
>>
>> A node that stores the full blockchain (I will use the term archival
>> node) requires over 100GB of disk space, which I believe is one of the most
>> significant barriers to more people running full nodes. And I believe the
>> ecosystem would benefit substantially if more users were running full nodes.
>>
>> The best alternative today to storing the full blockchain is to run a
>> pruned node, which keeps only the UTXO set and throws away already verified
>> blocks. The operator of the pruned node is able to enjoy the full security
>> benefits of a full node, but is essentially leeching the network, as they
>> performed a large download likely without contributing anything back.
>>
>> This puts more pressure on the archival nodes, as the archival nodes need
>> to pick up the slack and help new nodes bootstrap to the network. As the
>> pressure on archival nodes grows, fewer people will be able to actually run
>> archival nodes, and the situation will degrade. The situation would likely
>> become problematic quickly if bitcoin-core were to ship with the defaults
>> set to a pruned node.
>>
>> Even further, the people most likely to care about saving 100GB of disk
>> space are also the people least likely to care about some extra bandwidth
>> usage. For datacenter nodes, and for nodes doing lots of bandwidth, the
>> bandwidth is usually the biggest cost of running the node. For home users
>> however, as long as they stay under their bandwidth cap, the bandwidth is
>> actually free. Ideally, new nodes would be able to bootstrap from nodes
>> that do not have to pay for their bandwidth, instead of needing to rely on
>> a decreasing percentage of heavy-duty archival nodes.
>>
>> I have (perhaps incorrectly) identified disk space consumption as the
>> most significant factor in your average user choosing to run a pruned node
>> or a lite client instead of a full node. The average user is not typically
>> too worried about bandwidth, and is also not typically too worried about
>> initial blockchain download time. But the 100GB hit to your disk space can
>> be a huge psychological factor, especially if your hard drive only has
>> 500GB available in the first place, and 250+ GB is already consumed by
>> other files you have.
>>
>> I believe that improving the disk usage situation would greatly benefit
>> decentralization, especially if it could be done without putting pressure
>> on archival nodes.
>>
>> *Small Nodes Proposal:*
>>
>> I propose an alternative to the pruned node that does not put undue
>> pressure on archival nodes, and would be acceptable and non-risky to ship
>> as a default in bitcoin-core. For lack of a better name, I'll call this new
>> type of node a 'small node'. The intention is that bitcoin-core would
>> eventually ship 'small nodes' by default, such that the expected amount of
>> disk consumption drops from today's 100+ GB to less than 30 GB.
>>
>> My alternative proposal has the following properties:
>>
>> + Full nodes only need to store ~20% of the blockchain
>> + With very high probability, a new node will be able to recover the
>> entire blockchain by connecting to 6 random small node peers.
>> + An attacker that can eliminate a chosen+ 95% of the full nodes running
>> today will be unable to prevent new nodes from downloading the full
>> blockchain, even if the attacker is also able to eliminate all archival
>> nodes. (assuming all nodes today were small nodes instead of archival nodes)
>>
>> Method:
>>
>> A small node will pick an index [5, 256). This index is that node's
>> permanent index. When storing a block, instead of storing the full block,
>> the node will use Reed-Solomon coding to erasure code the block using a
>> 5-of-256 scheme. The result will be 256 pieces that are 20% of the size of
>> the block each. The node picks the piece that corresponds to its index, and
>> stores that instead. (Indexes 0-4 are reserved for archival nodes -
>> explained later)
>>
>> The node is now storing a fragment of every block. Alone, this fragment
>> cannot be used to recover any piece of the blockchain. However, when paired
>> with any 5 unique fragments (fragments of the same index will not be
>> unique), the full block can be recovered.
>>
>> Nodes can optionally store more than 1 fragment each. At 5 fragments, the
>> node becomes a full archival node, and the chosen indexes should be 0-4.
>> This is advantageous for the archival node as the encoded data for the
>> first 5 indexes will actually be identical to the block itself - there is
>> no computational overhead for selecting the first indexes. There is also no
>> need to choose random indexes, because the full block can be recovered no
>> matter which indexes are chosen.
>>
>> When connecting to new peers, the indexes of each peer needs to be known.
>> Once peers totaling 5 unique indexes are discovered, blockchain download
>> can begin. Connecting to just 5 small node peers provides a >95% chance of
>> getting 5 uniques, with exponentially improving odds of success as you
>> connect to more peers. Connecting to a single archive node guarantees that
>> any gaps can be filled.
>>
>> A good encoder should be able to turn a block into a 5-of-256 piece set
>> in under 10 milliseconds using a single core on a standard consumer
>> desktop. This should not slow down initial blockchain download
>> substantially, though the overhead is more than a rounding error.
>>
>> *DoS Prevention:*
>>
>> A malicious node may provide garbage data instead of the actual piece.
>> Given just the garbage data and 4 other correct pieces, it is impossible
>> (best I know anyway) to tell which piece is the garbage piece.
>>
>> One option in this case would be to seek out an archival node that could
>> verify the correctness of the pieces, and identify the malicious node.
>>
>> Another option would be to have the small nodes store a cryptographic
>> checksum of each piece. Obtaining the cryptographic checksum for all 256
>> pieces would incur a nontrivial amount of hashing (post segwit, as much as
>> 100MB of extra hashing per block), and would require an additional ~4kb of
>> storage per block. The hashing overhead here may be prohibitive.
>>
>> Another solution would be to find additional pieces and brute-force
>> combinations of 5 until a working combination was discovered. Though this
>> sounds nasty, it should take less than five seconds of computation to find
>> the working combination given 5 correct pieces and 2 incorrect pieces. This
>> computation only needs to be performed once to identify the malicious peers.
>>
>> I also believe that alternative erasure coding schemes exist which
>> actually are able to identify the bad pieces given sufficient good pieces,
>> however I don't know if they have the same computational performance as the
>> best Reed-Solomon coding implementations.
>>
>> *Deployment:*
>>
>> Small nodes are completely useless unless the critical mass of 5 pieces
>> can be obtained. The first version that supports small node block downloads
>> should default everyone to an archival node (meaning indexes 0-4 are used)
>>
>> Once there are enough small-node-enabled archive nodes, the default can
>> be switched so that nodes only have a single index by default. In the first
>> few days, when there are only a few small nodes, the previously-deployed
>> archival nodes can help fill in the gaps, and the small nodes can be useful
>> for blockchain download right away.
>>
>> ----------------------------------
>>
>> This represents a non-trivial amount of code, but I believe that the
>> result would be a non-trivial increase in the percentage of users running
>> full nodes, and a healthier overall network.
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170420/9e3837a3/attachment-0001.html>;
Erik Aronesty [ARCHIVE] /
npub1y2ā¦5taj0
2023-06-07 18:00:18
in reply to nevent1qā¦9azx
Author Public Key
npub1y22yec0znyzw8qndy5qn5c2wgejkj0k9zsqra7kvrd6cd6896z4qm5taj0Seen on
wss://relay.noswhere.comPublished at
2023-06-07 18:00:18Event JSON
{
"id": "30189cbacc37ceca9061b5aac2082f6d437cd9c3fce573fe65e891ea38fbc033",
"pubkey": "22944ce1e29904e3826d25013a614e4665693ec514003efacc1b7586e8e5d0aa",
"created_at": 1686160818,
"kind": 1,
"tags": [
[
"e",
"9e0bce6c49b056a94ef9193a53941e860a13142ae7014703964955e2e8de52b1",
"",
"root"
],
[
"e",
"113f18c71937d989842f438246ecb02641937d90f502d1e1b510599652e5543e",
"",
"reply"
],
[
"p",
"adcb53cdbd72e2a0c22452a9087f173c6b55edc7b504ff9b861e9c02af9a87d4"
]
],
"content": "š
Original date posted:2017-04-20\nš Original message:Try to find 1TB dedicated server hosting ...\n\nIf you want to set up an ecommerce site somewhere besides your living room,\nstorage costs are still a concern.\n\nOn Mon, Apr 17, 2017 at 3:11 AM, Danny Thorpe via bitcoin-dev \u003c\nbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\n\u003e 1TB HDD is now available for under $40 USD. How is the 100GB storage\n\u003e requirement preventing anyone from setting up full nodes?\n\u003e\n\u003e On Apr 16, 2017 11:55 PM, \"David Vorick via bitcoin-dev\" \u003c\n\u003e bitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\u003e\n\u003e\u003e *Rationale:*\n\u003e\u003e\n\u003e\u003e A node that stores the full blockchain (I will use the term archival\n\u003e\u003e node) requires over 100GB of disk space, which I believe is one of the most\n\u003e\u003e significant barriers to more people running full nodes. And I believe the\n\u003e\u003e ecosystem would benefit substantially if more users were running full nodes.\n\u003e\u003e\n\u003e\u003e The best alternative today to storing the full blockchain is to run a\n\u003e\u003e pruned node, which keeps only the UTXO set and throws away already verified\n\u003e\u003e blocks. The operator of the pruned node is able to enjoy the full security\n\u003e\u003e benefits of a full node, but is essentially leeching the network, as they\n\u003e\u003e performed a large download likely without contributing anything back.\n\u003e\u003e\n\u003e\u003e This puts more pressure on the archival nodes, as the archival nodes need\n\u003e\u003e to pick up the slack and help new nodes bootstrap to the network. As the\n\u003e\u003e pressure on archival nodes grows, fewer people will be able to actually run\n\u003e\u003e archival nodes, and the situation will degrade. The situation would likely\n\u003e\u003e become problematic quickly if bitcoin-core were to ship with the defaults\n\u003e\u003e set to a pruned node.\n\u003e\u003e\n\u003e\u003e Even further, the people most likely to care about saving 100GB of disk\n\u003e\u003e space are also the people least likely to care about some extra bandwidth\n\u003e\u003e usage. For datacenter nodes, and for nodes doing lots of bandwidth, the\n\u003e\u003e bandwidth is usually the biggest cost of running the node. For home users\n\u003e\u003e however, as long as they stay under their bandwidth cap, the bandwidth is\n\u003e\u003e actually free. Ideally, new nodes would be able to bootstrap from nodes\n\u003e\u003e that do not have to pay for their bandwidth, instead of needing to rely on\n\u003e\u003e a decreasing percentage of heavy-duty archival nodes.\n\u003e\u003e\n\u003e\u003e I have (perhaps incorrectly) identified disk space consumption as the\n\u003e\u003e most significant factor in your average user choosing to run a pruned node\n\u003e\u003e or a lite client instead of a full node. The average user is not typically\n\u003e\u003e too worried about bandwidth, and is also not typically too worried about\n\u003e\u003e initial blockchain download time. But the 100GB hit to your disk space can\n\u003e\u003e be a huge psychological factor, especially if your hard drive only has\n\u003e\u003e 500GB available in the first place, and 250+ GB is already consumed by\n\u003e\u003e other files you have.\n\u003e\u003e\n\u003e\u003e I believe that improving the disk usage situation would greatly benefit\n\u003e\u003e decentralization, especially if it could be done without putting pressure\n\u003e\u003e on archival nodes.\n\u003e\u003e\n\u003e\u003e *Small Nodes Proposal:*\n\u003e\u003e\n\u003e\u003e I propose an alternative to the pruned node that does not put undue\n\u003e\u003e pressure on archival nodes, and would be acceptable and non-risky to ship\n\u003e\u003e as a default in bitcoin-core. For lack of a better name, I'll call this new\n\u003e\u003e type of node a 'small node'. The intention is that bitcoin-core would\n\u003e\u003e eventually ship 'small nodes' by default, such that the expected amount of\n\u003e\u003e disk consumption drops from today's 100+ GB to less than 30 GB.\n\u003e\u003e\n\u003e\u003e My alternative proposal has the following properties:\n\u003e\u003e\n\u003e\u003e + Full nodes only need to store ~20% of the blockchain\n\u003e\u003e + With very high probability, a new node will be able to recover the\n\u003e\u003e entire blockchain by connecting to 6 random small node peers.\n\u003e\u003e + An attacker that can eliminate a chosen+ 95% of the full nodes running\n\u003e\u003e today will be unable to prevent new nodes from downloading the full\n\u003e\u003e blockchain, even if the attacker is also able to eliminate all archival\n\u003e\u003e nodes. (assuming all nodes today were small nodes instead of archival nodes)\n\u003e\u003e\n\u003e\u003e Method:\n\u003e\u003e\n\u003e\u003e A small node will pick an index [5, 256). This index is that node's\n\u003e\u003e permanent index. When storing a block, instead of storing the full block,\n\u003e\u003e the node will use Reed-Solomon coding to erasure code the block using a\n\u003e\u003e 5-of-256 scheme. The result will be 256 pieces that are 20% of the size of\n\u003e\u003e the block each. The node picks the piece that corresponds to its index, and\n\u003e\u003e stores that instead. (Indexes 0-4 are reserved for archival nodes -\n\u003e\u003e explained later)\n\u003e\u003e\n\u003e\u003e The node is now storing a fragment of every block. Alone, this fragment\n\u003e\u003e cannot be used to recover any piece of the blockchain. However, when paired\n\u003e\u003e with any 5 unique fragments (fragments of the same index will not be\n\u003e\u003e unique), the full block can be recovered.\n\u003e\u003e\n\u003e\u003e Nodes can optionally store more than 1 fragment each. At 5 fragments, the\n\u003e\u003e node becomes a full archival node, and the chosen indexes should be 0-4.\n\u003e\u003e This is advantageous for the archival node as the encoded data for the\n\u003e\u003e first 5 indexes will actually be identical to the block itself - there is\n\u003e\u003e no computational overhead for selecting the first indexes. There is also no\n\u003e\u003e need to choose random indexes, because the full block can be recovered no\n\u003e\u003e matter which indexes are chosen.\n\u003e\u003e\n\u003e\u003e When connecting to new peers, the indexes of each peer needs to be known.\n\u003e\u003e Once peers totaling 5 unique indexes are discovered, blockchain download\n\u003e\u003e can begin. Connecting to just 5 small node peers provides a \u003e95% chance of\n\u003e\u003e getting 5 uniques, with exponentially improving odds of success as you\n\u003e\u003e connect to more peers. Connecting to a single archive node guarantees that\n\u003e\u003e any gaps can be filled.\n\u003e\u003e\n\u003e\u003e A good encoder should be able to turn a block into a 5-of-256 piece set\n\u003e\u003e in under 10 milliseconds using a single core on a standard consumer\n\u003e\u003e desktop. This should not slow down initial blockchain download\n\u003e\u003e substantially, though the overhead is more than a rounding error.\n\u003e\u003e\n\u003e\u003e *DoS Prevention:*\n\u003e\u003e\n\u003e\u003e A malicious node may provide garbage data instead of the actual piece.\n\u003e\u003e Given just the garbage data and 4 other correct pieces, it is impossible\n\u003e\u003e (best I know anyway) to tell which piece is the garbage piece.\n\u003e\u003e\n\u003e\u003e One option in this case would be to seek out an archival node that could\n\u003e\u003e verify the correctness of the pieces, and identify the malicious node.\n\u003e\u003e\n\u003e\u003e Another option would be to have the small nodes store a cryptographic\n\u003e\u003e checksum of each piece. Obtaining the cryptographic checksum for all 256\n\u003e\u003e pieces would incur a nontrivial amount of hashing (post segwit, as much as\n\u003e\u003e 100MB of extra hashing per block), and would require an additional ~4kb of\n\u003e\u003e storage per block. The hashing overhead here may be prohibitive.\n\u003e\u003e\n\u003e\u003e Another solution would be to find additional pieces and brute-force\n\u003e\u003e combinations of 5 until a working combination was discovered. Though this\n\u003e\u003e sounds nasty, it should take less than five seconds of computation to find\n\u003e\u003e the working combination given 5 correct pieces and 2 incorrect pieces. This\n\u003e\u003e computation only needs to be performed once to identify the malicious peers.\n\u003e\u003e\n\u003e\u003e I also believe that alternative erasure coding schemes exist which\n\u003e\u003e actually are able to identify the bad pieces given sufficient good pieces,\n\u003e\u003e however I don't know if they have the same computational performance as the\n\u003e\u003e best Reed-Solomon coding implementations.\n\u003e\u003e\n\u003e\u003e *Deployment:*\n\u003e\u003e\n\u003e\u003e Small nodes are completely useless unless the critical mass of 5 pieces\n\u003e\u003e can be obtained. The first version that supports small node block downloads\n\u003e\u003e should default everyone to an archival node (meaning indexes 0-4 are used)\n\u003e\u003e\n\u003e\u003e Once there are enough small-node-enabled archive nodes, the default can\n\u003e\u003e be switched so that nodes only have a single index by default. In the first\n\u003e\u003e few days, when there are only a few small nodes, the previously-deployed\n\u003e\u003e archival nodes can help fill in the gaps, and the small nodes can be useful\n\u003e\u003e for blockchain download right away.\n\u003e\u003e\n\u003e\u003e ----------------------------------\n\u003e\u003e\n\u003e\u003e This represents a non-trivial amount of code, but I believe that the\n\u003e\u003e result would be a non-trivial increase in the percentage of users running\n\u003e\u003e full nodes, and a healthier overall network.\n\u003e\u003e\n\u003e\u003e _______________________________________________\n\u003e\u003e bitcoin-dev mailing list\n\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\u003e\n\u003e\u003e\n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20170420/9e3837a3/attachment-0001.html\u003e",
"sig": "18c7695b9bf807f898b3cea1768e06da3e7439e3c755f8c446c180167195cab560fb804880b89b751b14eb10992cc0a3e800e33968d2e1fd229be782ad4c797e"
}