📅 Original date posted:2015-11-10
📝 Original message:On 10/11/2015 8:11 AM, Peter Tschipper wrote:
> On 10/11/2015 1:44 AM, Tier Nolan via bitcoin-dev wrote:
>> The network protocol is not quite consensus critical, but it is
>> important.
>>
>> Two implementations of the decompressor might not be bug for bug
>> compatible. This (potentially) means that a block could be designed
>> that won't decode properly for some version of the client but would
>> work for another. This would fork the network.
>>
>> A "raw" network library is unlikely to have the same problem.
>>
>> Rather than just compress the stream, you could compress only block
>> messages only. A new "cblock" message could be created that is a
>> compressed block. This shouldn't reduce efficiency by much.
>>
> I chose the more generic datastream compression so we could in the
> future apply to possibly to transactions but currently all that is
> planned, is to compress blocks, and that was really my only original
> intent until I saw that there might be some bandwidth savings for
> transactions as well.
>
> The compression however could be applied to any datastream but is not
> *forced* . Basically it would just be a method call in CDatastream so
> we could do ss.compress and ss.decompress and apply that to blocks and
> possibly transactions if worthwhile and only IF compression is turned
> on. But there is no intend to apply this to every type of message
> since most would be too small to benefit from compression.
>
> Here are some results of using the code in the PR to
> compress/decompress blocks using zlib compression level = 6. This
> data was taken from the first 275K blocks in the mainnet blockchain.
> Clearly once we get past 10KB we get pretty decent compression but
> even below that there is some benefit. I'm still collecting data and
> will get the same for the whole blockchain.
>
> range = block size range
> ubytes = average size of uncompressed blocks
> cbytes = average size of compressed blocks
> ctime = average time to compress
> dtime = average time to decompress
> cmp_ratio% = compression ratio
> datapoints = number of datapoints taken
>
> range ubytes cbytes ctime dtime cmp_ratio% datapoints
> 0-250b 215 189 0.001 0.000 12.41 79498
> 250-500b 440 405 0.001 0.000 7.82 11903
> 500-1KB 762 702 0.001 0.000 7.83 10448
> 1KB-10KB 4166 3561 0.001 0.000 14.51 50572
> 10KB-100KB 40820 31597 0.005 0.001 22.59 75555
> 100KB-200KB 146238 106320 0.015 0.001 27.30 25024
> 200KB-300KB 242913 175482 0.025 0.002 27.76 20450
> 300KB-400KB 343430 251760 0.034 0.003 26.69 2069
> 400KB-500KB 457448 343495 0.045 0.004 24.91 1889
> 500KB-600KB 540736 424255 0.056 0.007 21.54 90
> 600KB-700KB 647851 506888 0.063 0.007 21.76 59
> 700KB-800KB 749513 586551 0.073 0.007 21.74 48
> 800KB-900KB 859439 652166 0.086 0.008 24.12 39
> 900KB-1MB 952333 725191 0.089 0.009 23.85 78
>
>> If a client fails to decode a cblock, then it can ask for the block
>> to be re-sent as a standard "block" message.
> interesting idea.
>>
>> This means that it is a pure performance improvement. If problems
>> occur, then the client can just switch back to uncompressed mode for
>> that block.
>>
>> You should look into the block relay system. This gives a larger
>> improvement than simply compressing the stream. The main benefit is
>> latency but it means that actual blocks don't have to be sent, so
>> gives a potential 50% compression ratio. Normally, a node receives
>> all the transactions and then those transactions are included later
>> in the block.
>>
> There are better ways of sending new blocks, that's certainly true but
> for sending historical blocks and seding transactions I don't think
> so. This PR is really designed to save bandwidth and not intended to
> be a huge performance improvement in terms of time spent sending.
>>
>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev
>> <bitcoin-dev at lists.linuxfoundation.org> wrote:
>>
>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev
>> <bitcoin-dev at lists.linuxfoundation.org
>> <mailto:bitcoin-dev at lists.linuxfoundation.org>> wrote:
>>
>>
>> I think 25% bandwidth savings is certainly considerable,
>> especially for people running full nodes in countries like
>> Australia where internet bandwidth is lower and there are
>> data caps.
>>
>>
>> This reinforces the idea that such trade-off decisions should be
>> be local and negotiated between peers, not a required feature of
>> the network P2P.
>>
>>
>> --
>> Johnathan Corgan
>> Corgan Labs - SDR Training and Development Services
>> http://corganlabs.com
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev at lists.linuxfoundation.org
>> <mailto:bitcoin-dev at lists.linuxfoundation.org>
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151110/a9fefde7/attachment.html>;
Peter Tschipper [ARCHIVE] /
npub14s…653tk
2023-06-07 17:44:22
in reply to nevent1q…mnvh
Author Public Key
npub14sxt5nq5l8hfvd7yfcmh8gwjqhtzcc72vz2lcwd2uw0xlllwdayqk653tkPublished at
2023-06-07 17:44:22Event JSON
{
"id": "ab2d58df1732cb851826a50c7d604fd227fc1af94be4f52b027c6085e7a0a376",
"pubkey": "ac0cba4c14f9ee9637c44e3773a1d205d62c63ca6095fc39aae39e6fffee6f48",
"created_at": 1686159862,
"kind": 1,
"tags": [
[
"e",
"135d35eb642f6996f1dd35c63783ed01db4ec3c84bf3636c660d5a16e2852ec0",
"",
"root"
],
[
"e",
"63eb70f5482fa0cdaa4abcef240dbede668ea743a9b70a4caa382cdbda3079fa",
"",
"reply"
],
[
"p",
"46986f86b97cc97829a031b03209644d134b939d0163375467f0b1363e0d875e"
]
],
"content": "📅 Original date posted:2015-11-10\n📝 Original message:On 10/11/2015 8:11 AM, Peter Tschipper wrote:\n\u003e On 10/11/2015 1:44 AM, Tier Nolan via bitcoin-dev wrote:\n\u003e\u003e The network protocol is not quite consensus critical, but it is\n\u003e\u003e important.\n\u003e\u003e\n\u003e\u003e Two implementations of the decompressor might not be bug for bug\n\u003e\u003e compatible. This (potentially) means that a block could be designed\n\u003e\u003e that won't decode properly for some version of the client but would\n\u003e\u003e work for another. This would fork the network.\n\u003e\u003e\n\u003e\u003e A \"raw\" network library is unlikely to have the same problem.\n\u003e\u003e\n\u003e\u003e Rather than just compress the stream, you could compress only block\n\u003e\u003e messages only. A new \"cblock\" message could be created that is a\n\u003e\u003e compressed block. This shouldn't reduce efficiency by much.\n\u003e\u003e\n\u003e I chose the more generic datastream compression so we could in the\n\u003e future apply to possibly to transactions but currently all that is\n\u003e planned, is to compress blocks, and that was really my only original\n\u003e intent until I saw that there might be some bandwidth savings for\n\u003e transactions as well. \n\u003e\n\u003e The compression however could be applied to any datastream but is not\n\u003e *forced* . Basically it would just be a method call in CDatastream so\n\u003e we could do ss.compress and ss.decompress and apply that to blocks and\n\u003e possibly transactions if worthwhile and only IF compression is turned\n\u003e on. But there is no intend to apply this to every type of message\n\u003e since most would be too small to benefit from compression.\n\u003e\n\u003e Here are some results of using the code in the PR to\n\u003e compress/decompress blocks using zlib compression level = 6. This\n\u003e data was taken from the first 275K blocks in the mainnet blockchain. \n\u003e Clearly once we get past 10KB we get pretty decent compression but\n\u003e even below that there is some benefit. I'm still collecting data and\n\u003e will get the same for the whole blockchain.\n\u003e\n\u003e range = block size range\n\u003e ubytes = average size of uncompressed blocks\n\u003e cbytes = average size of compressed blocks\n\u003e ctime = average time to compress\n\u003e dtime = average time to decompress\n\u003e cmp_ratio% = compression ratio\n\u003e datapoints = number of datapoints taken\n\u003e\n\u003e range ubytes cbytes ctime dtime cmp_ratio% datapoints\n\u003e 0-250b 215 189 0.001 0.000 12.41 79498\n\u003e 250-500b 440 405 0.001 0.000 7.82 11903\n\u003e 500-1KB 762 702 0.001 0.000 7.83 10448\n\u003e 1KB-10KB 4166 3561 0.001 0.000 14.51 50572\n\u003e 10KB-100KB 40820 31597 0.005 0.001 22.59 75555\n\u003e 100KB-200KB 146238 106320 0.015 0.001 27.30 25024\n\u003e 200KB-300KB 242913 175482 0.025 0.002 27.76 20450\n\u003e 300KB-400KB 343430 251760 0.034 0.003 26.69 2069\n\u003e 400KB-500KB 457448 343495 0.045 0.004 24.91 1889\n\u003e 500KB-600KB 540736 424255 0.056 0.007 21.54 90\n\u003e 600KB-700KB 647851 506888 0.063 0.007 21.76 59\n\u003e 700KB-800KB 749513 586551 0.073 0.007 21.74 48\n\u003e 800KB-900KB 859439 652166 0.086 0.008 24.12 39\n\u003e 900KB-1MB 952333 725191 0.089 0.009 23.85 78\n\u003e\n\u003e\u003e If a client fails to decode a cblock, then it can ask for the block\n\u003e\u003e to be re-sent as a standard \"block\" message. \n\u003e interesting idea.\n\u003e\u003e\n\u003e\u003e This means that it is a pure performance improvement. If problems\n\u003e\u003e occur, then the client can just switch back to uncompressed mode for\n\u003e\u003e that block.\n\u003e\u003e\n\u003e\u003e You should look into the block relay system. This gives a larger\n\u003e\u003e improvement than simply compressing the stream. The main benefit is\n\u003e\u003e latency but it means that actual blocks don't have to be sent, so\n\u003e\u003e gives a potential 50% compression ratio. Normally, a node receives\n\u003e\u003e all the transactions and then those transactions are included later\n\u003e\u003e in the block.\n\u003e\u003e\n\u003e There are better ways of sending new blocks, that's certainly true but\n\u003e for sending historical blocks and seding transactions I don't think\n\u003e so. This PR is really designed to save bandwidth and not intended to\n\u003e be a huge performance improvement in terms of time spent sending.\n\u003e\u003e\n\u003e\u003e On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev\n\u003e\u003e \u003cbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\u003e\u003e\n\u003e\u003e On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev\n\u003e\u003e \u003cbitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e \u003cmailto:bitcoin-dev at lists.linuxfoundation.org\u003e\u003e wrote:\n\u003e\u003e \n\u003e\u003e\n\u003e\u003e I think 25% bandwidth savings is certainly considerable,\n\u003e\u003e especially for people running full nodes in countries like\n\u003e\u003e Australia where internet bandwidth is lower and there are\n\u003e\u003e data caps.\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e This reinforces the idea that such trade-off decisions should be\n\u003e\u003e be local and negotiated between peers, not a required feature of\n\u003e\u003e the network P2P.\n\u003e\u003e \n\u003e\u003e\n\u003e\u003e -- \n\u003e\u003e Johnathan Corgan\n\u003e\u003e Corgan Labs - SDR Training and Development Services\n\u003e\u003e http://corganlabs.com\n\u003e\u003e\n\u003e\u003e _______________________________________________\n\u003e\u003e bitcoin-dev mailing list\n\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e \u003cmailto:bitcoin-dev at lists.linuxfoundation.org\u003e\n\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e _______________________________________________\n\u003e\u003e bitcoin-dev mailing list\n\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151110/a9fefde7/attachment.html\u003e",
"sig": "8ef8f32e8b574e2641dcc883006d83f66ad039b66f3b5ff13e2c5f165467f29f273285ee9d9612abd441b6346d9721fe18341299e9017dc059220793e69f6354"
}