📅 Original date posted:2015-11-10
📝 Original message:Comments:
1) cblock seems a reasonable way to extend the protocol. Further wrapping
should probably be done at the stream level.
2) zlib has crappy security track record.
3) A fallback path to non-compressed is required, should compression fail
or crash.
4) Most blocks and transactions have runs of zeroes and/or highly common
bit-patterns, which contributes to useful compression even at smaller
sizes. Peter Ts's most recent numbers bear this out. zlib has a
dictionary (32K?) which works well with repeated patterns such as those you
see with concatenated runs of transactions.
5) LZO should provide much better compression, at a cost of CPU performance
and using a less-reviewed, less-field-tested library.
On Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>
> On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper <
> peter.tschipper at gmail.com> wrote:
>
>> There are better ways of sending new blocks, that's certainly true but
>> for sending historical blocks and seding transactions I don't think so.
>> This PR is really designed to save bandwidth and not intended to be a huge
>> performance improvement in terms of time spent sending.
>>
>
> If the main point is for historical data, then sticking to just blocks is
> the best plan.
>
> Since small blocks don't compress well, you could define a "cblocks"
> message that handles multiple blocks (just concatenate the block messages
> as payload before compression).
>
> The sending peer could combine blocks so that each cblock is compressing
> at least 10kB of block data (or whatever is optimal). It is probably worth
> specifying a maximum size for network buffer reasons (either 1MB or 1 block
> maximum).
>
> Similarly, transactions could be combined together and compressed "ctxs".
> The inv messages could be modified so that you can request groups of 10-20
> transactions. That would depend on how much of an improvement compressed
> transactions would represent.
>
> More generally, you could define a message which is a compressed message
> holder. That is probably to complex to be worth the effort though.
>
>
>
>>
>> On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev <
>> <bitcoin-dev at lists.linuxfoundation.org>
>> bitcoin-dev at lists.linuxfoundation.org> wrote:
>>
>>> On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev <
>>> bitcoin-dev at lists.linuxfoundation.org> wrote:
>>>
>>>
>>>> I think 25% bandwidth savings is certainly considerable, especially for
>>>> people running full nodes in countries like Australia where internet
>>>> bandwidth is lower and there are data caps.
>>>>
>>>
>>> This reinforces the idea that such trade-off decisions should be be
>>> local and negotiated between peers, not a required feature of the network
>>> P2P.
>>>
>>>
>>> --
>>> Johnathan Corgan
>>> Corgan Labs - SDR Training and Development Services
>>> http://corganlabs.com
>>>
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev at lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing listbitcoin-dev at lists.linuxfoundation.orghttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151110/d0474410/attachment.html>;
Jeff Garzik [ARCHIVE] /
npub1kf…f3f58
2023-06-07 17:44:23
in reply to nevent1q…jelg
Author Public Key
npub1kf0ppcjaguxekg24yx6smgxlu73qn0k8lm0t2wrqc0scpl7u3sgsmf3f58Seen on
wss://relay.noswhere.comPublished at
2023-06-07 17:44:23Event JSON
{
"id": "307619f9e8f31115e9bdad654979a2a3c32717807e72d2198fccaba7b05c7672",
"pubkey": "b25e10e25d470d9b215521b50da0dfe7a209bec7fedeb53860c3e180ffdc8c11",
"created_at": 1686159863,
"kind": 1,
"tags": [
[
"e",
"135d35eb642f6996f1dd35c63783ed01db4ec3c84bf3636c660d5a16e2852ec0",
"",
"root"
],
[
"e",
"037b31b4998343e24604f29fd9cd96bccb9745edc352d5f920ea9488bc23028b",
"",
"reply"
],
[
"p",
"46986f86b97cc97829a031b03209644d134b939d0163375467f0b1363e0d875e"
]
],
"content": "📅 Original date posted:2015-11-10\n📝 Original message:Comments:\n\n1) cblock seems a reasonable way to extend the protocol. Further wrapping\nshould probably be done at the stream level.\n\n2) zlib has crappy security track record.\n\n3) A fallback path to non-compressed is required, should compression fail\nor crash.\n\n4) Most blocks and transactions have runs of zeroes and/or highly common\nbit-patterns, which contributes to useful compression even at smaller\nsizes. Peter Ts's most recent numbers bear this out. zlib has a\ndictionary (32K?) which works well with repeated patterns such as those you\nsee with concatenated runs of transactions.\n\n5) LZO should provide much better compression, at a cost of CPU performance\nand using a less-reviewed, less-field-tested library.\n\n\n\n\n\nOn Tue, Nov 10, 2015 at 11:30 AM, Tier Nolan via bitcoin-dev \u003c\nbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\n\u003e\n\u003e\n\u003e On Tue, Nov 10, 2015 at 4:11 PM, Peter Tschipper \u003c\n\u003e peter.tschipper at gmail.com\u003e wrote:\n\u003e\n\u003e\u003e There are better ways of sending new blocks, that's certainly true but\n\u003e\u003e for sending historical blocks and seding transactions I don't think so.\n\u003e\u003e This PR is really designed to save bandwidth and not intended to be a huge\n\u003e\u003e performance improvement in terms of time spent sending.\n\u003e\u003e\n\u003e\n\u003e If the main point is for historical data, then sticking to just blocks is\n\u003e the best plan.\n\u003e\n\u003e Since small blocks don't compress well, you could define a \"cblocks\"\n\u003e message that handles multiple blocks (just concatenate the block messages\n\u003e as payload before compression).\n\u003e\n\u003e The sending peer could combine blocks so that each cblock is compressing\n\u003e at least 10kB of block data (or whatever is optimal). It is probably worth\n\u003e specifying a maximum size for network buffer reasons (either 1MB or 1 block\n\u003e maximum).\n\u003e\n\u003e Similarly, transactions could be combined together and compressed \"ctxs\".\n\u003e The inv messages could be modified so that you can request groups of 10-20\n\u003e transactions. That would depend on how much of an improvement compressed\n\u003e transactions would represent.\n\u003e\n\u003e More generally, you could define a message which is a compressed message\n\u003e holder. That is probably to complex to be worth the effort though.\n\u003e\n\u003e\n\u003e\n\u003e\u003e\n\u003e\u003e On Tue, Nov 10, 2015 at 5:40 AM, Johnathan Corgan via bitcoin-dev \u003c\n\u003e\u003e \u003cbitcoin-dev at lists.linuxfoundation.org\u003e\n\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\u003e\u003e\n\u003e\u003e\u003e On Mon, Nov 9, 2015 at 5:58 PM, gladoscc via bitcoin-dev \u003c\n\u003e\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\u003e\u003e\u003e\n\u003e\u003e\u003e\n\u003e\u003e\u003e\u003e I think 25% bandwidth savings is certainly considerable, especially for\n\u003e\u003e\u003e\u003e people running full nodes in countries like Australia where internet\n\u003e\u003e\u003e\u003e bandwidth is lower and there are data caps.\n\u003e\u003e\u003e\u003e\n\u003e\u003e\u003e\n\u003e\u003e\u003e This reinforces the idea that such trade-off decisions should be be\n\u003e\u003e\u003e local and negotiated between peers, not a required feature of the network\n\u003e\u003e\u003e P2P.\n\u003e\u003e\u003e\n\u003e\u003e\u003e\n\u003e\u003e\u003e --\n\u003e\u003e\u003e Johnathan Corgan\n\u003e\u003e\u003e Corgan Labs - SDR Training and Development Services\n\u003e\u003e\u003e http://corganlabs.com\n\u003e\u003e\u003e\n\u003e\u003e\u003e _______________________________________________\n\u003e\u003e\u003e bitcoin-dev mailing list\n\u003e\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\u003e\u003e\n\u003e\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e _______________________________________________\n\u003e\u003e bitcoin-dev mailing listbitcoin-dev at lists.linuxfoundation.orghttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151110/d0474410/attachment.html\u003e",
"sig": "ec8c9d0c0a26b92c47542d66021e2af0824edb330ad6c739ad99ba1ce42e066af0a0955d4e8d2d9a81f4cdef4e3230bef4deb3d8f54afcaa8dc6e3fa97887e5c"
}