📅 Original date posted:2015-12-03
📝 Original message:Emin's email presents to me the idea of dictionaries that already contain
the data we'd want to compress. With 8 bytes of indexing data, we can
refer to a TxID or a Public Key or any existing part of the blockchain.
There are also data sequences like scripts that contain a few variable
chunks and are otherwise identical. Often, the receiver has the
blockchain, which contains a lot of the data that is in the message being
transmitted.
First, the receiver must indicate that compressed data is preferred and the
height of latest valid block it holds, and the sender must express the
ability to send compressed data. From this state, the sender sends
messages that are compressed. Compressed messages are the same as
uncompressed messages except that:
1. Data read is copied into the decompressed message until the first
occurrence of 0x00, which is discarded and is followed by compressed data.
2. Compressed data can use as a dictionary the first 16,777,215 blocks,
or the last 4,244,635,647 ending with the block at the tip of the
receiver's chain, or it can specify a run of zero bytes. The sender and
receiver must agree on the *receiver's* current block height in order to
use the last 4B blocks as the dictionary.
3. Within compressed data, the first byte identifies how to decompress:
1. 0xFF indicates that the following three bytes are a block height
with most significant byte 0x00 in network byte order.
2. 0xFE indicates that the following byte indicates how many zero
bytes to add to the decompressed data.
3. 0xFD is an error, so compressed messages are turned off and the
recipient fails the decompression process.
4. 0x00 indicates that the zero byte by itself should be added to the
decompressed data, and the data following is not compressed
(return to step
1).
5. All other values represent the most significant byte of a number
to be subtracted from the receiver's current block height to identify a
block height (not available until there are least 16,777,216
blocks so that
this byte can be at least 0x01, since 0x00 would indicate a single zero
byte, end compressed data, and return to step 1).
4. If decompression has identified a block height (previous byte was not
0xFD, 0x00, or 0xFE), then the next four bytes identify a *size *(one
byte) and a byte index into the block's data (three bytes), and *size *bytes
from that block are added to the decompressed data.
5. Steps 3 and 4 process a chunk of compressed data. If the next byte
is 0xFD, then decompression goes back to step 1 (add raw bytes until it
hits a 0x00). Otherwise, it proceeds through steps 3 (and maybe 4) again.
In Step 3.3, 0xFD causes an error, but it could be used to indicate a
parameterized dictionary entry, for example 0xFD, 0x01 followed by eight
more bytes to be interpreted according to steps 3.1 or 3.5 could mean
OP_DUP OP_HASH160 (20 bytes from the blockchain dictionary) OP_EQUALVERIFY
OP_CHECKSIG, replacing that very common occurrence of 24 bytes with 10
bytes. Well, 11 if you include the 0x00 required by step5. But that only
works on addresses that have spent inputs. Or 0xFD, 0x02 could be
shorthand for the four zeroes of lock_time, followed by Version (1),
followed by 0x01 (for one-input transactions), turning nine bytes into two
for the data at the end of a normal (lock_time = 0) Txn and the beginning
of a single-input Txn. But I left 0xFD as an error because those gains
didn't seem as frequent as the others.
Dave.
On Wed, Dec 2, 2015 at 3:05 PM, Peter Tschipper via bitcoin-dev <
bitcoin-dev at lists.linuxfoundation.org> wrote:
>
> On 30/11/2015 9:28 PM, Matt Corallo wrote:
>
> I'm really not a fan of this at all. To start with, adding a compression library that is directly accessible to the network on financial software is a really, really scary idea.
>
> Why scary? LZO has no current security issues, and it will be
> configureable by each node operator so it can be turned off completely if
> needed or desired.
>
> If there were a massive improvement, I'd find it acceptable, but the improvement you've shown really isn't all that much.
>
> Why is 15% at the low end, to 27% at the high end not good? It sounds
> like a very good boost.
>
> The numbers you recently posted show it improving the very beginning of IBD somewhat over high-latency connections, but if we're throughput-limited after the very beginning of IBD, we should fix that, not compress the blocks.
>
> I only did the compression up to the 200,000 block to better isolate the
> transmission of data from the post processing of blocks and determine
> whether the compressing of data was adding to much to the total
> transmission time.
>
> I think it's clear from the data that as the data (blocks, transactions)
> increase in size that (1) they compress better and (2) they have a bigger
> and positive impact on improving performance when compressed.
>
> Additionally, I'd be very surprised if this had any significant effect on the speed at which new blocks traverse the network (do you have any simulations or other thoughts on this?).
>
> From the table below, at 120000 blocks the time to sync the chain was
> roughly the same for compressed vs. uncompressed however after that point
> as block sizes start increasing, all compression libraries peformed much
> faster than uncompressed. The data provided in this testing clearly shows
> that as block size increases, the performance improvement by compressing
> data also increases.
>
> TABLE 5:
> Results shown in seconds with 60ms of induced latency
> Num blks sync'd Uncmp Zlib-1 Zlib-6 LZO1x-1 LZO1x-999
> --------------- ----- ------ ------ ------- ---------
> 120000 3226 3416 3397 3266 3302
> 130000 4010 3983 3773 3625 3703
> 140000 4914 4503 4292 4127 4287
> 150000 5806 4928 4719 4529 4821
> 160000 6674 5249 5164 4840 5314
> 170000 7563 5603 5669 5289 6002
> 180000 8477 6054 6268 5858 6638
> 190000 9843 7085 7278 6868 7679
> 200000 11338 8215 8433 8044 8795
>
>
> As far as, what happens after the block is received, then obviously
> compression isn't going to help in post processing and validating the
> block, but in the pure transmission of the object it most certainly and
> logically does and in a fairly direct proportion to the file size (a file
> that is 20% smaller will be transmited "at least" 20% faster, you can use
> any data transfer time calculator
> <http://www.calctool.org/CALC/prof/computing/transfer_time>; for that).
> The only issue, that I can see that required testing was to show how much
> compression there would be, and how much time the compression of the data
> would add to the sending of the data.
>
>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
--
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy <http://www.litmocracy.com>; and Meme Racing
<http://www.memeracing.net>; (in alpha).
I'm the webmaster for The Voluntaryist <http://www.voluntaryist.com>; which
now accepts Bitcoin.
I also code for The Dollar Vigilante <http://dollarvigilante.com/>;.
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151202/c8405e2f/attachment.html>;
Dave Scotese [ARCHIVE] /
npub15w…maru8
2023-06-07 17:45:27
in reply to nevent1q…yqrk
Author Public Key
npub15wz8j5cse6sexlw6f5q7arc5efe5hxn6kcxxyy222et8qms4u3lsemaru8Seen on
wss://relay.noswhere.comPublished at
2023-06-07 17:45:27Event JSON
{
"id": "3e1a022aa61bd65d5580177ed47cd9e33c9c3beb73f5f0ef85ec6d9206f0d3f2",
"pubkey": "a384795310cea1937dda4d01ee8f14ca734b9a7ab60c62114a5656706e15e47f",
"created_at": 1686159927,
"kind": 1,
"tags": [
[
"e",
"66e2fed0b01e52e71a1dd6b42fe6a9fda1f0a599c3d79a1ed608f15fbb8c1ad3",
"",
"root"
],
[
"e",
"27818121aab6238f9717880433b22965d1d5abee652b55dbf76cf57d1cf800fb",
"",
"reply"
],
[
"p",
"ac0cba4c14f9ee9637c44e3773a1d205d62c63ca6095fc39aae39e6fffee6f48"
]
],
"content": "📅 Original date posted:2015-12-03\n📝 Original message:Emin's email presents to me the idea of dictionaries that already contain\nthe data we'd want to compress. With 8 bytes of indexing data, we can\nrefer to a TxID or a Public Key or any existing part of the blockchain.\nThere are also data sequences like scripts that contain a few variable\nchunks and are otherwise identical. Often, the receiver has the\nblockchain, which contains a lot of the data that is in the message being\ntransmitted.\n\nFirst, the receiver must indicate that compressed data is preferred and the\nheight of latest valid block it holds, and the sender must express the\nability to send compressed data. From this state, the sender sends\nmessages that are compressed. Compressed messages are the same as\nuncompressed messages except that:\n\n 1. Data read is copied into the decompressed message until the first\n occurrence of 0x00, which is discarded and is followed by compressed data.\n 2. Compressed data can use as a dictionary the first 16,777,215 blocks,\n or the last 4,244,635,647 ending with the block at the tip of the\n receiver's chain, or it can specify a run of zero bytes. The sender and\n receiver must agree on the *receiver's* current block height in order to\n use the last 4B blocks as the dictionary.\n 3. Within compressed data, the first byte identifies how to decompress:\n 1. 0xFF indicates that the following three bytes are a block height\n with most significant byte 0x00 in network byte order.\n 2. 0xFE indicates that the following byte indicates how many zero\n bytes to add to the decompressed data.\n 3. 0xFD is an error, so compressed messages are turned off and the\n recipient fails the decompression process.\n 4. 0x00 indicates that the zero byte by itself should be added to the\n decompressed data, and the data following is not compressed\n(return to step\n 1).\n 5. All other values represent the most significant byte of a number\n to be subtracted from the receiver's current block height to identify a\n block height (not available until there are least 16,777,216\nblocks so that\n this byte can be at least 0x01, since 0x00 would indicate a single zero\n byte, end compressed data, and return to step 1).\n 4. If decompression has identified a block height (previous byte was not\n 0xFD, 0x00, or 0xFE), then the next four bytes identify a *size *(one\n byte) and a byte index into the block's data (three bytes), and *size *bytes\n from that block are added to the decompressed data.\n 5. Steps 3 and 4 process a chunk of compressed data. If the next byte\n is 0xFD, then decompression goes back to step 1 (add raw bytes until it\n hits a 0x00). Otherwise, it proceeds through steps 3 (and maybe 4) again.\n\nIn Step 3.3, 0xFD causes an error, but it could be used to indicate a\nparameterized dictionary entry, for example 0xFD, 0x01 followed by eight\nmore bytes to be interpreted according to steps 3.1 or 3.5 could mean\nOP_DUP OP_HASH160 (20 bytes from the blockchain dictionary) OP_EQUALVERIFY\nOP_CHECKSIG, replacing that very common occurrence of 24 bytes with 10\nbytes. Well, 11 if you include the 0x00 required by step5. But that only\nworks on addresses that have spent inputs. Or 0xFD, 0x02 could be\nshorthand for the four zeroes of lock_time, followed by Version (1),\nfollowed by 0x01 (for one-input transactions), turning nine bytes into two\nfor the data at the end of a normal (lock_time = 0) Txn and the beginning\nof a single-input Txn. But I left 0xFD as an error because those gains\ndidn't seem as frequent as the others.\n\nDave.\n\nOn Wed, Dec 2, 2015 at 3:05 PM, Peter Tschipper via bitcoin-dev \u003c\nbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\n\u003e\n\u003e On 30/11/2015 9:28 PM, Matt Corallo wrote:\n\u003e\n\u003e I'm really not a fan of this at all. To start with, adding a compression library that is directly accessible to the network on financial software is a really, really scary idea.\n\u003e\n\u003e Why scary? LZO has no current security issues, and it will be\n\u003e configureable by each node operator so it can be turned off completely if\n\u003e needed or desired.\n\u003e\n\u003e If there were a massive improvement, I'd find it acceptable, but the improvement you've shown really isn't all that much.\n\u003e\n\u003e Why is 15% at the low end, to 27% at the high end not good? It sounds\n\u003e like a very good boost.\n\u003e\n\u003e The numbers you recently posted show it improving the very beginning of IBD somewhat over high-latency connections, but if we're throughput-limited after the very beginning of IBD, we should fix that, not compress the blocks.\n\u003e\n\u003e I only did the compression up to the 200,000 block to better isolate the\n\u003e transmission of data from the post processing of blocks and determine\n\u003e whether the compressing of data was adding to much to the total\n\u003e transmission time.\n\u003e\n\u003e I think it's clear from the data that as the data (blocks, transactions)\n\u003e increase in size that (1) they compress better and (2) they have a bigger\n\u003e and positive impact on improving performance when compressed.\n\u003e\n\u003e Additionally, I'd be very surprised if this had any significant effect on the speed at which new blocks traverse the network (do you have any simulations or other thoughts on this?).\n\u003e\n\u003e From the table below, at 120000 blocks the time to sync the chain was\n\u003e roughly the same for compressed vs. uncompressed however after that point\n\u003e as block sizes start increasing, all compression libraries peformed much\n\u003e faster than uncompressed. The data provided in this testing clearly shows\n\u003e that as block size increases, the performance improvement by compressing\n\u003e data also increases.\n\u003e\n\u003e TABLE 5:\n\u003e Results shown in seconds with 60ms of induced latency\n\u003e Num blks sync'd Uncmp Zlib-1 Zlib-6 LZO1x-1 LZO1x-999\n\u003e --------------- ----- ------ ------ ------- ---------\n\u003e 120000 3226 3416 3397 3266 3302\n\u003e 130000 4010 3983 3773 3625 3703\n\u003e 140000 4914 4503 4292 4127 4287\n\u003e 150000 5806 4928 4719 4529 4821\n\u003e 160000 6674 5249 5164 4840 5314\n\u003e 170000 7563 5603 5669 5289 6002\n\u003e 180000 8477 6054 6268 5858 6638\n\u003e 190000 9843 7085 7278 6868 7679\n\u003e 200000 11338 8215 8433 8044 8795\n\u003e\n\u003e\n\u003e As far as, what happens after the block is received, then obviously\n\u003e compression isn't going to help in post processing and validating the\n\u003e block, but in the pure transmission of the object it most certainly and\n\u003e logically does and in a fairly direct proportion to the file size (a file\n\u003e that is 20% smaller will be transmited \"at least\" 20% faster, you can use\n\u003e any data transfer time calculator\n\u003e \u003chttp://www.calctool.org/CALC/prof/computing/transfer_time\u003e for that).\n\u003e The only issue, that I can see that required testing was to show how much\n\u003e compression there would be, and how much time the compression of the data\n\u003e would add to the sending of the data.\n\u003e\n\u003e\n\u003e\n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\n\u003e\n\n\n-- \nI like to provide some work at no charge to prove my value. Do you need a\ntechie?\nI own Litmocracy \u003chttp://www.litmocracy.com\u003e and Meme Racing\n\u003chttp://www.memeracing.net\u003e (in alpha).\nI'm the webmaster for The Voluntaryist \u003chttp://www.voluntaryist.com\u003e which\nnow accepts Bitcoin.\nI also code for The Dollar Vigilante \u003chttp://dollarvigilante.com/\u003e.\n\"He ought to find it more profitable to play by the rules\" - Satoshi\nNakamoto\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151202/c8405e2f/attachment.html\u003e",
"sig": "210c755b84bf0817b161a4c051dbe7e2077a6ab103090d02a8f10182bdb175ef4e7f14f11f931dac6ddcfe5d0195fa3aecac64ad70439d3af98a7f4620756936"
}