Tom [ARCHIVE]

⚠️ This account is not associated with a real person. It's an archival account for mailing list messages.
🗓️ Active from 2016-03-18 to 2016-09-27
💬 Posted a total of 26 messages
📚 Participated in the following categories: bitcoin-dev
🔇 Please do not DM this user as it's used for archive purposes only.

Public Key

npub1h394c0pkdum0jw4rufslt3ur9m9c2ym4x7a0t68spfpjr6zlq3msu8z4k5

Profile Code

nprofile1qqstcj6u8smx7dhe8237yc04c7pjaju9zd6n0wh4arcq5sepap0sgac8m9vkg

Author Public Key

npub1h394c0pkdum0jw4rufslt3ur9m9c2ym4x7a0t68spfpjr6zlq3msu8z4k5

Show more details

Last Notes

2023-06-07 17:53:38

- reply

📅 Original date posted:2016-09-24
📝 Original message:On Saturday, 24 September 2016 06:36:00 CEST Luke Dashjr via bitcoin-dev
wrote:
> * OPL will no longer be an acceptable license. Many in the community feel
> that prohibiting publication is unacceptable for BIPs, and I haven't
> heard any arguments in favour of allowing it.
My suggestion would be that we replace OPL as an allowed license with one
or two Creative Commons licenses. Following the suggestion from the OPL
creators themselves.
According to Wikipedia;
> Open Publication License was created by the Open Content Project in 1999
> as public copyright license for documents. The license was superseded
> in 2003/2007 by the Creative commons licenses.
I'd suggest saying that "Share alike" is required and "Attribution" is
optional.
Executive summary; give the user the choice (next to public domain) between
CCO and BY-SA
see;
https://en.wikipedia.org/wiki/
Creative_Commons_license#Seven_regularly_used_licenses

2023-06-07 17:53:37

- reply

📅 Original date posted:2016-09-27
📝 Original message:On Monday 26 Sep 2016 14:41:36 Peter Todd via bitcoin-dev wrote:
> Note how the OPL is significantly more restrictive than the Bitcoin Core
> license; not good if we can't ship documentation with the code.
Documentation and code can have different licenses, the sole existence of
various documentation licenses attests to that point.
Shipping your docs under a separate licence has never been a problem before,
so you don't have to worry that you can't ship documentation with code.
That said, I wrote my suggestion in reply to Luke's BIP2 revival which is a
more formal suggestion of a solution. Maybe you can ACK that one instead?
Last, in preparation of acceptance of BIP2 I changed the licence of my BIP to
be dual-licensed. Now its also available under a Creative Commons license.
Have a nice day!

2023-06-07 17:53:35

- reply

📅 Original date posted:2016-09-24
📝 Original message:Thank you Luke, this makes it clearer.
It doesn't change that this scenario is an attack that doesn't give the
attacker any benefit and the attacked doesn't loose anything either (as
Dave pointed out).
This is a completely academical problem that assumes so many stupid
mistakes from software and from people that its very unlikely. On top of
that it assumes a rather lengthy 51% attack in concert with this already
extremely unlikely usecase.
In the scenario you assume stupid people and then you solve it by requiring
the victim to suddenly be super smart and use a solution specifically
designed for this super unlikely usecase that probably will never actually
happen...
I don't buy it.
On Friday, 23 September 2016 22:34:41 CEST Luke Dashjr wrote:
> Joe sends Alice 5 BTC (UTXO 0).
> Fred sends Alice 4 BTC (UTXO 1).
> Alice sends Bob 4 BTC using UTXO 1 (creating UTXO 2).
> Fred double-spends UTXO 1 with UTXO 1-B. This invalidates Alice's
> transfer to Bob.
> Alice has UTXO 0 which she can send to Bob (UTXO 3), but if she does so,
> it is possible that UTXO 0 could be mined, and then both UTXO 2 and UTXO
> 3 which would result in her giving Bob a total of 8 BTC rather than
> merely 4 BTC. Even if Alice waits until Fred's UTXO 1-B confirms 10
> blocks deep, it is not impossible for a reorganization to reverse those
> 10 blocks and confirm UTXO 1 again.
> Using OP_CHECKBLOCKATHEIGHT, however, Alice can create UTXO 3 such that
> it is valid only in the blockchain where Fred's UTXO 1-B has confirmed.
> This way, if that block is reorganized out, UTXO 3 is invalid, and
> either Bob receives only the original UTXO 2, or Alice can create a UTXO
> 3-B which is valid in the reorganized blockchain if it again confirms
> the UTXO 1-B double-spend.
>
> Luke
>
> On Friday, September 23, 2016 2:37:39 PM Tom via bitcoin-dev wrote:
> > On Friday 23 Sep 2016 09:57:01 Luke Dashjr via bitcoin-dev wrote:
> > > This BIP describes a new opcode (OP_CHECKBLOCKATHEIGHT) for the
> > > Bitcoin
> > > scripting system to address reissuing bitcoin transactions when the
> > > coins they spend have been conflicted/double-spent.
> > >
> > > https://github.com/luke-jr/bips/blob/bip-cbah/bip-cbah.mediawiki
> >
> > Can you walk us through a real live usecase which this solves? I read
> > it and I think I understand it, but I can't see the attack every
> > giving the attacker any benefit (or the attacked losing anything).
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev at lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

2023-06-07 17:53:34

- reply

📅 Original date posted:2016-09-23
📝 Original message:On Friday 23 Sep 2016 09:57:01 Luke Dashjr via bitcoin-dev wrote:
> This BIP describes a new opcode (OP_CHECKBLOCKATHEIGHT) for the Bitcoin
> scripting system to address reissuing bitcoin transactions when the coins
> they spend have been conflicted/double-spent.
>
> https://github.com/luke-jr/bips/blob/bip-cbah/bip-cbah.mediawiki
Can you walk us through a real live usecase which this solves? I read it and
I think I understand it, but I can't see the attack every giving the attacker
any benefit (or the attacked losing anything).

2023-06-07 17:53:30

- reply

📅 Original date posted:2016-09-22
📝 Original message:On Thursday, 22 September 2016 21:59:12 CEST Jonas Schnelli via bitcoin-dev
wrote:
> Hi Tom
>
> > I think you misunderstand tagged systems at a very basic level. You
> > think that html can only use a bold tag <b> once in a document? Thats
> > equivalent to what you are saying.
>
> Would the "additional" segment contain the same amount of
> nSequence-equivalent token as the number of inputs in the "inputs"
> segment?
At this point I don't know what it should look like, I have not had time to
look deeply into BIP68. Is this what you would suggest it to look like?
I rather figured spending limitations would be assigned to an output, not
an input.
> However, I think that should be mentioned/specified in the BIP.
It can be, and likely should be. This BIP doesn't pretend to be finished
yet.
I welcome any and all discussion about this, it only serves to make the end
result stronger!

2023-06-07 17:53:30

- reply

📅 Original date posted:2016-09-23
📝 Original message:On Friday, 23 September 2016 13:55:50 CEST Christian Decker via bitcoin-dev
wrote:
> Not sure if the comparison to XML and HTML holds: the lack of closing
> tags makes the meaning of individual tokens ambiguous, like I pointed
> out before. The use of segments gives at most two levels of nesting,
> so any relationship among tokens in the same segment has to rely on
> their relative position, which could result in ambiguities, like
> whether a tag refers to a single input or the transaction as a whole.
Practically all tagged formats make ordering a requirement, so indeed this
is relevant, and not unique.
For instance if you write;
<div> Some line </br>Another line</br>3rd line</div>
you can get a good idea of how ordering is relevant. You can reuse any item
many times.
Whenever there is a possible confusion the specification specifically
explains which order to use.
I'm not sure what you mean with the idea this;
> The use of segments gives at most two levels of nesting
It looks like you assume there is some opening and closing tags, since
otherwise there would be no nesting.
Such tags are not intended, nor documented.
> so any relationship among tokens in the same segment has to rely on
> their relative position, which could result in ambiguities, like
> whether a tag refers to a single input or the transaction as a whole.
I quoted parts of the spec in your previous email stating the same thing,
but I'll repeat here.
Any place that has any sort of possibility to be ambiguous is specified
specifically to have an order. This makes writing and parsing easier.
Since you wrote two emails now with the same issue, and I addressed it
twice, I would urge you to write out some examples which may be confusing
and if you find that the spec is indeed missing requirements then please
share it with us. I did this some time ago and it helps understanding the
ideas by having actual explicit examples. I am not aware of any sort of
ambiguities that the spec allows.
Cheers!

2023-06-07 17:53:29

- reply

📅 Original date posted:2016-09-22
📝 Original message:On Wednesday 21 Sep 2016 18:45:55 adiabat via bitcoin-dev wrote:
> Hi-
>
> One concern is that this doesn't seem compatible with Lightning as
> currently written. Most relevant is that non-cooperative channel close
> transactions in Lightning use OP_CHECKSEQUENCEVERIFY, which references the
> sequence field of the txin; if the txin doesn't have a sequence number,
> OP_CHECKSEQUENCEVERIFY can't work.
>
> LockByBlock and LockByTime aren't described and there doesn't seem to be
> code for them in the PR (186). If there's a way to make OP_CLTV and OP_CSV
> work with this new format, please let us know, thanks!
LockByBlock and LockByTime are still TODOs because I didn't have time to go
in-dept into how BIP68 does the encoding.
The intent is that these tags, while loading, will set the sequence integer in
the txin as the old version does. And while saving we do the reverse.
In other words; the lack of sequence number in the saved format doesn't affect
the in-memory format of the transaction. The in-memory version is the one that
script will operate on.
This means that there is no change in how CSV will work before and after on
any level other than serialisation.
Flexible Transactions is definitely meant to support the Lightning Network, so
any problems you find is something we should solve before it ships.
Thanks for your email!

2023-06-07 17:53:29

- reply

📅 Original date posted:2016-09-23
📝 Original message:On Friday, 23 September 2016 13:42:36 CEST Christian Decker via bitcoin-dev wrote:
> > I have to disagree. That is not malleability. Creating a new document
> > and re- signing it is not changing anything. Its re-creating.
> > Something that the owner of the coin has every right to do.
> Same thing I was arguing back then, however Luke pointed out that
> malleability just refers to the possibility of modifying a transaction
> after the fact.
I am not a fan of redefining dictionary words. I'll stick to the
universally excepted one, thanks.
> Nope, that is exactly the kind of dependency I was talking
> about. Instead of nesting a construct like the current transactions
> do, you rely on the order of tokens to imply that they belong
> together.
> if we
> add new fields that a non-upgraded node doesn't know about and it
> rejects transactions containing it, we'll have a hard-fork. It should
> probably not reject transactions with unknown fields if the
> transaction is included in a block.
This is addressed here;
https://github.com/bitcoin/bips/blob/master/bip-0134.mediawiki#future-extensibility

2023-06-07 17:53:29

- reply

📅 Original date posted:2016-09-22
📝 Original message:On Thursday, 22 September 2016 14:27:29 CEST Peter Todd wrote:
> CSV uses per-input sequence numbers; you only have a per-tx equivalent.
I think you misunderstand tagged systems at a very basic level. You think
that html can only use a bold tag <b> once in a document? Thats equivalent
to what you are saying.
Your comment is rather embarrassing, I have to point out. You may want to
read a bit more before you comment more.

2023-06-07 17:53:28

- reply

📅 Original date posted:2016-09-22
📝 Original message:On Thursday 22 Sep 2016 13:10:49 Christian Decker via bitcoin-dev wrote:
> On Thu, Sep 22, 2016 at 10:56:31AM +0200, Tom via bitcoin-dev wrote:
> > On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:
> > > On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
> > >
> > > <bitcoin-dev at lists.linuxfoundation.org> wrote:
> > > > BIP number for my FT spec.
> > >
> > > This document does not appear to be concretely specified enough to
> > > review or implement from it.
> > >
> > > For example, it does not specify the serialization of "integer"
> >
> > It refers to the external specification which is linked at the bottom.
> > In that spec you'll see that "Integer" is the standard var-int that
> > Bitcoin
> > has used for years.
>
> I think BIPs should be self-contained, or rely on previous BIPs,
> whenever possible. Referencing an external formatting document should
> be avoided
If luke-jr thinks I should submit CMF as a BIP, I can certainly do that.
Luke, what do you think?
I don't have a preference either way.
> > > nor does it specify how the
> > > presence of the optional fields are signaled
> >
> > How does one signals an optional field except of in the spec? Thats the
> > job of a specification.
>
> So the presence is signaled by encountering the tag, which contains
> both token type and name-reference. The encoder and decoder operations
> could be described better.
I'm sorry, I'm not following you here. Is there a question?
> > > nor the cardinality of
> > > the inputs or outputs.
> >
> > Did you miss this in the 3rd table ? I suggest clicking on the github
> > bips
> > repo link as tables are not easy to read in mediawiki plain format that
> > the
> > email contained.
>
> Minor nit: that table is not well-formed.
I am not very well versed in mediawiki tables, and I found github has some
incompatibilities too.
The markdown one looks better;
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md
> As was pointed out in the
> normalized transaction ID BIP, your proposal only addresses
> third-party malleability, since signers can simply change the
> transaction and re-sign it.
I have to disagree. That is not malleability. Creating a new document and re-
signing it is not changing anything. Its re-creating. Something that the owner
of the coin has every right to do.
> This is evident from the fact that inputs
> and outputs do not have a canonical order and it would appear that
> tokens can be re-ordered in segments.
Sorry, what is evident? You seem to imply that it is uncommon that you can
create two transactions of similar intent but using different bytes.
You would be wrong with this implication as this is very common. You can just
alter the order of the inputs, for instance.
I am unable to see what the point is you are trying to make. Is there a
question or a suggestion for improvement here?
> Dependencies of tokens inside a
> segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
> TxOutScript <-> TxOutValue).
Maybe you missed this line;
«TxInPrevHash and TxInPrevIndex
Index can be skipped, but in any input the PrevHash always has
to come first»
If you still see something alarming, let me know.
You can look at the code in Bitcoin Classic and notice that it really isn't
anything complicated or worrying.
> Finally, allowing miners to reject transactions with unknown fields
> makes the OP_NOPs unusable
Hmm, it looks like you are mixing terminology and abstraction-levels. OP_NOP
is a field from script and there is no discussion about any rejection based on
script in this BIP at all.
Rejection of transactions is done on there being unrecognised tokens in the
transaction formatting itself.
Thank you for your email to my BIP, I hope you got the answers you were
looking for :)

2023-06-07 17:53:27

- reply

📅 Original date posted:2016-09-21
📝 Original message:On Wednesday 21 Sep 2016 14:00:23 Andreas Schildbach via bitcoin-dev wrote:
> Just glancing over your BIP, I wonder if we should use Protobuf. It uses
> this "flexible" format already and is quite compact/binary. We use
> Protobuf already for the payment protocol, and there is very good tool
> support.
There is a lot of overlap between different binary formats. Looking through
the on-the-wire protocol you'll see that my spec is very similar. Practically
all the advantages of protobuf are present in CMF. I can write you a java
parser if you want, it should be easy to port from Qt/C++ code :)
https://github.com/bitcoinclassic/transactions
CMF:
https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md
There is no tool support needed, just one or two classes. Which personally I
think is an advantage.
Some advantages of CMF over protobuf from the top of my head;
* It reuses the var-int parsing that Bitcoin uses (which is itself slightly
different from others).
* zero-copy support (not relevant for this bip, though).
* Additional values addition (i.e. adding new data) is .. tricky in protobuf.
https://developers.google.com/protocol-buffers/docs/proto#updating
* In my experience parsing a message manually (like a SOX parser) is much
better in reporting errors and detecting wrong usages than auto-generated code
(but personally I'm not much a fan of auto-generated APIs) at all...
* Generated parsing/writing code will not be as fast as we can make it.
* CMF is more compact (uses less bytes) for its messages.
Protobuf is something I've used before and I think we can do better. I think
that CMF together with some support classes can do this better.

2023-06-07 17:53:27

- reply

📅 Original date posted:2016-09-22
📝 Original message:On Wednesday 21 Sep 2016 18:01:30 Gregory Maxwell via bitcoin-dev wrote:
> On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
>
> <bitcoin-dev at lists.linuxfoundation.org> wrote:
> > BIP number for my FT spec.
>
> This document does not appear to be concretely specified enough to
> review or implement from it.
>
> For example, it does not specify the serialization of "integer"
It refers to the external specification which is linked at the bottom.
In that spec you'll see that "Integer" is the standard var-int that Bitcoin
has used for years.
> nor does it specify how the
> presence of the optional fields are signaled
How does one signals an optional field except of in the spec? Thats the job of
a specification.
> nor the cardinality of
> the inputs or outputs.
Did you miss this in the 3rd table ? I suggest clicking on the github bips
repo link as tables are not easy to read in mediawiki plain format that the
email contained.
> For clearly variable length elements
> ('bytearray') no mention is made of their length encoding. etc.
Also in the external CMF spec.
> Without information like this, I don't see how someone could
> realistically begin reviewing this proposal.
I agree, that would be bad. Luckily that you just missed the link :)
Here it is;
https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md
> The motivation seems unclear to me as well: The scheme is described as
> 'flexible' but it appears to remove flexibility from the existing
> system. The "schema" appears to be hardcoded and never communicated.
Being hardcoded and never communicated is what the current format does to. How
does that "remove flexibility".
Also read my reply to Peter Todd on why this is flexible.
> If the goal is to simply have {snip}
It is not.
Thanks for asking, I understand that the CMF spec is useful to see as well.
Hopefully you can now review it properly since I linked to it above.
Cheers!

2023-06-07 17:53:26

- reply

📅 Original date posted:2016-09-22
📝 Original message:On Thursday, 22 September 2016 14:26:18 CEST Peter Todd wrote:
> > «The way towards that flexibility is to use a generic concept made
> > popular various decades ago with the XML format. The idea is that we
> > give each field a name and this means that new fields can be added or
> > optional fields can be omitted from individual transactions»
>
> That argument is not applicable to required fields:
The argument that optional fields can be omitted is not applicable to
required fields, you are correct. That should be rather obvious because
required fields are not optional fields.
> the code to get the
> fields from the extensible format is every bit as complex as the very
> simple code required to deserialize/serialize objects in the current
> system.
Probably a tiny bit more complex as the current format assumes a lot more.
You may have misread my email because there was no argument made towards
complexity. The argument was towards flexibility.
> In any case your BIP needs to give some explicit examples of hypothetical
> upgrades in the future, how they'd take advantage of this, and what the
> code to do so would look like.
Why?
> > > Also, if you're going to break compatibility with all existing
> > > software, it makes sense to use a format that extends the merkle
> > > tree down into the transaction inputs and outputs.
> >
> > Please argue your case.
>
> See my arguments re: segwit a few months ago, e.g. the hardware wallet
> txin proof use-case.
Please consider that I'm not going to search for something based on a vague
reference like that, if you want to convince me you could you at least
provide a URL?
You want me to see the value of your idea, I think you should at least
provide the argument. Isn't that fair?
Thanks for your email Peter, would love you to put a bit more time into
understanding flexible transactions and we can have a proper discussion
about it.

2023-06-07 17:53:26

- reply

📅 Original date posted:2016-09-21
📝 Original message:Thanks for your email Peter!
On Tuesday 20 Sep 2016 17:56:44 Peter Todd wrote:
> On Tue, Sep 20, 2016 at 07:15:45PM +0200, Tom via bitcoin-dev wrote:
> > === Serialization order===
> >
> > The tokens defined above have to be serialized in a certain order for the
> > transaction to be well-formatted. Not serializing transactions in the
> > order specified would allow multiple interpretations of the data which
> > can't be allowed.
>
> If the order of the tokens is fixed, the tokens themselves are redundant
> information when tokens are required; when tokens may be omitted, a simple
> "Some/None" flag to mark whether or not the optional data has been omitted
> is appropriate.
This is addressed in the spec;
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md
«The way towards that flexibility is to use a generic concept made popular
various decades ago with the XML format. The idea is that we give each
field a name and this means that new fields can be added or optional fields
can be omitted from individual transactions»
> Also, if you're going to break compatibility with all existing software, it
> makes sense to use a format that extends the merkle tree down into the
> transaction inputs and outputs.
Please argue your case.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20160921/613bee38/attachment-0001.sig>

2023-06-07 17:53:25

- reply

📅 Original date posted:2016-09-21
📝 Original message:On Tuesday 20 Sep 2016 21:31:47 Luke Dashjr wrote:
> On Tuesday, September 20, 2016 5:15:45 PM Tom via bitcoin-dev wrote:
> > As the title suggests, I would like to formally request the assignment of
> > a
> > BIP number for my FT spec.
>
> Please open a pull request on the bitcoin/bips repo after this has been
> discussed a bit on the ML.
> It seems from the later comments, that it is the end of the transaction as a
> whole. Yet a separator between the txid and non-txid data would probably be
> valuable, rather than hard-coding txid to skip signature types (which may
> be unknown to old nodes, when extended).
>
> > The OP_CHECKSIG is the most well known and, as its name implies, it
> > validates a signature.
> > In the new version of 'script' (version 2) the data that is signed is
> > changed to be equivalent to the transaction-id. This is a massive
> > simplification and also the only change between version 1 and version 2 of
> > script.
>
> This seems to be a major regression. What is the replacement for
> SIGHASH_SINGLE and SIGHASH_ANYONECANPAY?
How is this a regression? Can you explain what functionality is lost please?
> When revising OP_CHECKSIG, it would also be nice to add the ability to use
> *only* a hash of the prevout's scriptPubKey in the input, so that *when* the
> prevtx is malleated, the spending one remains valid. (This use case is
> currently not supported.)
Maybe for the next version of script :)
> > Notice that the token ScriptVersion is currently not allowed
> What happens if I put ScriptVersion=1 here?
The transaction is invalid...
> > === Block-malleability ===
> >
> > For this reason the merkle tree is extended to include (append) the hash
> > of
> > the v4 transactions (and those alone) where the hash is taken over a
>
> > data-blob that is build up from:
>
> How should nodes know where in the merkle-tree the txids end, and the
> v4hashes begin?
Because the txid based ones are not going away. So the number of transactions
in the block can be used to determine when the pure tx-id segment stops and
when the v4 hashes begin. Then its up to the client to rebuild the tree from
that list based on the larger input set to get the same root-node.
I clarified many little things on my clone of the bips, check there if you
want to see the details.

2023-06-07 17:53:24

- reply

📅 Original date posted:2016-09-20
📝 Original message:As the title suggests, I would like to formally request the assignment of a
BIP number for my FT spec.
Thank you!
Source;
https://github.com/zander/bips/blob/FlexTrans/bip-9999.mediawiki
<pre>
BIP: ??
Title: Flexible Transactions
Author: Tom Zander <tomz at freedommail.ch>
Status: Draft
Type: Standards Track
Created: 2016-07-27
</pre>
==Abstract==
This BIP describes the next step in making Bitcoin's most basic element,
the transaction, more flexible and easier to extend. At the same time this
fixes all known cases of malleability and resolves significant amounts of
technical debt.
==Summary==
Flexible Transactions uses the fact that the first 4 bytes in a transaction
determine the version and that the majority of the clients use a
non-consensus rule (a policy) to not accept transaction version numbers
other than those specifically defined by Bitcoin.
This BIP chooses a new version number, 4, and defines that the data
following the bytes for the version is in a format called Compact Message
Format (CMF). CMF is a flexible, token based format where each token is a
combination of a name, a format and a value. Because the name is added we
can skip unused tokens and we can freely add new tokens in a simple manner
in future. Soft fork upgrades will become much easier and cleaner this
way.
This protocol upgrade cleans up past soft fork changes like BIP68 which
reuse existing fields and do them in a much better to maintain and easier
to parse system. It creates the building blocks to allow new features to be
added much cleaner in the future.
It also shows to be possible to remove signatures from transactions with
minimal upgrades of software and still maintain a coherent transaction
history. Tests show that this can reduce space usage to about 75%.
==Motivation==
Token based file-formats are not new, systems like XML and HTMl use a
similar system to allow future growth and they have been quite successful
for decades in part because of this property.
Bitcoin needs a similar way of making the transaction future-proof because
re-purposing not used fields for new features is not good for creating
maintainable code.
Next to that this protocol upgrade will re-order the data-fields which
allows us to cleanly fix the malleability issue which means that future
technologies like Lightning Network will depend on this BIP being deployed.
At the same time, due to this re-ordering of data fields, it becomes very
easy to remove signatures from a transaction without breaking its tx-id,
which is great for future pruning features.
=== Tokens ===
In the compact message format we define tokens and in this specification we
define how these tokens are named, where they can be placed and which are
optional. To refer to XML, this specification would be the schema of
a transaction.
CMF tokens are triplets of name, format (like PositiveInteger) and value.
Names in this scope are defined much like an enumeration where the actual
integer value (id, below) is equally important to the written name.
If any token found that is not covered in the next table will make the
transaction that contains it invalid.
{| class="wikitable"
|-
! Name !! id !! Format !! Default Value !! Description
|-
|TxEnd || 0 ||BoolTrue || Required ||A marker that is the last
byte in the txid calculation
|-
|TxInPrevHash || 1 ||ByteArray|| Required ||TxId we are spending
|-
|TxPrevIndex || 2 ||Integer || 0 ||Index in prev tx we are
spending (applied to previous TxInPrevHash)
|-
|TxInScript || 3 ||ByteArray|| Required ||The 'input' part of the
script
|-
|TxOutValue || 4 ||Integer || Required ||Amount of satoshi to
transfer
|-
|TxOutScript || 5 ||ByteArray|| Required ||The 'output' part of the
script
|-
|LockByBlock || 6 ||Integer || Optional ||BIP68 replacement
|-
|LockByTime || 7 ||Integer || Optional ||BIP68 replacement
|-
|ScriptVersion || 8 ||Integer || 2 ||Defines script version for
outputs following
|-
|NOP_1x || 1x || . || Optional ||Values that will be ignored by
anyone parsing the transaction
|}
=== Scripting changes ===
In the current version of Bitcoin-script, version 1, there are various
opcodes that are used to validate the cryptographic proofs that users have
to provide in order to spend outputs.
The OP_CHECKSIG is the most well known and, as its name implies, it
validates a signature.
In the new version of 'script' (version 2) the data that is signed is
changed to be equivalent to the transaction-id. This is a massive
simplification and also the only change between version 1 and version 2 of
script.
=== Serialization order===
The tokens defined above have to be serialized in a certain order for the
transaction to be well-formatted. Not serializing transactions in the
order specified would allow multiple interpretations of the data which
can't be allowed.
There is still some flexibility and for that reason it is important for
implementors to remember that the actual serialized data is used for the
calculation of the transaction-id. Reading and writing it may give you a
different output and when the txid changes, the signatures will break.
At a macro-level the transaction has these segments. The order of the
segments can not be changed, but you can skip segments.
{| class="wikitable"
!Segment !! Description
|-
| Inputs || Details about inputs.
|-
| Outputs || Details and scripts for outputs
|-
| Additional || For future expansion
|-
| Signatures || The scripts for the inputs
|-
| TxEnd || End of the transaction
|}
The TxId is calculated by taking the serialized transaction without the
Signatures and the TxEnd and hashing that.
{| class="wikitable"
!Segment !! Tags !! Description
|-
|Inputs||TxInPrevHash and TxInPrevIndex||Index can be skipped, but in any
input the PrevHash always has to come first
|-
|Outputs||TxOutScript, TxOutValue||Order is not relevant
|-
|Additional||LockByBlock LockByTime NOP_1x
|-
|Signatures||TxInScript||Exactly the same amount as there are inputs
|-
|TxEnd||TxEnd
|}
TxEnd is there to allow a parser to know when one transaction in a stream
has ended, allowing the next to be parsed.
Notice that the token ScriptVersion is currently not allowed because we
don't have any valid value to give it. But if we introduce a new script
version it would be placed in the outputs segment.
=== Script v2 ===
The default value of ScriptVersion is number 2, as opposed to the version 1
of script that the is in use today. The version 2 is mostly identical
to version one, including upgrades made to it over the years and in the
future. The only exception is that the OP_CHECKSIG is made dramatically
simpler. The input-type for OP_CHECKSIG is now no longer configurable, it is
always '1' and the content that will be signed is the txid.
TODO: does check-multisig need its own mention?
=== Block-malleability ===
The effect of leaving the signatures out of the calculation of the
transaction-id implies that the signatures are also not used for the
calculation of the merkle tree. This means that changes in signatures
would not be detectable. Except naturally by the fact that missing or
broken signatures breaks full validation. But it is important to detect
modifications to such signatures outside of validating all transactions.
For this reason the merkle tree is extended to include (append) the hash of
the v4 transactions (and those alone) where the hash is taken over a
data-blob that is build up from:
1. the tx-id
2. the CMF-tokens 'TxInScript'
=== Future extensibility ===
The NOP_1x wildcard used in the table explaining tokens is actually a list
of 10 values that currently are specified as NOP (no-operation) tags.
Any implementation that supports the v4 transaction format should ignore
this field in a transaction. Interpreting and using the transaction as if
that field was not present at all.
Future software may use these fields to decorate a transaction with
additional data or features. Transaction generating software should not
trivially use these tokens for their own usage without cooperation and
communication with the rest of the Bitcoin ecosystem as miners certainly
have the option to reject transactions that use unknown-to-them tokens.
==Reference Implementation==
Bitcoin Classic includes this in its beta releases and a reference
implementation can be found at;
https://github.com/bitcoinclassic/bitcoinclassic/pull/186
==Deployment==
To be determined
==References==
[https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md]
CMF

2023-06-07 17:52:06

- reply

📅 Original date posted:2016-08-01
📝 Original message:I've been asked one question quite regularly and recently with more force.
The question is about Segregated Witness and specifically what a hard
fork based version would look like.
This is available online at my blog;
http://zander.github.io/posts/Flexible_Transactions/
But I'll publish the actual text here as well, hoping to hear from others in
the Bitcoin industry what they think about this approach.
Segregated Witness (or SegWit for short) is complex. It tries to solve
quite a lot of completely different and not related issues and it tries to
do this in a backwards compatible manner. Not a small feat!
So, what exactly does SegWit try to solve? We can find info of that in the
[benefits](https://bitcoincore.org/en/2016/01/26/segwit-benefits/) document.
* Malleability Fixes
* Linear scaling of sighash operations
* Signing of input values
* Increased security for multisig via pay-to-script-hash (P2SH)
* Script versioning
* Reducing UTXO growth
* Compact fraud proofs
As mentioned above, SegWit tries to solve these problems in a backwards
compatible way. This requirement is there only because the authors of
SegWit set themselves this requirement. They set this because they wished
to use a softfork to roll out this protocol upgrade.
**This post is going to attempt to answer the question if that is indeed
the best way of solving these problems.**
Starting with Malleability, the problem is that a transaction between being
created by the owner of the funds and being mined in a block is possible to
change in such a way that it still is valid, but the transaction identifier
(TX-id) has been changed. But before we dive into the deep, lets have some
general look at the transaction data first.
If we look at a
[Transaction](http://bitcoinfactswiki.github.io/Transaction) as it is
today, we notice some issues.
<table>
<tr><td colspan=2>Version</td><td>4 bytes</td></tr>
<tr><td colspan=2>Number of inputs</td><td>VarInt (between 1 and 9
bytes)</td></tr>
<tr><td rowspan=5 class=vertical>inputs</td><td>Prev transaction
hash</td><td>32 bytes.
<font color="red">Stored in reverse</font></td></tr>
<tr><td>Prev transaction index</td><td>4 bytes</td></tr>
<tr><td>TX-in script length</td><td><font color=red>Compact-
int</font></td></tr>
<tr><td>TX-in script</td><td>This is the witness data</td></tr>
<tr><td>Sequence-no/<a
href="https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki">CSV</a>
</td><td>4 bytes</td></tr>
<tr><td colspan=2>Number of outputs</td><td>VarInt (between 1 and 9
byte)</td></tr>
<tr><td rowspan=3 class=vertical>outputs</td><td>Value</td><td>Var
int</td></tr>
<tr><td>TX-out script length</td><td><font color=red>Compact-
int</font></td></tr>
<tr><td>TX-out script</td><td>bytearray</td></tr>
<tr><td colspan=2>NLockTime</td><td>4 bytes</td></tr>
</table>
</table>
The original transaction format as designed by
[Satoshi Nakamoto](http://bitcoinfactswiki.github.io/Satoshi_Nakamoto/)
had a 4 byte version. This design approach is common in the industry and
the way that this is used is that a new version is defined whenever any
field in the data structure needs changing. In Bitcoin we have not done
this and we are still at version 1.
What Bitcoin has done instead is make small, semi backwards-compatible,
changes.
For instance the [CHECKSEQUENCEVERIFY]
(http://bitcoinfactswiki.github.io/Script/#Locktime)
feature repurposes the sequence field as a way to add data
that would not break old clients. Incidentally, this specific change
(described in BIP68)
is not backwards compatible in the main clients as it depends on a
transaction version number being greater than 1, they all check for
Standard transactions and say that only version 1 is standard.
The design of having a version number implies that the designer wanted to
use hard forks for changes. A new client is required to know how to parse a
newly designed data structure, this should be obvious. So the idea is to
change the version number and so older clients would know they can't parse
this new transaction version. To keep operating, everyone would have to
upgrade to a client that supports this new transaction version.
Lets look at why we would want to change the version; I marked some items in
red that are confusing. Most specifically is that numbers are stored in 3
different, incompatible formats in transactions. Not really great and
certainly a source of bugs.
Transactions are cryptographically signed by the owner of the coin so
others can validate that he is actually allowed to move the coins.
The signature is stored in the `TX-in-script`.
Crypto-geeks may have noticed something weird that goes against any
textbooks knowledge. What this is is that a digital
signature has to be placed outside of the thing it signs. This is because
a digital signature protects against changes. But a signature itself would
cause this change. So you have to store the signature outside the thing you
sign.
Bitcoin's creator did something smart with how transactions are actually
signed so the signature actually doesn't have to be outside the
transaction. It works. Mostly. But we want it to work flawlessly
because currently this being too smart causes the dreaded malleability
issues where people have been known to lose money.
## What about SegWit?
SegWit actually solves only one of these items. It moves the signature out
of the transaction. SegWit doesn't fix any of the other problems in Bitcoin
transactions, it also doesn't change the version after making the
transaction's-meaning essentially unable to be understood by old clients.
Old clients will stop being able to check the SegWit type of transaction,
because the authors of SegWit made it so that SegWit transactions just have
a sticker of "All-Ok" on the car while moving the real data to the trailer,
knowing that the old clients will ignore the trailer.
SegWit wants to keep the data-structure of the transaction unchanged and it
tries to fix the data structure of the transaction. This causes friction
as you can't do both at the same time, so there will be a non-ideal
situation and hacks are to be expected.
The problem, then, is that SegWit introduces more technical debt, a term
software developers use to say the system-design isn't done and needs
significant more work. And the term 'debt' is accurate as over time
everyone that uses transactions will have to understand the defects to work
with this properly. Which is quite similar to paying interest.
Using a Soft fork means old clients will stop being able to validate
transactions, or even parses them fully. But these old clients are
themselves convinced they are doing full validation.
## Can we improve on that?
I want to suggest a way to **one-time** change the data-structure of the
transaction so it becomes much more future-proof and fix the issues it
gained over time as well. Including the malleability issue. It turns out
that this new data-structure makes all the other issues that SegWit fixes
quite trivial to fix.
I'm going to propose an upgrade I called;
> **Flexible Transactions**
Last weekend I wrote a little app (sources [here]
(http://zander.github.io/scaling/transactions))
that reads a transaction and then writes it out in a new format I've
designed for Bitcoin. Its based on ideas I've used for some time in other
projects as well, but this is the first open source version.
The basic idea is to change the transaction to be much more like modern
systems like JSON, HTML and XML. Its a 'tag' based format and has various
advantages over the closed binary-blob format.
For instance if you add a new field, much like tags in HTML, your old
browser will just ignore that field making it backwards compatible and
friendly to future upgrades.
Further advantages;
* Solving the malleability problem becomes trivial.
* tag based systems allow you to skip writing of unused or default values.
* Since we are changing things anyway, we can default to use only var-int
encoded data instead of having 3 different types in transactions.
* Adding a new tag later, (for instance ScriptVersion) is easy and doesn't
require further changes to the transaction data structure. All old clients
can still make sense of all the known data.
* The actual transaction turns out to be about 3% shorter average (calculated
over 200K transactions)
* Where SegWit adds a huge amount of technical debt, my Flexible
Transactions proposal instead amortizes a good chunk of technical debt.
An average **Flexible Transaction** will look like this;
<table>
<tr><td colspan=2>TxStart (Version)</td><td>0x04</td>
<td rowspan=5 class="vertical">TX-ID data</td></tr>
<tr><td rowspan=2 class=vertical>inputs</td><td>TX-ID I try to spent</td><td>1
+ 32 bytes</td></tr>
<tr><td>Index in prev TX-ID</td><td>varint</td></tr>
<tr><td rowspan=2 class=vertical>outputs</td><td>TX-out Value (in
Satoshis)</td><td>VarInt</td></tr>
<tr><td>TX-out script</td><td>bytearray</td></tr>
<tr><td>inputs</td><td>TX-in-script (Witness data)<td>bytearray</td>
<td rowspan=2 class=vertical>WID-data</td></tr>
<tr><td colspan=2>TxEnd</td><td>0x2C</td></tr>
</table>
Notice how the not used tags are skipped. The `NLockTime` and the
`Sequence` were not used, so they are skipped in the transaction.
The Flexible Transaction proposal uses a list of tags. Like JSON; `"Name:"
"Value"`. Which makes the content very flexible and extensible. Just
instead of using text, Flexible Transactions use a binary format.
The biggest change here is that the `TX-in-script` (aka the witness data) is
moved to be at the end of the transaction. When a wallet generates this new
type of transaction they will append the witness data at the end but the
transaction ID is calculated by hashing the data that ends before the
witness data.
The witness data typically contains a public key as well as a signature.
In the Flexible Transactions proposal the signature is made by signing exactly
the same set of data as is being hashed to generate the TX-input. Thereby
solving the malleability issue. If someone would change the transaction, it
would invalidate the signature.
I took 187000 recent transactions and checked what this change would do to
the size of a transaction with my test app I linked to above.
* Transactions went from a average size of 1712 bytes to 1660 bytes and a
median size of 333 to 318 bytes.
* Transactions can be pruned (removing of signatures) after they have been
confirmed. Then the size goes down to an average of 450 bytes or a median
of 101 bytes
* In contrary to SegWit new transactions get smaller for all clients with this
upgrade.
* Transactions, and blocks, where signatures are removed can expect up to
75% reduction in size.
## Broken OP_CHECKSIG scripting instruction
To actually fix the malleability issues at its source we need to fix this
instruction. But we can't change the original until we decide to make a
version 2 of the Script language.
This change is not really a good trigger to do a version two, and it
would be madness to do that at the same time as we roll out a new format of
the transaction itself. (too many changes at the same time is bound to
cause issues)
This means that in order to make the Flexible Transaction proposal actually
work we need to use one of the NOP codes unused in Script right now and
make it do essentially the same as OP_CHECKSIG, but instead of using the
overly complicated manner of deciding what it will sign, we just define it
to sign exactly the same area of the transaction that we also use to create
the TX-ID. (see right most column in the above table)
This new opcode should be relatively easy to code and it becomes really
easy to clean up the scripting issues we introduce in a future version of
script.
## So, how does this compare to SegWit.
First of all, introducing a new version of a transaction doesn't mean we
stop supporting the current version. So all this is perfectly backwards
compatible because clients can just continue making old style transactions.
Naturally, with the problems that had, but nobody will end up stuck.
Using a tagged format for a transaction is a one time hard fork to upgrade
the protocol and allow many more changes to be made with much lower impact
on the system in the future. There are parallels to SegWit, it strives for
the same goals, after-all. But where SegWit tries to adjust a static
memory-format by re-purposing existing fields, Flexible transactions presents
a coherent simple design that removes lots of conflicting concepts.
Most importantly, years after Flexible transactions has been introduced we
can continue to benefit from the tagged system to extend and fix issues we
find then we haven't thought of today. In the same, consistent, concepts.
We can fit more transactions in the same (block) space similarly to SegWit,
the
signatures (witness part) can be pruned by full nodes without causing any
security implications in both solutions. What SegWit doesn't do is
allowing unused features to not use space. So if a transaction doesn't use
NLockTime (which is near 100% of them) they will take space in SegWit but
not in this proposal. Expect your transactions to be smaller and thus lower
fee!
On size, SegWit proposes to gain 60% space. Which is by removing the
signatures minus the overhead introduced. In my tests Flexible
transactions showed 75% gain.
SegWit also describes changes how data is stored in the block. It creates
an extra 'branch' in the merkle tree. The Flexible Transactions proposal
is in essence solving the same problem as SegWit and the same solution for
blocks can be applied. Which means we can have that merkle tree solution
as well. No change.
At the start of the blog I mentioned a list of advantages that the authors
of SegWit included. It turns out that those advantages themselves are
completely not related to each other and they each have a very separate
solution to their individual problems. The tricky part is that due to the
requirement of old clients staying forwards-compatible they are forced to
push them all into the one 'fix'.
Lets go over them individually;
### Malleability Fixes
Using this new version of a transaction data-structure solves all forms of
known malleability.
### Linear scaling of sighash operations
This has been fixed in the BIP109 2MB hardfork quite some months ago.
### Signing of input values
This is included in this proposal.
### Increased security for multisig via pay-to-script-hash (P2SH)
The *Flexible transactions* proposal outlined in this document makes many
of the additional changes in SegWit really easy to add at a later time.
This change is one of them.
Bottom line, changing the security with a bigger hash in SegWit is only
included in SegWit because SegWit didn't solve the transaction versioning
problem making it trivial to do separately.
With flexible transactions this change can now be done at any time in the
future with minimal impact.
### Script versioning
Notice that this *only* introduces the versioning byte. It doesn't actually
introduce a new version of script.
This is an excellent example where tagged formats shine brighter than a
static memory format that SegWit uses because adding such a versioning tag
is much cleaner and much easier and less intrusive to do with
flexible transactions. Just add a new tag that defaults to version 1 so the
old transactions not having the tag stay consistent.
Imagine having to include "body background=white" in each and every html
page because it was not allowed to leave it out. Thats what SegWit does
right now. Even though it doesn't actually support changing it yet.
### Reducing UTXO growth
I suggest you read this
[point](https://bitcoincore.org/en/2016/01/26/segwit-benefits/#reducing-utxo-growth)
for yourself, its rather interestingly technical and I'm sure many will not
fully grasp the idea. The bottom line of that they are claiming the
UTXO database will avoid growing because SegWit doesn't allow more
customers to be served.
I don't even know how to respond to such a solution. Its throwing out the
baby with the bath water.
Database technology has matured immensely over the last 20 years, the
database is tiny in comparison to what free and open source databases can
do today. Granted, the UTXO database is slightly unfit for a normal SQL
database, but telling customers to go elsewhere has never worked out for
products in the long term.
### Compact fraud proofs
Again, not really included in SegWit, just started as a basis. The exact
same basis is suggested for flexible transactions, and as such this is
identical.
### What do we offer that SegWit doesn't offer?
* A transaction becomes extensible. Future modifications are cheap.
* A transaction gets smaller. Using less features also takes less space.
* We only use one instead of 3 types of encodings for integers.
* We remove technical debt and simplify implementations. SegWit does the
opposite.
## Conclusions
SegWit has some good ideas and some needed fixes. Stealing all the good
ideas and improving on them can be done, but require a hard fork. This post
shows that the advantages are quite significant and certainly worth it.
We introduced a tagged data structure. Conceptually like JSON and XML in
that it is flexible, but the proposal is a compact and fast binary format.
Using the Flexible Transaction data format allows many future
innovations to be done cleanly in a consistent and, at a later stage, a
more backwards compatible manner than SegWit is able to do, even if given
much more time. We realize that separating the fundamental issues that
SegWit tries to fix all in one go, is possible and each becomes much lower
risk to Bitcoin as a system.
After SegWit has been in the design stage for a year and still we find
show-stopping issues, delaying the release, I would argue that dropping
the requirement of staying backwards compatible should be on the table.
The introduction of the *Flexible Transaction* upgrade has big benefits
because the transaction design becomes extensible.
A hardfork is done once to allow us to do soft upgrades in the future.
The Flexible transaction solution lowers the amount of changes required in
the entire ecosystem. Not just for full nodes. Especially considering that
many tools and wallets use shared libraries between them to actually create
and parse transactions.
The Flexible Transaction upgrade proposal should be considered by anyone
that cares about the protocol stability because its risk of failures
during or after upgrading is several magnitudes lower than SegWit is and
it removes technical debt that will allow us to innovate better into the
future.
Flexible transactions are smaller, giving significant savings after pruning
over SegWit.
Thanks

2023-06-07 17:51:58

- reply

📅 Original date posted:2016-07-26
📝 Original message:> #Basic idea:
>
> Ideally, all miners would begin hashing the next block at exactly the same
> time. Miners with a head start are more profitable, and the techniques that
> help miners receive and validate blocks quickly create centralization
> pressure.
>
> What if there was something that acted like the starting flag at a race,
> which could suddenly wave and cause all of the miners to simultaneously
> begin hashing the next block?
>
> #Implementation:
>
> Let a sync flag be a message consisting of:
>
> 1. Hash of the previous block.
> 2. Bitcoin address
> 3. Nonce
>
> This tiny message could propagate through the network at maximum speed. If
> miners had to include the hash of this flag in the next block, then all
> miners wait for this flag, and when it suddenly spread through the network,
> all miners could simultaneously begin hashing the next block.
What you describe in this part of your message can be done with no forks
whatsoever and I think that this is enough. Don't really see the reason for
any change in funding.
The idea of sending out a block header is essentially what I called
"optimistic mining" and has been described in more detail in my blog here;
http://zander.github.io/posts/Innovation%20-%20OnlineScaling/
The video explains with graphics too...
You may find this interesting :)

2023-06-07 17:50:36

- reply

📅 Original date posted:2016-05-12
📝 Original message:On Wednesday 11 May 2016 22:58:48 Gregory Maxwell via bitcoin-dev wrote:
> On Wed, May 11, 2016 at 10:42 PM, Timo Hanke via bitcoin-dev
>
> <bitcoin-dev at lists.linuxfoundation.org> wrote:
> > This is what I meant. If existing hardware gets forked-out it will
> > inevitably lead to the creation of an altcoin. Simply because the hardware
> > exists and can't be used for anything else both chains will survive. I was
> > only comparing the situation to a contentious hardfork that does not fork
> > out any hardware. If the latter one is suspected to lead to the permanent
> > existence of two chains then a hardfork that forks out hardware is even
> > more likely to do so (I claim it's guaranteed).
>
> There are already many altcoins out there, we could not prevent that
> even if we wanted to. New ones are created all the time.
Comparing apples and oranges.
Altcoins have their own genesis block, the example Timo was talking about was
a fork in the Bitcoin blockchain.
But its good to know you don't mind a fork in the Bitcoin chain, I'll remember
that.

2023-06-07 17:50:20

- reply

📅 Original date posted:2016-05-09
📝 Original message:On Monday 09 May 2016 13:40:55 Peter Todd wrote:
> >> [It's a little disconcerting that you appear to be maintaining a fork
> >> and are unaware of this.]
> >
> >ehm...
>
> Can you please explain why you moved the above part of gmaxwell's reply to
> here,
A personal attack had no place in the technical discussion, I moved it out.
Initially I asked him to please avoid personal attacks, but I thought better
of it and edited my reply to just "ehm...".
The moderators failed to catch his aggressive tone while moderating my post
(see archives) for being too aggressive.
I'm sure this message will also not be allowed through. I would not even blame
the moderators since this, and Peters, messages were both off-topic.
I thank you for todays talks, it makes me certain of the thing I said this
weekend on Reddit that this list is not a suitable place for all the different
stakeholders to talk on a level playing field.
If any of you agree, please urge the approach that we replace the entire
moderation team with a new one. This will be the least painful solution for
everyone in the ecosystem.
Thanks again.

2023-06-07 17:50:19

- reply

📅 Original date posted:2016-05-09
📝 Original message:On Monday 09 May 2016 10:43:02 Gregory Maxwell wrote:
> On Mon, May 9, 2016 at 9:35 AM, Tom Zander via bitcoin-dev
>
> <bitcoin-dev at lists.linuxfoundation.org> wrote:
> > You misunderstand the networking effects.
> > The fact that your node is required to choose which one to set the
> > announce
> > bit on implies that it needs to predict which node will have the best data
> > in the future.
>
> Not required. It may.
It is required, in the reference of wanting to actually use compact block
relay.
> Testing on actual nodes in the actual network (not a "lab") shows
Apologies, I thought that the term was wider known. "Laboratory situations"
is used where I am from as the opposite of real-world messy and unpredictable
situations.
So, your measurements may be true, but are not useful to decide how well it
behaves under less optimal situations. aka "the real world".
> This also _increases_ robustness. Right now a single peer failing at
> the wrong time will delay blocks with a long time out.
If your peers that were supposed to send you a compact block fail, then you'll
end up in exactly that same situation again. Only with various timeouts in
between before you get your block making it a magnitude slower.
In networking this is solved by reacting instead of predicting. The network is
not stable. Your protocol design assumes it to be.
> > Another problem with your solution is that nodes send a much larger amount
> > of unsolicited data to peers in the form of the thin-block compared to
> > the normal inv or header-first data.
>
> "High bandwidth" mode
Another place where I may have explained better.
This is not about the difference about the two modes of your design.
This is about the design as a whole. As compared to current.
> > Am I to understand that you choose the solution based on the fact that
> > service bits are too expensive to extend? (if not, please respond to my
> > previous question actually answering the suggestion)
> >
> > That sounds like a rather bad way of doing design. Maybe you can add a
> > second service bits field of message instead and then do the compact
> > blocks correctly.
> Service bits are not generally a good mechanism for negating optional
> peer-local parameters.
Service bits are exactly the right solution to indicate additional p2p
feature-support.
> [It's a little disconcerting that you appear to be maintaining a fork
> and are unaware of this.]
ehm...
> > Wait, you didn't steal the variable length encoding from an existing
> > standard and you programmed a new one?
>
> This is one of the two variable length encodings used for years in
> Bitcoin Core. This is just the first time it's shown up in a BIP.
>
> > Look at UTF-8 on wikipedia, you may have "invented" the same encoding that
> > IBM published in 1992.
>
> The similarity with UTF-8 is that both are variable length and some
> control information is in the high bits. The similarity ends there.
That's all fine and well, it doesn't at any point take away from my point that
any specification should NOT invent something new that has for decades had a
great specification already.
If you make a spec to be used by all nodes, on the wire, don't base it on your
proprietary implementation. Please.
> > Just the first (highest) 8 bytes of a sha256 hash.
> >
> > The amount of collisions will not be less if you start xoring the rest.
> > The whole reason for doing this extra work is also irrelevant as a spam
> > protection.
>
> Then you expose it to a trivial collision attack: To find two 64 bit
> hashes that collide I need perform only roughly 2^32 computation. Then
> I can send them to the network.
No, you still need to have done a POW.
Next to that, your scheme is 2^32 computations *and* some XORs. The XORs are
percentage wise a rounding error on the total time. So your argument also
destroys your own addition.
> This issue is eliminated by salting the hash.
The issue is better eliminated by not allowing nodes to send uninvited large
messages.
I don't think we're getting anywhere.
I'm not sold on your design and I explained why. I tried explaining in this
email some misconceptions that may have appeared after my initial emails. I
hope things are more clear.

2023-06-07 17:50:00

- reply

📅 Original date posted:2016-03-25
📝 Original message:On Thursday 24 Mar 2016 13:20:48 Chris via bitcoin-dev wrote:
> As far as the use cases others mentioned, connecting and SPV wallet to
> your full node is certainly one. It would make it easy to, say, connect
> the android bitcoin-wallet to your own node. I've hacked on that wallet
> to make it connect to my .onion node, but it's very slow border-line
> unusable. Basic encryption and authentication would make that viable.
What about using some interface, much like the JSON one (but more likely the
zeroMQ one) instead? Would that not solve the problem?
I'm thinking that would not be a replacement for a full-node-connection but in
addition.
Which means that some questions can be asked over that channel that you need
authentication for. It would be a much better separation of concerns.

2023-06-07 17:49:58

- reply

📅 Original date posted:2016-03-25
📝 Original message:On Friday 25 Mar 2016 19:43:00 Jonas Schnelli via bitcoin-dev wrote:
> An encrypted channel together with a trusted full node would finally
> allow to have a secure and save SPV communication.
I guess my question didn't get across.
Why would you want to make your usecase do connections over the peer2peer
(net.cpp) connection at all?
Mixing messages that are being sent to everyone and encrypted messages is
asking for trouble.
Making your private connection out-of-band would work much better.
> > Also, you didn't actually address the attack-vector.
>
> Which attack-vector?
The statistical attack I mentioned earlier. Which comes from knowing which
plain text messages are being sent over the encrypted channel, So as long as
you keep saying you want to encrypt data that identical copies of are being
sent to other nodes at practically the same time, you will keep being
vulnerable to that.

2023-06-07 17:49:57

- reply

📅 Original date posted:2016-03-25
📝 Original message:On Wednesday 23 Mar 2016 22:55:34 Jonas Schnelli via bitcoin-dev wrote:
> >> I have just PRed a draft version of two BIPs I recently wrote.
> > * why would you not allow encryption on non-pre-approved connections?
>
> The encryption should be optional.
> The proposed authentication scheme does take care of the
> identity-management and therefor prevent MITM attacks.
> Without the identity management, you might not detect sending/receiving
> encrypted data from/to a MITM.
If you want to extend the Bitcoin protocol itself, you will have to resolve
that. Which many other solutions do (ssh for instance).
It would not be Ok to have an peer-to-peer encryption system that doesn't
allow non-pre-approved connections.
> > * What is the reason for using the p2p code to connect a wallet to a node?
> > I suggest using one of the other connection methods to connect to the
> > node.
> > This avoids a change in the bitcoin protocol for a very specific usecase.
>
> Most known use-case: SPV.
You didn't answer the question.
> > * Why do you want to do a per-message encryption (wrapping the original)?
> > Smaller messages that contain predictable content and are able to be
> > matched to the unencrypted versions on the wire send to other nodes will
> > open this scheme up to various old statistical attacks.
>
> It's probably extremely inefficient to create a constant time stream.
Your use of "probably" makes me wonder if you already have an implementation.
Doing any encryption and handshaking design *without* actually having it coded
and gone though testing yet makes no sense.
I do not belief Bitcoin will benefit from "design by committee" where a
specification is drawn before an implementation is written.
Also, you didn't actually address the attack-vector.
> >> Responding peers must ignore (banning would lead to fingerprinting) the
> >
> > requesting peer after 5 unsuccessfully authentication tries to avoid
> > resource attacks.
> >
> > Any implementation of that kind would itself again be open to resource
> > attacks.
> > Why 5? Do you want to allow a node to make a typo?
>
> Good point. Maybe one false try should lead to ignoring the peer.
That doesn't take away the resource attack at all.
> >> To ensure that no message was dropped or blocked, the complete
> >> communication>
> > must be hashed (sha256). Both peers keep the SHA256 context of the
> > encryption session. The complete <code>enc</code> message (leaving out
> > the hash itself) must be added to the hash-context by both parties.
> > Before sending a <code>enc</code> command, the sha256 context will be
> > copied and finalized.
> >
> > You write "the complete communication must be hashed" and every message
> > has a hash of the state until it is at that point.
> > I think you need to explain how that works specifically.
>
> This is a relative simple concept and does not require rehashing the
> whole communication.
Apologies, I should have been more clear; the BIP should specify the actual
algorithm, otherwise you can't create an implementation from just reading the
BIP.
Also, this may be a good time to ask why you want to have a per-message
encryption?
Practically every single popular end-to-end encryption uses one approach or
another were it just encrypts as another layer. (the L in ssl). You are
mixing layers, and unless you do that for a very good reason, or have a very
good reason why everyone else is doing it wrong, I suggest using a layered
encryption approach.

2023-06-07 17:49:56

- reply

📅 Original date posted:2016-03-23
📝 Original message:On Wednesday 23 Mar 2016 16:24:12 Jonas Schnelli via bitcoin-dev wrote:
> Hi
>
> I have just PRed a draft version of two BIPs I recently wrote.
> https://github.com/bitcoin/bips/pull/362
I suggest running a spellchecker ;)
Some questions;
* why would you not allow encryption on non-pre-approved connections?
* we just removed (ssl) encryption from the JSON interface, how do you suggest
this encryption to be implemented without openSSL?
* What is the reason for using the p2p code to connect a wallet to a node?
I suggest using one of the other connection methods to connect to the node.
This avoids a change in the bitcoin protocol for a very specific usecase.
* Why do you want to do a per-message encryption (wrapping the original)?
Smaller messages that contain predictable content and are able to be matched
to the unencrypted versions on the wire send to other nodes will open this
scheme up to various old statistical attacks.
> Responding peers must ignore (banning would lead to fingerprinting) the
requesting peer after 5 unsuccessfully authentication tries to avoid resource
attacks.
Any implementation of that kind would itself again be open to resource
attacks.
Why 5? Do you want to allow a node to make a typo?
> To ensure that no message was dropped or blocked, the complete communication
must be hashed (sha256). Both peers keep the SHA256 context of the encryption
session. The complete <code>enc</code> message (leaving out the hash itself)
must be added to the hash-context by both parties. Before sending a
<code>enc</code> command, the sha256 context will be copied and finalized.
You write "the complete communication must be hashed" and every message has a
hash of the state until it is at that point.
I think you need to explain how that works specifically.

2023-06-07 17:49:47

- reply

📅 Original date posted:2016-03-18
📝 Original message:On Wednesday 16 Mar 2016 22:24:30 Luke Dashjr via bitcoin-dev wrote:
> It is important that the forum for comments have a low barrier of use. The
> Bitcoin Wiki requires only a request for editing privileges, whereas GitHub
> wiki would require reading and agreeing to a lengthy Terms of Service
> contract.
I'd argue that neither of those two qualifies in that case.
I second BTCDraks' objection.