đź“… Original date posted:2020-08-18
đź“ť Original message:> On Aug 18, 2020, at 10:26, Matt Corallo <lf-lists at mattcorallo.com> wrote:
>
> There are several cases where a new message has been sent as a part of a negotiation without changing the protocol version.
Such as?
> You may chose to ignore that, but that doesn't mean that it isn't an understood and even relied upon feature of the Bitcoin P2P protocol. If you wish to fail connections to new nodes (and risk network splits, as Suhas points out), then you may do so, but that doesn't make it a part of the Bitcoin P2P protocol that you must do so. Of course there is no "official document" by which we can make a formal appeal, but historical precedent suggests otherwise.
You are misrepresenting “historical precedent”. I’ve seen several attempts to require arbitrary traffic over the years and none have been realized.
> Still, I think we're talking pedantics here, and not in a useful way.
Not to be pedantic, but I don’t know what that means.
> Ultimately we need some kind of negotiation which is flexible in allowing different software to negotiate different features without a global lock-step version number increase.
I have shown below how that already works.
> Or, to put it another way, if a feature is fully optional, why should there be a version number increase for it
For the reasons previously given.
> - the negotiation of it is independent and a version number only increases confusion over which change "owns" a given version number.
Presumably this is why we have a standards process. Any new message implies ownership. Deconflicting that is required, which implies it can easily be version associated (as it has been).
> I presume you'd support a single message that lists the set of features which a node (optionally) wishes to support on the connection. This proposal is fully equivalent to that, instead opting to list them as individual messages instead of one message, which is a bit nicer in that they can be handled more independently or by different subsystems including even the message hashing.
This presumes an implementation. As part of the handshake, collection of an arbitrary set of messages is a significant and unnecessary complication *of the protocol*. Extension of the verack is not. It is the simplest change possible to implement the desired behavior. Each peer simply supplies the matrix of sub-protocols it supports and only those supported by both are allowed. There is no reason for the protocol to split that matrix up into multiple messages, requiring termination. Independent messages exist because of timing or ordering requirements. Implementing dependent messages as if they were independent is wasteful and complicating.
I’m well aware of the inefficiency produced by version linearity in the face of optional sub-protocols. The protocol must negotiate to the version where it can then negotiate support, which has been done. I support creating a simpler system, eliminating these extra messages. The existing numeric version can be reserved exclusively for “must” implement, and can be used to signal an extension to the verack. The verack can then carry a list of “may” or “should” sub-protocols for final negotiation.
The format of the matrix is arbitrary, but the requirement is to list a set of optional sub-protocols. This implies a namespace. This implies “ownership“ of names. In other words, that coordination requirement is not eliminated.
e
> Matt
>
>> On 8/18/20 12:54 PM, Eric Voskuil wrote:
>> “Bitcoin protocol has always expected clients to ignore unknown messages”
>> This is not true. Bitcoin has long implemented version negotiation, which is the opposite expectation. Libbitcoin’s p2p protocol implementation immediately drops a peer that sends an invalid message according to the negotiated version. The fact that a given client does not validate the protocol does not make it an expectation that the protocol not be validated.
>> Features can clearly be optional within an actual protocol. There have been post-handshake negotiations implemented for optional messages which are valid at the negotiated version. The protocol may be flexible while remaining validateable. There is no reason to force a client to accept unknown message traffic.
>> A generalized versioning change can be implemented in or after the handshake. The latter is already done on an ad-hoc basis. The former is possible as long as the peer’s version is sufficient to be aware of the behavior. This does not imply any need to send invalid messages. The verack itself can simply be extended with a matrix of feature support. There is no reason to complicate negotiation with an additional message(s).
>> FWIW, bip37 did this poorly, adding a feature field to the version message, resulting in bip60. Due to this design, older protocol-validating clients were broken. In this case it was message length that was presumed to not be validated.
>> e
>>>> On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org> wrote:
>>>
>>> This sounds like a great idea!
>>>
>>> Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. The Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple numerical version field, but that is wholly impractical for any diverse network - some clients may not wish to implement every possible new relay mechanic, and why should they have to in order to use other new features?
>>>
>>> Bitcoin protocol changes have, many times in recent history, been made via new dummy "negotiation" messages, which take advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given that pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the list of features that you support to negotiate what the connection will be capable of. The exact way we do that doesn't matter much, and sending it as a stream of messages which each indicate support for a given protocol feature perfectly captures the pattern that has been used in several recent network upgrades, keeping consistency.
>>>
>>> Matt
>>>
>>> On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:
>>>> Hi,
>>>> Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side. In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.
>>>> Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.
>>>> In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK. A draft of my proposal is available here[2].
>>>> I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem). When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version. So having an understanding that this is the standard of how other network clients operate would be helpful.
>>>> If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.
>>>> Thanks,
>>>> Suhas Daftuar
>>>> [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html <https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html>;
>>>> [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki <https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki>;
>>>> _______________________________________________
>>>> bitcoin-dev mailing list
>>>> bitcoin-dev at lists.linuxfoundation.org
>>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>> _______________________________________________
>>> bitcoin-dev mailing list
>>> bitcoin-dev at lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Eric Voskuil [ARCHIVE] /
npub1sg…ppx3c
2023-06-07 18:26:26
in reply to nevent1q…q2hz
Author Public Key
npub1sgs97fe0n9wehe6zw7drcxdz4cy9yt9pfqjv8gasz5jlk4zezc0quppx3cPublished at
2023-06-07 18:26:26Event JSON
{
"id": "a00b26bfeaf363d72b5fe7c7939b742043930347bfe020307cf95af2abe2df46",
"pubkey": "82205f272f995d9be742779a3c19a2ae08522ca14824c3a3b01525fb5459161e",
"created_at": 1686162386,
"kind": 1,
"tags": [
[
"e",
"43edfd73733fa9cb455dfd9022ab700e39020c27283a7e2f0343a8c6fb631001",
"",
"root"
],
[
"e",
"0cb9475874c62c3ae23a0bd96089ed21233e55067df7a6b4ef4f24bcb0575e03",
"",
"reply"
],
[
"p",
"cd753aa8fbc112e14ffe9fe09d3630f0eff76ca68e376e004b8e77b687adddba"
]
],
"content": "📅 Original date posted:2020-08-18\n📝 Original message:\u003e On Aug 18, 2020, at 10:26, Matt Corallo \u003clf-lists at mattcorallo.com\u003e wrote:\n\u003e \n\u003e There are several cases where a new message has been sent as a part of a negotiation without changing the protocol version.\n\nSuch as?\n\n\u003e You may chose to ignore that, but that doesn't mean that it isn't an understood and even relied upon feature of the Bitcoin P2P protocol. If you wish to fail connections to new nodes (and risk network splits, as Suhas points out), then you may do so, but that doesn't make it a part of the Bitcoin P2P protocol that you must do so. Of course there is no \"official document\" by which we can make a formal appeal, but historical precedent suggests otherwise.\n\nYou are misrepresenting “historical precedent”. I’ve seen several attempts to require arbitrary traffic over the years and none have been realized.\n\n\u003e Still, I think we're talking pedantics here, and not in a useful way.\n\nNot to be pedantic, but I don’t know what that means.\n\n\u003e Ultimately we need some kind of negotiation which is flexible in allowing different software to negotiate different features without a global lock-step version number increase.\n\nI have shown below how that already works.\n\n\u003e Or, to put it another way, if a feature is fully optional, why should there be a version number increase for it\n\nFor the reasons previously given.\n\n\u003e - the negotiation of it is independent and a version number only increases confusion over which change \"owns\" a given version number.\n\nPresumably this is why we have a standards process. Any new message implies ownership. Deconflicting that is required, which implies it can easily be version associated (as it has been).\n\n\u003e I presume you'd support a single message that lists the set of features which a node (optionally) wishes to support on the connection. This proposal is fully equivalent to that, instead opting to list them as individual messages instead of one message, which is a bit nicer in that they can be handled more independently or by different subsystems including even the message hashing.\n\nThis presumes an implementation. As part of the handshake, collection of an arbitrary set of messages is a significant and unnecessary complication *of the protocol*. Extension of the verack is not. It is the simplest change possible to implement the desired behavior. Each peer simply supplies the matrix of sub-protocols it supports and only those supported by both are allowed. There is no reason for the protocol to split that matrix up into multiple messages, requiring termination. Independent messages exist because of timing or ordering requirements. Implementing dependent messages as if they were independent is wasteful and complicating.\n\nI’m well aware of the inefficiency produced by version linearity in the face of optional sub-protocols. The protocol must negotiate to the version where it can then negotiate support, which has been done. I support creating a simpler system, eliminating these extra messages. The existing numeric version can be reserved exclusively for “must” implement, and can be used to signal an extension to the verack. The verack can then carry a list of “may” or “should” sub-protocols for final negotiation.\n\nThe format of the matrix is arbitrary, but the requirement is to list a set of optional sub-protocols. This implies a namespace. This implies “ownership“ of names. In other words, that coordination requirement is not eliminated.\n\ne\n\n\u003e Matt\n\u003e \n\u003e\u003e On 8/18/20 12:54 PM, Eric Voskuil wrote:\n\u003e\u003e “Bitcoin protocol has always expected clients to ignore unknown messages”\n\u003e\u003e This is not true. Bitcoin has long implemented version negotiation, which is the opposite expectation. Libbitcoin’s p2p protocol implementation immediately drops a peer that sends an invalid message according to the negotiated version. The fact that a given client does not validate the protocol does not make it an expectation that the protocol not be validated.\n\u003e\u003e Features can clearly be optional within an actual protocol. There have been post-handshake negotiations implemented for optional messages which are valid at the negotiated version. The protocol may be flexible while remaining validateable. There is no reason to force a client to accept unknown message traffic.\n\u003e\u003e A generalized versioning change can be implemented in or after the handshake. The latter is already done on an ad-hoc basis. The former is possible as long as the peer’s version is sufficient to be aware of the behavior. This does not imply any need to send invalid messages. The verack itself can simply be extended with a matrix of feature support. There is no reason to complicate negotiation with an additional message(s).\n\u003e\u003e FWIW, bip37 did this poorly, adding a feature field to the version message, resulting in bip60. Due to this design, older protocol-validating clients were broken. In this case it was message length that was presumed to not be validated.\n\u003e\u003e e\n\u003e\u003e\u003e\u003e On Aug 18, 2020, at 07:59, Matt Corallo via bitcoin-dev \u003cbitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\u003e\u003e\u003e \n\u003e\u003e\u003e This sounds like a great idea!\n\u003e\u003e\u003e \n\u003e\u003e\u003e Bitcoin is no longer a homogeneous network of one client - it is many, with different features implemented in each. The Bitcoin protocol hasn't (fully) evolved to capture that reality. Initially the Bitcoin protocol had a simple numerical version field, but that is wholly impractical for any diverse network - some clients may not wish to implement every possible new relay mechanic, and why should they have to in order to use other new features?\n\u003e\u003e\u003e \n\u003e\u003e\u003e Bitcoin protocol changes have, many times in recent history, been made via new dummy \"negotiation\" messages, which take advantage of the fact that the Bitcoin protocol has always expected clients to ignore unknown messages. Given that pattern, it makes sense to have an explicit negotiation phase - after version and before verack, just send the list of features that you support to negotiate what the connection will be capable of. The exact way we do that doesn't matter much, and sending it as a stream of messages which each indicate support for a given protocol feature perfectly captures the pattern that has been used in several recent network upgrades, keeping consistency.\n\u003e\u003e\u003e \n\u003e\u003e\u003e Matt\n\u003e\u003e\u003e \n\u003e\u003e\u003e On 8/14/20 3:28 PM, Suhas Daftuar via bitcoin-dev wrote:\n\u003e\u003e\u003e\u003e Hi,\n\u003e\u003e\u003e\u003e Back in February I posted a proposal for WTXID-based transaction relay[1] (now known as BIP 339), which included a proposal for feature negotiation to take place prior to the VERACK message being received by each side. In my email to this list, I had asked for feedback as to whether that proposal was problematic, and didn't receive any responses.\n\u003e\u003e\u003e\u003e Since then, the implementation of BIP 339 has been merged into Bitcoin Core, though it has not yet been released.\n\u003e\u003e\u003e\u003e In thinking about the mechanism used there, I thought it would be helpful to codify in a BIP the idea that Bitcoin network clients should ignore unknown messages received before a VERACK. A draft of my proposal is available here[2].\n\u003e\u003e\u003e\u003e I presume that software upgrading past protocol version 70016 was already planning to either implement BIP 339, or ignore the wtxidrelay message proposed in BIP 339 (if not, then this would create network split concerns in the future -- so I hope that someone would speak up if this were a problem). When we propose future protocol upgrades that would benefit from feature negotiation at the time of connection, I think it would be nice to be able to use the same method as proposed in BIP 339, without even needing to bump the protocol version. So having an understanding that this is the standard of how other network clients operate would be helpful.\n\u003e\u003e\u003e\u003e If, on the other hand, this is problematic for some reason, I look forward to hearing that as well, so that we can be careful about how we deploy future p2p changes to avoid disruption.\n\u003e\u003e\u003e\u003e Thanks,\n\u003e\u003e\u003e\u003e Suhas Daftuar\n\u003e\u003e\u003e\u003e [1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html \u003chttps://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-February/017648.html\u003e\n\u003e\u003e\u003e\u003e [2] https://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki \u003chttps://github.com/sdaftuar/bips/blob/2020-08-generalized-feature-negotiation/bip-p2p-feature-negotiation.mediawiki\u003e\n\u003e\u003e\u003e\u003e _______________________________________________\n\u003e\u003e\u003e\u003e bitcoin-dev mailing list\n\u003e\u003e\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\u003e\u003e _______________________________________________\n\u003e\u003e\u003e bitcoin-dev mailing list\n\u003e\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev",
"sig": "cf965d4537a1b46bdcfc6451d538a65e6ea64c0af4e545e888aa9206da85ab61091c1ce160e3a6657cf3aaacb0e1bd1db6d297a4f81475a6e5c3514e9e07d0ee"
}