📅 Original date posted:2014-07-15
📝 Original message:I was part of adding in that test vector, and I think it's a good test
vector since it is an extreme edge-case of the current definition: If the
BIP38 proposal allows any password that can be in UTF-8, NFC normalized
form, those characters cover the various edge cases (combining characters,
null character, astral range) that if your implementation doesn't handle,
then it can't really be said to be "BIP38-compatible/compliant", right?
The "passphrase" in the test vector is NOT in NFC form; that's the point.
Whatever implementation gets designed has to assume the input is not
already NFC-normalized and needs to handle/sanitize that input before
further processing. To test your implementation for compliance, you should
not be inputting the NFC-normalized bytestring as the password input, you
should be entering the original passphrase as the test. My original pull
request for this change (https://github.com/bitcoin/bips/pull/29) shows a
Python and a NodeJS way to input that test vector password as intended.
Some input devices may already handle the input as NFC, which is great, but
per the BIP38 proposal, that shouldn't be assumed, so various
implementations are cross-compatible. If one implementation assumes the
input is already NFC, they may encode/decode the password incorrectly, and
lock a user out of their wallet. Android allows different user keyboards to
be used, so I'm guessing there's one somewhere that allows manual entry of
unicode codepoints that could be used to enter a null character, and with
the next version of iOS, Apple devices will also get custom keyboard
options, too, so even if the default Apple keyboard does NFC-form properly,
other developers' keyboards may not. So while it is an extreme edge case,
that is not very likely to be used as a "real password" by any user, that's
what test vectors are for: to test for the edge case that you might not
have expected and handled in your implementation.
Brooks
On Tue, Jul 15, 2014 at 8:07 AM, Eric Winer <enwiner at gmail.com> wrote:
> I don't know for sure if the test vector is correct NFC form. But for
> what it's worth, the Pile of Poo character is pretty easily accessible on
> the iPhone and Android keyboards, and in this string it's already in NFC
> form (f09f92a9 in the test result). I've certainly seen it in usernames
> around the internet, and wouldn't be surprised to see it in passphrases
> entered on smartphones, especially if the author of a BIP38-compatible app
> includes a (possibly ill-advised) suggestion to have your passphrase
> "include special characters".
>
> I haven't seen the NULL character on any smartphone keyboards, though - I
> assume the iOS and Android developers had the foresight to know how much
> havoc that would wreak on systems assuming null-terminated strings. It
> seems unlikely that NULL would be in a real-world passphrase entered by a
> sane user.
>
>
> On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn <mike at plan99.net> wrote:
>
>> [+cc aaron]
>>
>> We recently added an implementation of BIP 38 (password protected private
>> keys) to bitcoinj. It came to my attention that the third test vector may
>> be broken. It gives a hex version of what the NFC normalised version of the
>> input string should be, but this does not match the results of the Java
>> unicode normaliser, and in fact I can't even get Python to print the names
>> of the characters past the embedded null. I'm curious where this normalised
>> version came from.
>>
>> Given that "pile of poo" is not a character I think any sane user would
>> put into a passphrase, I question the value of this test vector. NFC form
>> is intended to collapse things like umlaut control characters onto their
>> prior code point, but here we're feeding the algorithm what is basically
>> garbage so I'm not totally surprised that different implementations appear
>> to disagree on the outcome.
>>
>> Proposed action: we remove this test vector as it does not represent any
>> real world usage of the spec, or if we desperately need to verify NFC
>> normalisation I suggest using a different, more realistic test string, like
>> Zürich, or something written in Thai.
>>
>>
>>
>> Test 3:
>>
>> - Passphrase ϓ␀𐐀💩 (\u03D2\u0301\u0000\U00010400\U0001F4A9; GREEK
>> UPSILON WITH HOOK <http://codepoints.net/U+03D2>;, COMBINING ACUTE
>> ACCENT <http://codepoints.net/U+0301>;, NULL
>> <http://codepoints.net/U+0000>;, DESERET CAPITAL LETTER LONG I
>> <http://codepoints.net/U+10400>;, PILE OF POO
>> <http://codepoints.net/U+1F4A9>;)
>> - Encrypted key:
>> 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn
>> - Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF
>> - Unencrypted private key (WIF):
>> 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4
>> - *Note:* The non-standard UTF-8 characters in this passphrase should
>> be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before
>> further processing
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Want fast and easy access to all the code in your enterprise? Index and
>> search up to 200,000 lines of code with a free copy of Black Duck
>> Code Sight - the same software that powers the world's largest code
>> search on Ohloh, the Black Duck Open Hub! Try it now.
>> http://p.sf.net/sfu/bds
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20140715/298353e5/attachment.html>;
Brooks Boyd [ARCHIVE] /
npub12h…vtmxa
2023-06-07 15:23:50
in reply to nevent1q…ajfu
Author Public Key
npub12hzwfhx9ptjf2f0va8qllc0h4tcvz5aaefvlvn9uq8z2ssrq027qtvtmxaPublished at
2023-06-07 15:23:50Event JSON
{
"id": "40ef6f39180802ab996cf8dbc4c1cf60c0b02932bb8bdb5258a5ff29da766797",
"pubkey": "55c4e4dcc50ae49525ece9c1ffe1f7aaf0c153bdca59f64cbc01c4a840607abc",
"created_at": 1686151430,
"kind": 1,
"tags": [
[
"e",
"60726055f5690f49a0698605fd9e440f1a7b0fbf2e702b7f0382a84f86474ca9",
"",
"root"
],
[
"e",
"83cfe8736294a98fc062de388aa55e16deb1fa14043e198a323d72146d79b9d8",
"",
"reply"
],
[
"p",
"5dfbe446644f4260520516c70aa838e0d4a0c8577a552092ac76ea0d71e4f5cb"
]
],
"content": "📅 Original date posted:2014-07-15\n📝 Original message:I was part of adding in that test vector, and I think it's a good test\nvector since it is an extreme edge-case of the current definition: If the\nBIP38 proposal allows any password that can be in UTF-8, NFC normalized\nform, those characters cover the various edge cases (combining characters,\nnull character, astral range) that if your implementation doesn't handle,\nthen it can't really be said to be \"BIP38-compatible/compliant\", right?\n\nThe \"passphrase\" in the test vector is NOT in NFC form; that's the point.\nWhatever implementation gets designed has to assume the input is not\nalready NFC-normalized and needs to handle/sanitize that input before\nfurther processing. To test your implementation for compliance, you should\nnot be inputting the NFC-normalized bytestring as the password input, you\nshould be entering the original passphrase as the test. My original pull\nrequest for this change (https://github.com/bitcoin/bips/pull/29) shows a\nPython and a NodeJS way to input that test vector password as intended.\n\nSome input devices may already handle the input as NFC, which is great, but\nper the BIP38 proposal, that shouldn't be assumed, so various\nimplementations are cross-compatible. If one implementation assumes the\ninput is already NFC, they may encode/decode the password incorrectly, and\nlock a user out of their wallet. Android allows different user keyboards to\nbe used, so I'm guessing there's one somewhere that allows manual entry of\nunicode codepoints that could be used to enter a null character, and with\nthe next version of iOS, Apple devices will also get custom keyboard\noptions, too, so even if the default Apple keyboard does NFC-form properly,\nother developers' keyboards may not. So while it is an extreme edge case,\nthat is not very likely to be used as a \"real password\" by any user, that's\nwhat test vectors are for: to test for the edge case that you might not\nhave expected and handled in your implementation.\n\nBrooks\n\n\nOn Tue, Jul 15, 2014 at 8:07 AM, Eric Winer \u003cenwiner at gmail.com\u003e wrote:\n\n\u003e I don't know for sure if the test vector is correct NFC form. But for\n\u003e what it's worth, the Pile of Poo character is pretty easily accessible on\n\u003e the iPhone and Android keyboards, and in this string it's already in NFC\n\u003e form (f09f92a9 in the test result). I've certainly seen it in usernames\n\u003e around the internet, and wouldn't be surprised to see it in passphrases\n\u003e entered on smartphones, especially if the author of a BIP38-compatible app\n\u003e includes a (possibly ill-advised) suggestion to have your passphrase\n\u003e \"include special characters\".\n\u003e\n\u003e I haven't seen the NULL character on any smartphone keyboards, though - I\n\u003e assume the iOS and Android developers had the foresight to know how much\n\u003e havoc that would wreak on systems assuming null-terminated strings. It\n\u003e seems unlikely that NULL would be in a real-world passphrase entered by a\n\u003e sane user.\n\u003e\n\u003e\n\u003e On Tue, Jul 15, 2014 at 8:03 AM, Mike Hearn \u003cmike at plan99.net\u003e wrote:\n\u003e\n\u003e\u003e [+cc aaron]\n\u003e\u003e\n\u003e\u003e We recently added an implementation of BIP 38 (password protected private\n\u003e\u003e keys) to bitcoinj. It came to my attention that the third test vector may\n\u003e\u003e be broken. It gives a hex version of what the NFC normalised version of the\n\u003e\u003e input string should be, but this does not match the results of the Java\n\u003e\u003e unicode normaliser, and in fact I can't even get Python to print the names\n\u003e\u003e of the characters past the embedded null. I'm curious where this normalised\n\u003e\u003e version came from.\n\u003e\u003e\n\u003e\u003e Given that \"pile of poo\" is not a character I think any sane user would\n\u003e\u003e put into a passphrase, I question the value of this test vector. NFC form\n\u003e\u003e is intended to collapse things like umlaut control characters onto their\n\u003e\u003e prior code point, but here we're feeding the algorithm what is basically\n\u003e\u003e garbage so I'm not totally surprised that different implementations appear\n\u003e\u003e to disagree on the outcome.\n\u003e\u003e\n\u003e\u003e Proposed action: we remove this test vector as it does not represent any\n\u003e\u003e real world usage of the spec, or if we desperately need to verify NFC\n\u003e\u003e normalisation I suggest using a different, more realistic test string, like\n\u003e\u003e Zürich, or something written in Thai.\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e Test 3:\n\u003e\u003e\n\u003e\u003e - Passphrase ϓ␀𐐀💩 (\\u03D2\\u0301\\u0000\\U00010400\\U0001F4A9; GREEK\n\u003e\u003e UPSILON WITH HOOK \u003chttp://codepoints.net/U+03D2\u003e, COMBINING ACUTE\n\u003e\u003e ACCENT \u003chttp://codepoints.net/U+0301\u003e, NULL\n\u003e\u003e \u003chttp://codepoints.net/U+0000\u003e, DESERET CAPITAL LETTER LONG I\n\u003e\u003e \u003chttp://codepoints.net/U+10400\u003e, PILE OF POO\n\u003e\u003e \u003chttp://codepoints.net/U+1F4A9\u003e)\n\u003e\u003e - Encrypted key:\n\u003e\u003e 6PRW5o9FLp4gJDDVqJQKJFTpMvdsSGJxMYHtHaQBF3ooa8mwD69bapcDQn\n\u003e\u003e - Bitcoin Address: 16ktGzmfrurhbhi6JGqsMWf7TyqK9HNAeF\n\u003e\u003e - Unencrypted private key (WIF):\n\u003e\u003e 5Jajm8eQ22H3pGWLEVCXyvND8dQZhiQhoLJNKjYXk9roUFTMSZ4\n\u003e\u003e - *Note:* The non-standard UTF-8 characters in this passphrase should\n\u003e\u003e be NFC normalized to result in a passphrase of0xcf9300f0909080f09f92a9 before\n\u003e\u003e further processing\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e ------------------------------------------------------------------------------\n\u003e\u003e Want fast and easy access to all the code in your enterprise? Index and\n\u003e\u003e search up to 200,000 lines of code with a free copy of Black Duck\n\u003e\u003e Code Sight - the same software that powers the world's largest code\n\u003e\u003e search on Ohloh, the Black Duck Open Hub! Try it now.\n\u003e\u003e http://p.sf.net/sfu/bds\n\u003e\u003e _______________________________________________\n\u003e\u003e Bitcoin-development mailing list\n\u003e\u003e Bitcoin-development at lists.sourceforge.net\n\u003e\u003e https://lists.sourceforge.net/lists/listinfo/bitcoin-development\n\u003e\u003e\n\u003e\u003e\n\u003e\n\u003e\n\u003e ------------------------------------------------------------------------------\n\u003e Want fast and easy access to all the code in your enterprise? Index and\n\u003e search up to 200,000 lines of code with a free copy of Black Duck\n\u003e Code Sight - the same software that powers the world's largest code\n\u003e search on Ohloh, the Black Duck Open Hub! Try it now.\n\u003e http://p.sf.net/sfu/bds\n\u003e _______________________________________________\n\u003e Bitcoin-development mailing list\n\u003e Bitcoin-development at lists.sourceforge.net\n\u003e https://lists.sourceforge.net/lists/listinfo/bitcoin-development\n\u003e\n\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20140715/298353e5/attachment.html\u003e",
"sig": "6e583a1450c4e323014c4b7041141566d7bb178aaac75fa90dcf51cd62cf1200f42edaf659111598285a9f010e743c82958dbcf6ffe36114aac52536066074a2"
}