📅 Original date posted:2015-10-19
📝 Original message:I should also mention that this is definitely not an attack coming from
connected nodes. My node experiencing the issue is only connected to 3
other nodes, all of which I control (via connect=).
--Adam
On Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin <admin at multipool.us>
wrote:
> My nodes are continuously running getblocktemplate and getinfo, and I also
> suspected the issue is in either gbt or the rpc server.
>
> The instance only takes a few hours to get up to that memory usage.
> On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <
> bitcoin-dev at lists.linuxfoundation.org> wrote:
>
>> On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan <laanwj at gmail.com>
>> wrote:
>>
>> This is *most likely* the mempool, but is just not reported correctly.
>>
>>
>> I did some testing with PR #6410's better mempool reporting. The improved
>> reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
>> about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
>> The excess memory usage that I'm seeing is on the order of 100x higher than
>> the mempool "bytes": value. As such, I think it's unlikely that this is the
>> mempool, or at least not normal/correct mempool behavior.
>>
>> Another user (admin at multipool.us) reported 35 GB of RSS usage. I'm
>> guessing his bitcoind has been running longer than any of mine. His server
>> definitely has more RAM. I don't know which email list he is subscribed to
>> (probably XT), so I'm sharing it with both lists to make sure you're all
>> aware of how big an issue this can be.
>>
>> In the meantime you can mitigate the mempool growth by setting
>> `-mintxfee`, see
>>
>> https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
>>
>>
>> I have mintxfee and minrelaytxfee set to about 0.00003, which is high
>> enough to exclude essentially all of the of the 14700-14800 byte flood
>> transactions. My nodes' mempools only contain about one or two blocks'
>> worth of transactions. So I don't think this is correct either.
>>
>>
>>
>> Some additional notes on this issue:
>>
>> 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
>> Core bitcoind process (commit d78a880) overnight with no mining connected
>> to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
>> 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
>> ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
>> 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
>> and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
>> and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
>> getmempoolinfo is still showing "usage" around 25MB or less.
>>
>> I'll do some more testing with this and see if I can make it repeatable,
>> and record the results more carefully. Expect a follow-up from me in a day
>> or two.
>>
>> 2. valgrind did not show anything super promising. It did report this:
>>
>> ==6880== LEAK SUMMARY:
>> ==6880== definitely lost: 0 bytes in 0 blocks
>> ==6880== indirectly lost: 0 bytes in 0 blocks
>> ==6880== possibly lost: 288 bytes in 1 blocks
>> ==6880== still reachable: 10,552 bytes in 39 blocks
>> ==6880== suppressed: 0 bytes in 0 blocks
>> (Bitcoin Core commit d78a880)
>>
>> and this:
>> ==6778== LEAK SUMMARY:
>> ==6778== definitely lost: 0 bytes in 0 blocks
>> ==6778== indirectly lost: 0 bytes in 0 blocks
>> ==6778== possibly lost: 320 bytes in 1 blocks
>> ==6778== still reachable: 10,080 bytes in 32 blocks
>> ==6778== suppressed: 0 bytes in 0 blocks
>> (Bitcoin XT commit fe446d)
>>
>> I haven't found anything in there yet that I think would produce the
>> multi-GB memory usage after running for a few days, but I could be missing
>> it. Email me if you want the full log.
>>
>> I did not try running getblocktemplate while valgrind was running. I'll
>> have to try that. I also have not let valgrind run for more than an hour.
>>
>>
>>
>> P.S.: Sorry for all the cross-post confusion and consequent flamewar
>> fallout. While it's probably too late for this thread, I'll make sure to
>> post in a manner that keeps the threads clearly separate in the future
>> (e.g. different subject lines).
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151019/aaf129a5/attachment.html>;
Multipool Admin [ARCHIVE] /
npub1wu…6fwsk
2023-06-07 17:43:19
in reply to nevent1q…ky44
Author Public Key
npub1wu8nl4rek0s38jw0p5tx4h8a0vu75f33v5ucc0ndj2fza5yav05q36fwskPublished at
2023-06-07 17:43:19Event JSON
{
"id": "757cc6447ac986e61837e53e3362d82b2fc48152b81a8347aa69f45129b76657",
"pubkey": "770f3fd479b3e113c9cf0d166adcfd7b39ea263165398c3e6d92922ed09d63e8",
"created_at": 1686159799,
"kind": 1,
"tags": [
[
"e",
"257c60cbd2c94bcbc1af242f96780d9665ec0aa19ec5454bed0371ff096c7c39",
"",
"root"
],
[
"e",
"a237a99b62b75ff88d5bbc90b264549604bba744806a93cb6d82005aa9b78bc8",
"",
"reply"
],
[
"p",
"770f3fd479b3e113c9cf0d166adcfd7b39ea263165398c3e6d92922ed09d63e8"
]
],
"content": "📅 Original date posted:2015-10-19\n📝 Original message:I should also mention that this is definitely not an attack coming from\nconnected nodes. My node experiencing the issue is only connected to 3\nother nodes, all of which I control (via connect=).\n\n--Adam\n\nOn Mon, Oct 19, 2015 at 12:17 PM, Multipool Admin \u003cadmin at multipool.us\u003e\nwrote:\n\n\u003e My nodes are continuously running getblocktemplate and getinfo, and I also\n\u003e suspected the issue is in either gbt or the rpc server.\n\u003e\n\u003e The instance only takes a few hours to get up to that memory usage.\n\u003e On Oct 18, 2015 8:59 AM, \"Jonathan Toomim via bitcoin-dev\" \u003c\n\u003e bitcoin-dev at lists.linuxfoundation.org\u003e wrote:\n\u003e\n\u003e\u003e On Oct 14, 2015, at 2:39 AM, Wladimir J. van der Laan \u003claanwj at gmail.com\u003e\n\u003e\u003e wrote:\n\u003e\u003e\n\u003e\u003e This is *most likely* the mempool, but is just not reported correctly.\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e I did some testing with PR #6410's better mempool reporting. The improved\n\u003e\u003e reporting suggests that actual in-memory usage (\"usage\":) by CTxMemPool is\n\u003e\u003e about 2.5x to 3x higher than the serialized transaction sizes (\"bytes\":).\n\u003e\u003e The excess memory usage that I'm seeing is on the order of 100x higher than\n\u003e\u003e the mempool \"bytes\": value. As such, I think it's unlikely that this is the\n\u003e\u003e mempool, or at least not normal/correct mempool behavior.\n\u003e\u003e\n\u003e\u003e Another user (admin at multipool.us) reported 35 GB of RSS usage. I'm\n\u003e\u003e guessing his bitcoind has been running longer than any of mine. His server\n\u003e\u003e definitely has more RAM. I don't know which email list he is subscribed to\n\u003e\u003e (probably XT), so I'm sharing it with both lists to make sure you're all\n\u003e\u003e aware of how big an issue this can be.\n\u003e\u003e\n\u003e\u003e In the meantime you can mitigate the mempool growth by setting\n\u003e\u003e `-mintxfee`, see\n\u003e\u003e\n\u003e\u003e https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e I have mintxfee and minrelaytxfee set to about 0.00003, which is high\n\u003e\u003e enough to exclude essentially all of the of the 14700-14800 byte flood\n\u003e\u003e transactions. My nodes' mempools only contain about one or two blocks'\n\u003e\u003e worth of transactions. So I don't think this is correct either.\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e Some additional notes on this issue:\n\u003e\u003e\n\u003e\u003e 1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a\n\u003e\u003e Core bitcoind process (commit d78a880) overnight with no mining connected\n\u003e\u003e to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around\n\u003e\u003e 400 MB of RSS and the mempool was at around \"bytes\":10MB, \"usage\": 25MB. I\n\u003e\u003e ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around\n\u003e\u003e 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,\n\u003e\u003e and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,\n\u003e\u003e and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]\n\u003e\u003e getmempoolinfo is still showing \"usage\" around 25MB or less.\n\u003e\u003e\n\u003e\u003e I'll do some more testing with this and see if I can make it repeatable,\n\u003e\u003e and record the results more carefully. Expect a follow-up from me in a day\n\u003e\u003e or two.\n\u003e\u003e\n\u003e\u003e 2. valgrind did not show anything super promising. It did report this:\n\u003e\u003e\n\u003e\u003e ==6880== LEAK SUMMARY:\n\u003e\u003e ==6880== definitely lost: 0 bytes in 0 blocks\n\u003e\u003e ==6880== indirectly lost: 0 bytes in 0 blocks\n\u003e\u003e ==6880== possibly lost: 288 bytes in 1 blocks\n\u003e\u003e ==6880== still reachable: 10,552 bytes in 39 blocks\n\u003e\u003e ==6880== suppressed: 0 bytes in 0 blocks\n\u003e\u003e (Bitcoin Core commit d78a880)\n\u003e\u003e\n\u003e\u003e and this:\n\u003e\u003e ==6778== LEAK SUMMARY:\n\u003e\u003e ==6778== definitely lost: 0 bytes in 0 blocks\n\u003e\u003e ==6778== indirectly lost: 0 bytes in 0 blocks\n\u003e\u003e ==6778== possibly lost: 320 bytes in 1 blocks\n\u003e\u003e ==6778== still reachable: 10,080 bytes in 32 blocks\n\u003e\u003e ==6778== suppressed: 0 bytes in 0 blocks\n\u003e\u003e (Bitcoin XT commit fe446d)\n\u003e\u003e\n\u003e\u003e I haven't found anything in there yet that I think would produce the\n\u003e\u003e multi-GB memory usage after running for a few days, but I could be missing\n\u003e\u003e it. Email me if you want the full log.\n\u003e\u003e\n\u003e\u003e I did not try running getblocktemplate while valgrind was running. I'll\n\u003e\u003e have to try that. I also have not let valgrind run for more than an hour.\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e\n\u003e\u003e P.S.: Sorry for all the cross-post confusion and consequent flamewar\n\u003e\u003e fallout. While it's probably too late for this thread, I'll make sure to\n\u003e\u003e post in a manner that keeps the threads clearly separate in the future\n\u003e\u003e (e.g. different subject lines).\n\u003e\u003e\n\u003e\u003e _______________________________________________\n\u003e\u003e bitcoin-dev mailing list\n\u003e\u003e bitcoin-dev at lists.linuxfoundation.org\n\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\n\u003e\u003e\n\u003e\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20151019/aaf129a5/attachment.html\u003e",
"sig": "0cfde5034102230edd6517a25965404f0f257e66755abcdfa9fcbfd8548667219292891b94d3bd8678c34167054324b1835f83907bce240a08757deb70b5df8a"
}