đź“… Original date posted:2015-08-23
đź“ť Original message:I see the huge amount of sweat and love that went into core and it actually hurts to see that most is expended in friction and lack of a vision for the software architecture.
To be concrete, this was my plan if dealing with the Core code base:
1) I'd consider the separation of networking and storage as suggested for a future extended libconsensus low priority, as their design should be (are) dominated by the need of the consensus logic only.
2) create an API to the consensus+networking+storage service that is not at the C++ language level but some scaleable cross-platform remoting, like eg. ZeroMQ.
This API should be minimal and simple, assuming that one fully trusts the node answering it. This API would unlock user land development by distinct teams with diverse technologies.
3) move the wallet, QT and RPC and other backward compatibility stuff (if e.g. there is some mining support) in-top of the new API and into distinct source code repositories.
Tamas Blummer
> On Aug 23, 2015, at 03:23, Eric Lombrozo <elombrozo at gmail.com> wrote:
>
> I've been pushing for greater modularization since I first got into bitcoin. I got quickly frustrated when I was only able to get through very few things (i.e. moving core structure serialization classes to a separate unit not called main). Working on Bitcoin has an added layer of frustration that goes beyond most open source projects: even though we're clearly in userland working at the application layer, a good layered protocol design is still lacking. We have no standards process separate from what basically amount to updates to one specific reference implementation. And we all need to agree on any major change, since a blockchain that is easily forked in contentious ways pretty much defeats its own purpose.
>
> I went off to develop my own stack, where I could more easily avoid politics and focus on engineering. But I now understand the politics are inevitable. Bitcoin is inherently a cooperative project. Several people have poured themselves passionately into the reference codebase, most of whom did it (at least initially) purely as unpaid volunteers. There's a lot of love that's gone into this. But it's become pretty clear that the modularization is no longer merely a matter of good engineering - it is essential to resolving serious political challenges.
>
> Perhaps the most frustrating thing of all is watching people pushing for relatively superficial yet highly controversial changes while we still lack the proper infrastructure to handle these kinds of divergences of opinion without either stagnating or becoming polarized.
>
> I could continue working to reimplement an entire stack from scratch, as several others have also done - but besides the serious effort duplication this entails, it doesn't really seem like it will ultimately be a convergent process. It's too easy to let ego and habit dictate one's preferences rather than rational engineering considerations.
>
> I know that some might feel I'm just preaching to the choir, but we should probably take a step back from implementation hackery and try to specify some core protocol layers, focusing on interfaces. Specifically, we need a consensus layer that doesn't try to specify networking, storage, wallets, UI, etc. Let different people improve upon these things independently in their own implementations. What matters is that we all converge on a common history and state. At the same time, let's open up more competition on all these other things that are separate from the consensus layer.
>
> If only we were to dedicate a fraction of the effort we've put into this whole block size circus into what's actually important...and I blame myself as well...
>
>
> On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <bitcoin-dev at lists.linuxfoundation.org <mailto:bitcoin-dev at lists.linuxfoundation.org>> wrote:
>> On Aug 21, 2015, at 21:46, Jorge TimĂłn <jtimon at jtimon.cc <mailto:jtimon at jtimon.cc>> wrote:
>>
>> On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer <tamas at bitsofproof.com <mailto:tamas at bitsofproof.com>> wrote:
>>> Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
>>> but also open the chance of doing the work better, in the sense of software engineering.
>>
>> But you don't want something better, you want something functionally identical.
>> You may want to watch sipa's explanation on why "the implementation is
>> the specification" and the reasons to separate libconsensus:
>> https://youtu.be/l3O4nh79CUU?t=764 <https://youtu.be/l3O4nh79CUU?t=764>;
>
> I do want something better, but not for the focus you have.
>
> Not because what you produce was not high quality, but because quality is achieved at a very
> high cost and is hard to uphold over generations of developer. You focus on a single use case
> while there are many out there for distributed ledgers.
>
> I think in an infrastructure for enterprise applications, building consensus on the ledger is a
> cornerstone there, but is only a piece of the solution. I built several commercially successful
> deployments where I delegated the consensus building to a border router, a Bitcoin Core,
> then interfaced that trusted peer with my implementation that accepted Core’s decisions
> in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices”
> therefore an example of centralization people here try to avoid.
>
> Enterprises have sufficient resources. Solving the business problem is valuable to them even at
> magnitudes higher cost than a hobbyist would bear.
>
> For mainstream adoption you need to get enterprises on board too, and that is what I care of.
> Enterprises want code that is not only high quality, but is easy to maintain with a development
> team with high attrition. One has to take whatever help is offered for that, and one is modern
> languages and runtimes.
>
> Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially
> successful deployments, because of the use of a border router, but it helped development,
> enabling easier debug and precise error feedback esp. end even after Core had a reject message.
>
> I integrated libconsensus only for the hope that is significantly fastens application side tx verification,
> which it has turned out it does not, until secp265k1 is integrated.
>
> I would likely use an other extended libconsensus too, but do not think there was a dependency on
> that for enterprise development.
>
> It would help there more to have a slim protocol server, no wallet, no rpc, no qt but a high
> performance remoting API.
>
>> Since you already depend on libconsensus for VerifyScript, wouldn't it
>> be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
>> You would still have complete control over storage, concurrency,
>> networking, policy...
>> My plan is for the C API to interface with the external storage by
>> passing a function pointer to it.
>
>
> Storage and validation is non-trivially interconnected, but I now the separation can be done,
> since I did it.
>
> Excuse me, but function pointers is a pattern I used in the 80’s. I know that they are behind
> the curtain of modern abstractions with similar use, I still prefer not to see them again.
>
> Tamas Blummer
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev at lists.linuxfoundation.org <mailto:bitcoin-dev at lists.linuxfoundation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150823/9129de9d/attachment-0001.html>;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150823/9129de9d/attachment-0001.sig>;
Tamas Blummer [ARCHIVE] /
npub1cc…ex8m8
2023-06-07 15:48:49
in reply to nevent1q…wg20
Author Public Key
npub1ccegg9n9lnx6huppxg43m95488yur7pfemkn3pz0agjws5ffvtts0ex8m8Published at
2023-06-07 15:48:49Event JSON
{
"id": "229d43f30ae5de6cecc05f54ab8b4463f34507f04b57bc18d6fe99945a286378",
"pubkey": "c632841665fccdabf021322b1d969539c9c1f829ceed38844fea24e8512962d7",
"created_at": 1686152929,
"kind": 1,
"tags": [
[
"e",
"2cb77a81d8c40c165dca19129371f9698d50bec611bf964a3c59cbb47a7d1941",
"",
"root"
],
[
"e",
"8e1b45e5ced0ca366d3cd0d82451558093e8e9e5e51fbedf19bb55cb74b0b359",
"",
"reply"
],
[
"p",
"e899768d254f3537af7e26455968583632d0ab0bd4c780445eacfa087ac80d8f"
]
],
"content": "📅 Original date posted:2015-08-23\n📝 Original message:I see the huge amount of sweat and love that went into core and it actually hurts to see that most is expended in friction and lack of a vision for the software architecture.\n\nTo be concrete, this was my plan if dealing with the Core code base:\n\n1) I'd consider the separation of networking and storage as suggested for a future extended libconsensus low priority, as their design should be (are) dominated by the need of the consensus logic only.\n\n2) create an API to the consensus+networking+storage service that is not at the C++ language level but some scaleable cross-platform remoting, like eg. ZeroMQ.\nThis API should be minimal and simple, assuming that one fully trusts the node answering it. This API would unlock user land development by distinct teams with diverse technologies.\n\n3) move the wallet, QT and RPC and other backward compatibility stuff (if e.g. there is some mining support) in-top of the new API and into distinct source code repositories.\n\n\nTamas Blummer\n\n\u003e On Aug 23, 2015, at 03:23, Eric Lombrozo \u003celombrozo at gmail.com\u003e wrote:\n\u003e \n\u003e I've been pushing for greater modularization since I first got into bitcoin. I got quickly frustrated when I was only able to get through very few things (i.e. moving core structure serialization classes to a separate unit not called main). Working on Bitcoin has an added layer of frustration that goes beyond most open source projects: even though we're clearly in userland working at the application layer, a good layered protocol design is still lacking. We have no standards process separate from what basically amount to updates to one specific reference implementation. And we all need to agree on any major change, since a blockchain that is easily forked in contentious ways pretty much defeats its own purpose.\n\u003e \n\u003e I went off to develop my own stack, where I could more easily avoid politics and focus on engineering. But I now understand the politics are inevitable. Bitcoin is inherently a cooperative project. Several people have poured themselves passionately into the reference codebase, most of whom did it (at least initially) purely as unpaid volunteers. There's a lot of love that's gone into this. But it's become pretty clear that the modularization is no longer merely a matter of good engineering - it is essential to resolving serious political challenges.\n\u003e \n\u003e Perhaps the most frustrating thing of all is watching people pushing for relatively superficial yet highly controversial changes while we still lack the proper infrastructure to handle these kinds of divergences of opinion without either stagnating or becoming polarized.\n\u003e \n\u003e I could continue working to reimplement an entire stack from scratch, as several others have also done - but besides the serious effort duplication this entails, it doesn't really seem like it will ultimately be a convergent process. It's too easy to let ego and habit dictate one's preferences rather than rational engineering considerations.\n\u003e \n\u003e I know that some might feel I'm just preaching to the choir, but we should probably take a step back from implementation hackery and try to specify some core protocol layers, focusing on interfaces. Specifically, we need a consensus layer that doesn't try to specify networking, storage, wallets, UI, etc. Let different people improve upon these things independently in their own implementations. What matters is that we all converge on a common history and state. At the same time, let's open up more competition on all these other things that are separate from the consensus layer.\n\u003e \n\u003e If only we were to dedicate a fraction of the effort we've put into this whole block size circus into what's actually important...and I blame myself as well...\n\u003e \n\u003e \n\u003e On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev \u003cbitcoin-dev at lists.linuxfoundation.org \u003cmailto:bitcoin-dev at lists.linuxfoundation.org\u003e\u003e wrote:\n\u003e\u003e On Aug 21, 2015, at 21:46, Jorge Timón \u003cjtimon at jtimon.cc \u003cmailto:jtimon at jtimon.cc\u003e\u003e wrote:\n\u003e\u003e \n\u003e\u003e On Thu, Aug 20, 2015 at 10:35 AM, Tamas Blummer \u003ctamas at bitsofproof.com \u003cmailto:tamas at bitsofproof.com\u003e\u003e wrote:\n\u003e\u003e\u003e Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,\n\u003e\u003e\u003e but also open the chance of doing the work better, in the sense of software engineering.\n\u003e\u003e \n\u003e\u003e But you don't want something better, you want something functionally identical.\n\u003e\u003e You may want to watch sipa's explanation on why \"the implementation is\n\u003e\u003e the specification\" and the reasons to separate libconsensus:\n\u003e\u003e https://youtu.be/l3O4nh79CUU?t=764 \u003chttps://youtu.be/l3O4nh79CUU?t=764\u003e\n\u003e \n\u003e I do want something better, but not for the focus you have.\n\u003e \n\u003e Not because what you produce was not high quality, but because quality is achieved at a very\n\u003e high cost and is hard to uphold over generations of developer. You focus on a single use case\n\u003e while there are many out there for distributed ledgers.\n\u003e \n\u003e I think in an infrastructure for enterprise applications, building consensus on the ledger is a\n\u003e cornerstone there, but is only a piece of the solution. I built several commercially successful\n\u003e deployments where I delegated the consensus building to a border router, a Bitcoin Core,\n\u003e then interfaced that trusted peer with my implementation that accepted Core’s decisions\n\u003e in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices”\n\u003e therefore an example of centralization people here try to avoid.\n\u003e \n\u003e Enterprises have sufficient resources. Solving the business problem is valuable to them even at\n\u003e magnitudes higher cost than a hobbyist would bear.\n\u003e \n\u003e For mainstream adoption you need to get enterprises on board too, and that is what I care of.\n\u003e Enterprises want code that is not only high quality, but is easy to maintain with a development\n\u003e team with high attrition. One has to take whatever help is offered for that, and one is modern\n\u003e languages and runtimes.\n\u003e \n\u003e Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially\n\u003e successful deployments, because of the use of a border router, but it helped development,\n\u003e enabling easier debug and precise error feedback esp. end even after Core had a reject message.\n\u003e \n\u003e I integrated libconsensus only for the hope that is significantly fastens application side tx verification,\n\u003e which it has turned out it does not, until secp265k1 is integrated.\n\u003e \n\u003e I would likely use an other extended libconsensus too, but do not think there was a dependency on\n\u003e that for enterprise development.\n\u003e \n\u003e It would help there more to have a slim protocol server, no wallet, no rpc, no qt but a high\n\u003e performance remoting API.\n\u003e \n\u003e\u003e Since you already depend on libconsensus for VerifyScript, wouldn't it\n\u003e\u003e be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?\n\u003e\u003e You would still have complete control over storage, concurrency,\n\u003e\u003e networking, policy...\n\u003e\u003e My plan is for the C API to interface with the external storage by\n\u003e\u003e passing a function pointer to it.\n\u003e \n\u003e \n\u003e Storage and validation is non-trivially interconnected, but I now the separation can be done,\n\u003e since I did it.\n\u003e \n\u003e Excuse me, but function pointers is a pattern I used in the 80’s. I know that they are behind\n\u003e the curtain of modern abstractions with similar use, I still prefer not to see them again.\n\u003e \n\u003e Tamas Blummer\n\u003e \n\u003e _______________________________________________\n\u003e bitcoin-dev mailing list\n\u003e bitcoin-dev at lists.linuxfoundation.org \u003cmailto:bitcoin-dev at lists.linuxfoundation.org\u003e\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev \u003chttps://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev\u003e\n\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150823/9129de9d/attachment-0001.html\u003e\n-------------- next part --------------\nA non-text attachment was scrubbed...\nName: signature.asc\nType: application/pgp-signature\nSize: 496 bytes\nDesc: Message signed with OpenPGP using GPGMail\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20150823/9129de9d/attachment-0001.sig\u003e",
"sig": "45858c85beffd6b4a4821e24358c8834d54690482a5ed3fd1e934aeb86e0a92dd152bb12f52db531623596adcb569d1df73b209de201d17583253925ef6648f5"
}