📅 Original date posted:2014-05-07
📝 Original message:I think there a few different possible ways to go here.
One is to try and simplify the setup of all the components so it all gets
installed together. That might be feasible in some quite restricted setups
but the installation instructions for Graphite look kind of terrifying.
Another is to export stats over regular TCP and make them public so
literally anyone can listen to the stats feed for any node. Then people who
dig stats and graphs could work on stats aggregators that give global
network visibility independently, effectively crawling the p2p network for
data. It'd have the advantage of having zero setup for the node operators
and not require much in the way of resources.
For what it's worth, although the environment is a bit different inside
Google the latter approach is used. Monitoring servers locate servers of
interest via a discovery service, connect to them and start streaming stats
data into a database service that can then be queried later to get graphs.
The stats are also run through various rules to obtain alerts about
problematic conditions. For example, if a subset of the network splits it
might be hard to notice that if the node operators aren't paying attention
and Matt's fork alert/emailing code isn't set up. But if there was a site
crawling nodes and aggregating chain heights by version, that could trigger
an alert to people who *are* paying attention.
I know from practical experience that monitoring and analysis tends to
appeal more to certain types of people than others. So I quite like the
"let anyone monitor" approach. However, it may not be appropriate in a P2P
network, I did not think about it much.
Obviously I'm assuming none of the stats expose privacy sensitive data.
On Wed, May 7, 2014 at 10:18 PM, Wladimir <laanwj at gmail.com> wrote:
> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp at gmail.com>
> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > I agree that it would be awesome to offer these types of stats with the
> installer; unfortunately the route I've taken has dependencies on several
> other other pieces of software to do all the heavy lifting of stats
> aggregation and chart rendering. I'm assuming that you would not want to
> build any of that processing into Bitcoin Core itself; would you be opposed
> to packaging other software along with the installer?
>
> Depends on just how much stuff it is. The idea is primarily to have an
> installer for running a (wallet-less) node as an OS background
> service.
>
> Having some statistics available would be worth some extra download
> size, otherwise it would be pretty much invisible.
>
> We'd already decided that we would need something like Python for the
> stats service. Implementing things like web services in C++ is just
> not realistic given the time constraints and the great already-written
> code that is out there. As an optional tool it should be external, not
> part of bitcoind itself.
>
> I suppose the chart rendering happens client-side? In that case the
> web service just has to collect and provide the data, and serve static
> html/js files.
>
> Wladimir
>
>
> ------------------------------------------------------------------------------
> Is your legacy SCM system holding you back? Join Perforce May 7 to find
> out:
> • 3 signs your SCM is hindering your productivity
> • Requirements for releasing software faster
> • Expert tips and advice for migrating your SCM now
> http://p.sf.net/sfu/perforce
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20140507/2826ee4a/attachment.html>;
Mike Hearn [ARCHIVE] /
npub17t…4qgyd
2023-06-07 15:21:05
in reply to nevent1q…vfut
Author Public Key
npub17ty4mumkv43w8wtt0xsz2jypck0gvw0j8xrcg6tpea25z2nh7meqf4qgydPublished at
2023-06-07 15:21:05Event JSON
{
"id": "7ab43e43392c92aba867c48dd3ab9799158f3c05b3100e27397e7f46f3ea1807",
"pubkey": "f2c95df3766562e3b96b79a0254881c59e8639f23987846961cf55412a77f6f2",
"created_at": 1686151265,
"kind": 1,
"tags": [
[
"e",
"eee9f45331490750cb4b2457486fbbd67822220193953d499fd36e4811e321a2",
"",
"root"
],
[
"e",
"93bfd35df9cc961a8b0f710d7d499df6066b0c5a6a711ccea936d74d79cc9ac1",
"",
"reply"
],
[
"p",
"30217b018a47b99ed4c20399b44b02f70ec4f58ed77a2814a563fa28322ef722"
]
],
"content": "📅 Original date posted:2014-05-07\n📝 Original message:I think there a few different possible ways to go here.\n\nOne is to try and simplify the setup of all the components so it all gets\ninstalled together. That might be feasible in some quite restricted setups\nbut the installation instructions for Graphite look kind of terrifying.\n\nAnother is to export stats over regular TCP and make them public so\nliterally anyone can listen to the stats feed for any node. Then people who\ndig stats and graphs could work on stats aggregators that give global\nnetwork visibility independently, effectively crawling the p2p network for\ndata. It'd have the advantage of having zero setup for the node operators\nand not require much in the way of resources.\n\nFor what it's worth, although the environment is a bit different inside\nGoogle the latter approach is used. Monitoring servers locate servers of\ninterest via a discovery service, connect to them and start streaming stats\ndata into a database service that can then be queried later to get graphs.\n\nThe stats are also run through various rules to obtain alerts about\nproblematic conditions. For example, if a subset of the network splits it\nmight be hard to notice that if the node operators aren't paying attention\nand Matt's fork alert/emailing code isn't set up. But if there was a site\ncrawling nodes and aggregating chain heights by version, that could trigger\nan alert to people who *are* paying attention.\n\nI know from practical experience that monitoring and analysis tends to\nappeal more to certain types of people than others. So I quite like the\n\"let anyone monitor\" approach. However, it may not be appropriate in a P2P\nnetwork, I did not think about it much.\n\nObviously I'm assuming none of the stats expose privacy sensitive data.\n\n\n\nOn Wed, May 7, 2014 at 10:18 PM, Wladimir \u003claanwj at gmail.com\u003e wrote:\n\n\u003e On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp \u003cjameson.lopp at gmail.com\u003e\n\u003e wrote:\n\u003e \u003e -----BEGIN PGP SIGNED MESSAGE-----\n\u003e \u003e Hash: SHA1\n\u003e \u003e\n\u003e \u003e I agree that it would be awesome to offer these types of stats with the\n\u003e installer; unfortunately the route I've taken has dependencies on several\n\u003e other other pieces of software to do all the heavy lifting of stats\n\u003e aggregation and chart rendering. I'm assuming that you would not want to\n\u003e build any of that processing into Bitcoin Core itself; would you be opposed\n\u003e to packaging other software along with the installer?\n\u003e\n\u003e Depends on just how much stuff it is. The idea is primarily to have an\n\u003e installer for running a (wallet-less) node as an OS background\n\u003e service.\n\u003e\n\u003e Having some statistics available would be worth some extra download\n\u003e size, otherwise it would be pretty much invisible.\n\u003e\n\u003e We'd already decided that we would need something like Python for the\n\u003e stats service. Implementing things like web services in C++ is just\n\u003e not realistic given the time constraints and the great already-written\n\u003e code that is out there. As an optional tool it should be external, not\n\u003e part of bitcoind itself.\n\u003e\n\u003e I suppose the chart rendering happens client-side? In that case the\n\u003e web service just has to collect and provide the data, and serve static\n\u003e html/js files.\n\u003e\n\u003e Wladimir\n\u003e\n\u003e\n\u003e ------------------------------------------------------------------------------\n\u003e Is your legacy SCM system holding you back? Join Perforce May 7 to find\n\u003e out:\n\u003e \u0026#149; 3 signs your SCM is hindering your productivity\n\u003e \u0026#149; Requirements for releasing software faster\n\u003e \u0026#149; Expert tips and advice for migrating your SCM now\n\u003e http://p.sf.net/sfu/perforce\n\u003e _______________________________________________\n\u003e Bitcoin-development mailing list\n\u003e Bitcoin-development at lists.sourceforge.net\n\u003e https://lists.sourceforge.net/lists/listinfo/bitcoin-development\n\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20140507/2826ee4a/attachment.html\u003e",
"sig": "ddc1e8b23032905a2a4e2167738b18eea79f748b97df1508d66860277fada58fa1acf5d8a779ce3748feabf6b9cf32bc5f5aaa1ffeed6ba2e008398a360f2e98"
}