📅 Original date posted:2018-02-07
📝 Original message:
I like Christian's proposal of adding a simple announcement cutoff
timestamp with the intention of designing something more sophisticated
given more time.
I prefer the approach of having an optional feature bit signalling that a
`set_gossip_timestamp` message must be sent immediately after `init`, as
Laolu suggested. This way it doesn't conflict with and other possible
handshake extensions.
On Feb 7, 2018 9:50 AM, "Fabrice Drouin" <fabrice.drouin at acinq.fr> wrote:
Hi,
Suppose you partition nodes into 3 generic roles:
- payers: they mostly send payments, are typically small and operated
by end users, and are offline quite a lot
- relayers: they mostly relay payments, and would be online most of
the time (if they're too unreliable other nodes will eventually close
their channels with them)
- payees: they mostly receive payments, how often they can be online
is directly link to their particular mode of operations (since you
need to be online to receive payments)
Of course most nodes would play more or less all roles. However,
mobile nodes would probably be mostly "payers", and they have specific
properties:
- if they don't relay payments they don't have to be announced. There
could be millions of mobile nodes that would have no impact on the
size of the routing table
- it does not impact the network when they're offline
- but they need an accurate routing table. This is very different from
nodes who mostly relay or accept payements
- they would be connected to a very small number of nodes
- they would typically be online for just a few hours every day, but
could be stopped/paused/restarted many times a day
Laolu wrote:
> So I think the primary distinction between y'alls proposals is that
> cdecker's proposal focuses on eventually synchronizing all the set of
> _updates_, while Fabrice's proposal cares *only* about the newly created
> channels. It only cares about new channels as the rationale is that if
once
> tries to route over a channel with a state channel update for it, then
> you'll get an error with the latest update encapsulated.
If you have one filter per day and they don't match (because your peer
has channels that you missed, or
have been closed and you were not aware of it) then you will receive
all channel announcements for
this particular day, and the associated updates
Laolu wrote:
> I think he's actually proposing just a general update horizon in which
> vertexes+edges with a lower time stamp just shouldn't be set at all. In
the
> case of an old zombie channel which was resurrected, it would eventually
be
> re-propagated as the node on either end of the channel should broadcast a
> fresh update along with the original chan ann.
Yes but it could take a long time. It may be worse on testnet since it
seems that nodes
don't change their fees very often. "Payer nodes" need a good routing
table (as opposed
to "relayers" which could work without one if they never initiate payments)
Laolu wrote:
> This seems to assume that both nodes have a strongly synchronized view of
> the network. Otherwise, they'll fall back to sending everything that went
on
> during the entire epoch regularly. It also doesn't address the zombie
churn
> issue as they may eventually send you very old channels you'll have to
deal
> with (or discard).
Yes I agree that for nodes which have connections to a lot of peers,
strongly synchronized routing tables is
harder to achieve since a small change may invalidate an entire
bucket. Real queryable filters would be much
better, but worst case scenario is we've sent an additionnal 30 Kb or
o of sync messages.
(A very naive filter would be sort + pack all short ids for example)
But we focus on nodes which are connected to a very small number of
peers, and in this particular
case it is not an unrealistic expectation.
We have built a prototype and on testnet it works fairly well. I also
found nodes which have no direct
channel betweem them but produce the same filters for 75% of the
buckets ("produce" here means
that I opened a simple gossip connection to them, got their routing
table and used it to generate filters).
Laolu wrote:
> How far back would this go? Weeks, months, years?
Since forever :)
One filter per day for all annoucements that are older than now - 1
week (modulo 144)
One filter per block for recent announcements
>
> FWIW this approach optimizes for just learning of new channels instead of
> learning of the freshest state you haven't yet seen.
I'd say it optimizes the case where you are connected to very few
peers, and are online a few times every day (?)
>
> -- Laolu
>
>
> On Mon, Feb 5, 2018 at 7:08 AM Fabrice Drouin <fabrice.drouin at acinq.fr>
> wrote:
>>
>> Hi,
>>
>> On 5 February 2018 at 14:02, Christian Decker
>> <decker.christian at gmail.com> wrote:
>> > Hi everyone
>> >
>> > The feature bit is even, meaning that it is required from the peer,
>> > since we extend the `init` message itself, and a peer that does not
>> > support this feature would be unable to parse any future extensions to
>> > the `init` message. Alternatively we could create a new
>> > `set_gossip_timestamp` message that is only sent if both endpoints
>> > support this proposal, but that could result in duplicate messages
being
>> > delivered between the `init` and the `set_gossip_timestamp` message and
>> > it'd require additional messages.
>>
>> We chose the other aproach and propose to use an optional feature
>>
>> > The reason I'm using timestamp and not the blockheight in the short
>> > channel ID is that we already use the timestamp for pruning. In the
>> > blockheight based timestamp we might ignore channels that were created,
>> > then not announced or forgotten, and then later came back and are now
>> > stable.
>>
>> Just to be clear, you propose to use the timestamp of the most recent
>> channel updates to filter
>> the associated channel announcements ?
>>
>> > I hope this rather simple proposal is sufficient to fix the short-term
>> > issues we are facing with the initial sync, while we wait for a real
>> > sync protocol. It is definitely not meant to allow perfect
>> > synchronization of the topology between peers, but then again I don't
>> > believe that is strictly necessary to make the routing successful.
>> >
>> > Please let me know what you think, and I'd love to discuss Pierre's
>> > proposal as well.
>> >
>> > Cheers,
>> > Christian
>>
>> Our idea is to group channel announcements by "buckets", create a
>> filter for each bucket, exchange and use them to filter out channel
>> announcements.
>>
>> We would add a new `use_channel_announcement_filters` optional feature
>> bit (7 for example), and a new `channel_announcement_filters` message.
>>
>> When a node that supports channel announcement filters receives an
>> `init` message with the `use_channel_announcement_filters` bit set, it
>> sends back its channel filters.
>>
>> When a node that supports channel announcement filters receives
>> a`channel_announcement_filters` message, it uses it to filter channel
>> announcements (and, implicitly ,channel updates) before sending them.
>>
>> The filters we have in mind are simple:
>> - Sort announcements by short channel id
>> - Compute a marker height, which is `144 * ((now - 7 * 144) / 144)`
>> (we round to multiples of 144 to make sync easier)
>> - Group channel announcements that were created before this marker by
>> groups of 144 blocks
>> - Group channel announcements that were created after this marker by
>> groups of 1 block
>> - For each group, sort and concatenate all channel announcements short
>> channel ids and hash the result (we could use sha256, or the first 16
>> bytes of the sha256 hash)
>>
>> The new `channel_announcement_filters` would then be a list of
>> (height, hash) pairs ordered by increasing heights.
>>
>> This implies that implementation can easily sort announcements by
>> short channel id, which should not be very difficult.
>> An additional step could be to send all short channel ids for all
>> groups for which the group hash did not match. Alternatively we could
>> use smarter filters
>>
>> The use case we have in mind is mobile nodes, or more generally nodes
>> which are often offline and need to resync very often.
>>
>> Cheers,
>> Fabrice
>> _______________________________________________
>> Lightning-dev mailing list
>> Lightning-dev at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
_______________________________________________
Lightning-dev mailing list
Lightning-dev at lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180207/cb4ed693/attachment-0001.html>;
Jim Posen [ARCHIVE] /
npub1nc…lqt2n
2023-06-09 12:48:51
in reply to nevent1q…25jv
Author Public Key
npub1ncnj8arudstdxzfhxk7k4nwgkrw3hyw8sgt0wqqmm5hh2c4knmgs2lqt2nPublished at
2023-06-09 12:48:51Event JSON
{
"id": "d8b35a7c9245cda518a356553e6c8020c10ef983028846d7e11b966e4930766a",
"pubkey": "9e2723f47c6c16d3093735bd6acdc8b0dd1b91c78216f7001bdd2f7562b69ed1",
"created_at": 1686314931,
"kind": 1,
"tags": [
[
"e",
"c3f2627a28f3f44c130da7dce36c10040222ba86792d87bfb3e3dcd970958ae9",
"",
"root"
],
[
"e",
"88e903826d34f0c90835bf50bc2d8396e73efea6d2526dd532ebcf7a8d651947",
"",
"reply"
],
[
"p",
"81c48ba46c211bc8fdb490d1ccfb03609c7ea090f8587ddca1c990676f09cfd3"
]
],
"content": "📅 Original date posted:2018-02-07\n📝 Original message:\nI like Christian's proposal of adding a simple announcement cutoff\ntimestamp with the intention of designing something more sophisticated\ngiven more time.\n\nI prefer the approach of having an optional feature bit signalling that a\n`set_gossip_timestamp` message must be sent immediately after `init`, as\nLaolu suggested. This way it doesn't conflict with and other possible\nhandshake extensions.\n\n\nOn Feb 7, 2018 9:50 AM, \"Fabrice Drouin\" \u003cfabrice.drouin at acinq.fr\u003e wrote:\n\nHi,\n\nSuppose you partition nodes into 3 generic roles:\n- payers: they mostly send payments, are typically small and operated\nby end users, and are offline quite a lot\n- relayers: they mostly relay payments, and would be online most of\nthe time (if they're too unreliable other nodes will eventually close\ntheir channels with them)\n- payees: they mostly receive payments, how often they can be online\nis directly link to their particular mode of operations (since you\nneed to be online to receive payments)\n\nOf course most nodes would play more or less all roles. However,\nmobile nodes would probably be mostly \"payers\", and they have specific\nproperties:\n- if they don't relay payments they don't have to be announced. There\ncould be millions of mobile nodes that would have no impact on the\nsize of the routing table\n- it does not impact the network when they're offline\n- but they need an accurate routing table. This is very different from\nnodes who mostly relay or accept payements\n- they would be connected to a very small number of nodes\n- they would typically be online for just a few hours every day, but\ncould be stopped/paused/restarted many times a day\n\nLaolu wrote:\n\u003e So I think the primary distinction between y'alls proposals is that\n\u003e cdecker's proposal focuses on eventually synchronizing all the set of\n\u003e _updates_, while Fabrice's proposal cares *only* about the newly created\n\u003e channels. It only cares about new channels as the rationale is that if\nonce\n\u003e tries to route over a channel with a state channel update for it, then\n\u003e you'll get an error with the latest update encapsulated.\n\nIf you have one filter per day and they don't match (because your peer\nhas channels that you missed, or\n have been closed and you were not aware of it) then you will receive\nall channel announcements for\nthis particular day, and the associated updates\n\nLaolu wrote:\n\u003e I think he's actually proposing just a general update horizon in which\n\u003e vertexes+edges with a lower time stamp just shouldn't be set at all. In\nthe\n\u003e case of an old zombie channel which was resurrected, it would eventually\nbe\n\u003e re-propagated as the node on either end of the channel should broadcast a\n\u003e fresh update along with the original chan ann.\n\nYes but it could take a long time. It may be worse on testnet since it\nseems that nodes\ndon't change their fees very often. \"Payer nodes\" need a good routing\ntable (as opposed\nto \"relayers\" which could work without one if they never initiate payments)\n\nLaolu wrote:\n\u003e This seems to assume that both nodes have a strongly synchronized view of\n\u003e the network. Otherwise, they'll fall back to sending everything that went\non\n\u003e during the entire epoch regularly. It also doesn't address the zombie\nchurn\n\u003e issue as they may eventually send you very old channels you'll have to\ndeal\n\u003e with (or discard).\n\nYes I agree that for nodes which have connections to a lot of peers,\nstrongly synchronized routing tables is\nharder to achieve since a small change may invalidate an entire\nbucket. Real queryable filters would be much\nbetter, but worst case scenario is we've sent an additionnal 30 Kb or\no of sync messages.\n(A very naive filter would be sort + pack all short ids for example)\n\nBut we focus on nodes which are connected to a very small number of\npeers, and in this particular\ncase it is not an unrealistic expectation.\nWe have built a prototype and on testnet it works fairly well. I also\nfound nodes which have no direct\nchannel betweem them but produce the same filters for 75% of the\nbuckets (\"produce\" here means\nthat I opened a simple gossip connection to them, got their routing\ntable and used it to generate filters).\n\n\nLaolu wrote:\n\u003e How far back would this go? Weeks, months, years?\nSince forever :)\nOne filter per day for all annoucements that are older than now - 1\nweek (modulo 144)\nOne filter per block for recent announcements\n\n\u003e\n\u003e FWIW this approach optimizes for just learning of new channels instead of\n\u003e learning of the freshest state you haven't yet seen.\n\nI'd say it optimizes the case where you are connected to very few\npeers, and are online a few times every day (?)\n\n\u003e\n\u003e -- Laolu\n\u003e\n\u003e\n\u003e On Mon, Feb 5, 2018 at 7:08 AM Fabrice Drouin \u003cfabrice.drouin at acinq.fr\u003e\n\u003e wrote:\n\u003e\u003e\n\u003e\u003e Hi,\n\u003e\u003e\n\u003e\u003e On 5 February 2018 at 14:02, Christian Decker\n\u003e\u003e \u003cdecker.christian at gmail.com\u003e wrote:\n\u003e\u003e \u003e Hi everyone\n\u003e\u003e \u003e\n\u003e\u003e \u003e The feature bit is even, meaning that it is required from the peer,\n\u003e\u003e \u003e since we extend the `init` message itself, and a peer that does not\n\u003e\u003e \u003e support this feature would be unable to parse any future extensions to\n\u003e\u003e \u003e the `init` message. Alternatively we could create a new\n\u003e\u003e \u003e `set_gossip_timestamp` message that is only sent if both endpoints\n\u003e\u003e \u003e support this proposal, but that could result in duplicate messages\nbeing\n\u003e\u003e \u003e delivered between the `init` and the `set_gossip_timestamp` message and\n\u003e\u003e \u003e it'd require additional messages.\n\u003e\u003e\n\u003e\u003e We chose the other aproach and propose to use an optional feature\n\u003e\u003e\n\u003e\u003e \u003e The reason I'm using timestamp and not the blockheight in the short\n\u003e\u003e \u003e channel ID is that we already use the timestamp for pruning. In the\n\u003e\u003e \u003e blockheight based timestamp we might ignore channels that were created,\n\u003e\u003e \u003e then not announced or forgotten, and then later came back and are now\n\u003e\u003e \u003e stable.\n\u003e\u003e\n\u003e\u003e Just to be clear, you propose to use the timestamp of the most recent\n\u003e\u003e channel updates to filter\n\u003e\u003e the associated channel announcements ?\n\u003e\u003e\n\u003e\u003e \u003e I hope this rather simple proposal is sufficient to fix the short-term\n\u003e\u003e \u003e issues we are facing with the initial sync, while we wait for a real\n\u003e\u003e \u003e sync protocol. It is definitely not meant to allow perfect\n\u003e\u003e \u003e synchronization of the topology between peers, but then again I don't\n\u003e\u003e \u003e believe that is strictly necessary to make the routing successful.\n\u003e\u003e \u003e\n\u003e\u003e \u003e Please let me know what you think, and I'd love to discuss Pierre's\n\u003e\u003e \u003e proposal as well.\n\u003e\u003e \u003e\n\u003e\u003e \u003e Cheers,\n\u003e\u003e \u003e Christian\n\u003e\u003e\n\u003e\u003e Our idea is to group channel announcements by \"buckets\", create a\n\u003e\u003e filter for each bucket, exchange and use them to filter out channel\n\u003e\u003e announcements.\n\u003e\u003e\n\u003e\u003e We would add a new `use_channel_announcement_filters` optional feature\n\u003e\u003e bit (7 for example), and a new `channel_announcement_filters` message.\n\u003e\u003e\n\u003e\u003e When a node that supports channel announcement filters receives an\n\u003e\u003e `init` message with the `use_channel_announcement_filters` bit set, it\n\u003e\u003e sends back its channel filters.\n\u003e\u003e\n\u003e\u003e When a node that supports channel announcement filters receives\n\u003e\u003e a`channel_announcement_filters` message, it uses it to filter channel\n\u003e\u003e announcements (and, implicitly ,channel updates) before sending them.\n\u003e\u003e\n\u003e\u003e The filters we have in mind are simple:\n\u003e\u003e - Sort announcements by short channel id\n\u003e\u003e - Compute a marker height, which is `144 * ((now - 7 * 144) / 144)`\n\u003e\u003e (we round to multiples of 144 to make sync easier)\n\u003e\u003e - Group channel announcements that were created before this marker by\n\u003e\u003e groups of 144 blocks\n\u003e\u003e - Group channel announcements that were created after this marker by\n\u003e\u003e groups of 1 block\n\u003e\u003e - For each group, sort and concatenate all channel announcements short\n\u003e\u003e channel ids and hash the result (we could use sha256, or the first 16\n\u003e\u003e bytes of the sha256 hash)\n\u003e\u003e\n\u003e\u003e The new `channel_announcement_filters` would then be a list of\n\u003e\u003e (height, hash) pairs ordered by increasing heights.\n\u003e\u003e\n\u003e\u003e This implies that implementation can easily sort announcements by\n\u003e\u003e short channel id, which should not be very difficult.\n\u003e\u003e An additional step could be to send all short channel ids for all\n\u003e\u003e groups for which the group hash did not match. Alternatively we could\n\u003e\u003e use smarter filters\n\u003e\u003e\n\u003e\u003e The use case we have in mind is mobile nodes, or more generally nodes\n\u003e\u003e which are often offline and need to resync very often.\n\u003e\u003e\n\u003e\u003e Cheers,\n\u003e\u003e Fabrice\n\u003e\u003e _______________________________________________\n\u003e\u003e Lightning-dev mailing list\n\u003e\u003e Lightning-dev at lists.linuxfoundation.org\n\u003e\u003e https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev\n_______________________________________________\nLightning-dev mailing list\nLightning-dev at lists.linuxfoundation.org\nhttps://lists.linuxfoundation.org/mailman/listinfo/lightning-dev\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180207/cb4ed693/attachment-0001.html\u003e",
"sig": "7482669010e7c6081c9439368b359388f6be59b12af23dd2906db90a7b74095ba919c449dfcb476596fdc7d02cb3f585b0cfa64a796c38f6c7ca87f5d4a977e2"
}