📅 Original date posted:2018-02-06
📝 Original message:
Hi Y'all,
Definitely agree that we need a stop-gap solution to fix the naive table
dump on initial connect. I've been sketching out some fancier stuff, but we
would need time to properly tune the fanciness, and I'm inclined to get out
a stop-gap solution asap. On testnet, the zombie churn is pretty bad atm.
It results in uneasily wasted bandwidth as the churn is now almost constant.
There still exist some very old testnet nodes our there it seems. Beyond the
zombie churn, with the size of the testnet graph, we're forced to send tens
of thousands of messages (even if we're already fully synced) upon initial
connect, so very wasteful over all.
So I think the primary distinction between y'alls proposals is that
cdecker's proposal focuses on eventually synchronizing all the set of
_updates_, while Fabrice's proposal cares *only* about the newly created
channels. It only cares about new channels as the rationale is that if once
tries to route over a channel with a state channel update for it, then
you'll get an error with the latest update encapsulated.
Christian wrote:
> I propose adding a new feature bit (6, i.e., bitmask 0x40) indicating that
> the `init` message is extended with a u32 `gossip_timestamp`, interpreted
as
> a UNIX timestamp.
As the `init` message solely contains two variably sized byte slices, I
don't think we can actually safely extend it in this manner. Instead, a new
message is required, where the semantics of the feature bit _require_ the
other side to send it directly after receiving the `init` message from the
other side.
Aside from that, overall I like the simplicity of the protocol: it
eliminates both the zombie churn, and the intensive initial connection graph
dump without any extra messaging overhead (for reconciliation, etc).
Fabrice wrote:
> Just to be clear, you propose to use the timestamp of the most recent
> channel updates to filter the associated channel announcements ?
I think he's actually proposing just a general update horizon in which
vertexes+edges with a lower time stamp just shouldn't be set at all. In the
case of an old zombie channel which was resurrected, it would eventually be
re-propagated as the node on either end of the channel should broadcast a
fresh update along with the original chan ann.
> When a node that supports channel announcement filters receives
> a`channel_announcement_filters` message, it uses it to filter channel
> announcements (and, implicitly ,channel updates) before sending them
This seems to assume that both nodes have a strongly synchronized view of
the network. Otherwise, they'll fall back to sending everything that went on
during the entire epoch regularly. It also doesn't address the zombie churn
issue as they may eventually send you very old channels you'll have to deal
with (or discard).
> The use case we have in mind is mobile nodes, or more generally nodes
> which are often offline and need to resync very often.
How far back would this go? Weeks, months, years?
FWIW this approach optimizes for just learning of new channels instead of
learning of the freshest state you haven't yet seen.
-- Laolu
On Mon, Feb 5, 2018 at 7:08 AM Fabrice Drouin <fabrice.drouin at acinq.fr>
wrote:
> Hi,
>
> On 5 February 2018 at 14:02, Christian Decker
> <decker.christian at gmail.com> wrote:
> > Hi everyone
> >
> > The feature bit is even, meaning that it is required from the peer,
> > since we extend the `init` message itself, and a peer that does not
> > support this feature would be unable to parse any future extensions to
> > the `init` message. Alternatively we could create a new
> > `set_gossip_timestamp` message that is only sent if both endpoints
> > support this proposal, but that could result in duplicate messages being
> > delivered between the `init` and the `set_gossip_timestamp` message and
> > it'd require additional messages.
>
> We chose the other aproach and propose to use an optional feature
>
> > The reason I'm using timestamp and not the blockheight in the short
> > channel ID is that we already use the timestamp for pruning. In the
> > blockheight based timestamp we might ignore channels that were created,
> > then not announced or forgotten, and then later came back and are now
> > stable.
>
> Just to be clear, you propose to use the timestamp of the most recent
> channel updates to filter
> the associated channel announcements ?
>
> > I hope this rather simple proposal is sufficient to fix the short-term
> > issues we are facing with the initial sync, while we wait for a real
> > sync protocol. It is definitely not meant to allow perfect
> > synchronization of the topology between peers, but then again I don't
> > believe that is strictly necessary to make the routing successful.
> >
> > Please let me know what you think, and I'd love to discuss Pierre's
> > proposal as well.
> >
> > Cheers,
> > Christian
>
> Our idea is to group channel announcements by "buckets", create a
> filter for each bucket, exchange and use them to filter out channel
> announcements.
>
> We would add a new `use_channel_announcement_filters` optional feature
> bit (7 for example), and a new `channel_announcement_filters` message.
>
> When a node that supports channel announcement filters receives an
> `init` message with the `use_channel_announcement_filters` bit set, it
> sends back its channel filters.
>
> When a node that supports channel announcement filters receives
> a`channel_announcement_filters` message, it uses it to filter channel
> announcements (and, implicitly ,channel updates) before sending them.
>
> The filters we have in mind are simple:
> - Sort announcements by short channel id
> - Compute a marker height, which is `144 * ((now - 7 * 144) / 144)`
> (we round to multiples of 144 to make sync easier)
> - Group channel announcements that were created before this marker by
> groups of 144 blocks
> - Group channel announcements that were created after this marker by
> groups of 1 block
> - For each group, sort and concatenate all channel announcements short
> channel ids and hash the result (we could use sha256, or the first 16
> bytes of the sha256 hash)
>
> The new `channel_announcement_filters` would then be a list of
> (height, hash) pairs ordered by increasing heights.
>
> This implies that implementation can easily sort announcements by
> short channel id, which should not be very difficult.
> An additional step could be to send all short channel ids for all
> groups for which the group hash did not match. Alternatively we could
> use smarter filters
>
> The use case we have in mind is mobile nodes, or more generally nodes
> which are often offline and need to resync very often.
>
> Cheers,
> Fabrice
> _______________________________________________
> Lightning-dev mailing list
> Lightning-dev at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180207/2ea76f0a/attachment-0001.html>;
Olaoluwa Osuntokun [ARCHIVE] /
npub19h…zkvn4
2023-06-09 12:48:50
in reply to nevent1q…25g5
Author Public Key
npub19helcfnqgk2jrwzjex2aflq6jwfc8zd9uzzkwlgwhve7lykv23mq5zkvn4Published at
2023-06-09 12:48:50Event JSON
{
"id": "6fab90b9e3f3a749564b4cccaf96205e258ba0bfe7744213443bd00252b8ec03",
"pubkey": "2df3fc2660459521b852c995d4fc1a93938389a5e085677d0ebb33ef92cc5476",
"created_at": 1686314930,
"kind": 1,
"tags": [
[
"e",
"c3f2627a28f3f44c130da7dce36c10040222ba86792d87bfb3e3dcd970958ae9",
"",
"root"
],
[
"e",
"1342c07b361f0f54ffa916c4d97c02c565db71b94bf5e2e6cf6de286a11e44a2",
"",
"reply"
],
[
"p",
"81c48ba46c211bc8fdb490d1ccfb03609c7ea090f8587ddca1c990676f09cfd3"
]
],
"content": "📅 Original date posted:2018-02-06\n📝 Original message:\nHi Y'all,\n\nDefinitely agree that we need a stop-gap solution to fix the naive table\ndump on initial connect. I've been sketching out some fancier stuff, but we\nwould need time to properly tune the fanciness, and I'm inclined to get out\na stop-gap solution asap. On testnet, the zombie churn is pretty bad atm.\nIt results in uneasily wasted bandwidth as the churn is now almost constant.\nThere still exist some very old testnet nodes our there it seems. Beyond the\nzombie churn, with the size of the testnet graph, we're forced to send tens\nof thousands of messages (even if we're already fully synced) upon initial\nconnect, so very wasteful over all.\n\nSo I think the primary distinction between y'alls proposals is that\ncdecker's proposal focuses on eventually synchronizing all the set of\n_updates_, while Fabrice's proposal cares *only* about the newly created\nchannels. It only cares about new channels as the rationale is that if once\ntries to route over a channel with a state channel update for it, then\nyou'll get an error with the latest update encapsulated.\n\nChristian wrote:\n\u003e I propose adding a new feature bit (6, i.e., bitmask 0x40) indicating that\n\u003e the `init` message is extended with a u32 `gossip_timestamp`, interpreted\nas\n\u003e a UNIX timestamp.\n\nAs the `init` message solely contains two variably sized byte slices, I\ndon't think we can actually safely extend it in this manner. Instead, a new\nmessage is required, where the semantics of the feature bit _require_ the\nother side to send it directly after receiving the `init` message from the\nother side.\n\nAside from that, overall I like the simplicity of the protocol: it\neliminates both the zombie churn, and the intensive initial connection graph\ndump without any extra messaging overhead (for reconciliation, etc).\n\nFabrice wrote:\n\u003e Just to be clear, you propose to use the timestamp of the most recent\n\u003e channel updates to filter the associated channel announcements ?\n\nI think he's actually proposing just a general update horizon in which\nvertexes+edges with a lower time stamp just shouldn't be set at all. In the\ncase of an old zombie channel which was resurrected, it would eventually be\nre-propagated as the node on either end of the channel should broadcast a\nfresh update along with the original chan ann.\n\n\u003e When a node that supports channel announcement filters receives\n\u003e a`channel_announcement_filters` message, it uses it to filter channel\n\u003e announcements (and, implicitly ,channel updates) before sending them\n\nThis seems to assume that both nodes have a strongly synchronized view of\nthe network. Otherwise, they'll fall back to sending everything that went on\nduring the entire epoch regularly. It also doesn't address the zombie churn\nissue as they may eventually send you very old channels you'll have to deal\nwith (or discard).\n\n\u003e The use case we have in mind is mobile nodes, or more generally nodes\n\u003e which are often offline and need to resync very often.\n\nHow far back would this go? Weeks, months, years?\n\nFWIW this approach optimizes for just learning of new channels instead of\nlearning of the freshest state you haven't yet seen.\n\n\n-- Laolu\n\n\nOn Mon, Feb 5, 2018 at 7:08 AM Fabrice Drouin \u003cfabrice.drouin at acinq.fr\u003e\nwrote:\n\n\u003e Hi,\n\u003e\n\u003e On 5 February 2018 at 14:02, Christian Decker\n\u003e \u003cdecker.christian at gmail.com\u003e wrote:\n\u003e \u003e Hi everyone\n\u003e \u003e\n\u003e \u003e The feature bit is even, meaning that it is required from the peer,\n\u003e \u003e since we extend the `init` message itself, and a peer that does not\n\u003e \u003e support this feature would be unable to parse any future extensions to\n\u003e \u003e the `init` message. Alternatively we could create a new\n\u003e \u003e `set_gossip_timestamp` message that is only sent if both endpoints\n\u003e \u003e support this proposal, but that could result in duplicate messages being\n\u003e \u003e delivered between the `init` and the `set_gossip_timestamp` message and\n\u003e \u003e it'd require additional messages.\n\u003e\n\u003e We chose the other aproach and propose to use an optional feature\n\u003e\n\u003e \u003e The reason I'm using timestamp and not the blockheight in the short\n\u003e \u003e channel ID is that we already use the timestamp for pruning. In the\n\u003e \u003e blockheight based timestamp we might ignore channels that were created,\n\u003e \u003e then not announced or forgotten, and then later came back and are now\n\u003e \u003e stable.\n\u003e\n\u003e Just to be clear, you propose to use the timestamp of the most recent\n\u003e channel updates to filter\n\u003e the associated channel announcements ?\n\u003e\n\u003e \u003e I hope this rather simple proposal is sufficient to fix the short-term\n\u003e \u003e issues we are facing with the initial sync, while we wait for a real\n\u003e \u003e sync protocol. It is definitely not meant to allow perfect\n\u003e \u003e synchronization of the topology between peers, but then again I don't\n\u003e \u003e believe that is strictly necessary to make the routing successful.\n\u003e \u003e\n\u003e \u003e Please let me know what you think, and I'd love to discuss Pierre's\n\u003e \u003e proposal as well.\n\u003e \u003e\n\u003e \u003e Cheers,\n\u003e \u003e Christian\n\u003e\n\u003e Our idea is to group channel announcements by \"buckets\", create a\n\u003e filter for each bucket, exchange and use them to filter out channel\n\u003e announcements.\n\u003e\n\u003e We would add a new `use_channel_announcement_filters` optional feature\n\u003e bit (7 for example), and a new `channel_announcement_filters` message.\n\u003e\n\u003e When a node that supports channel announcement filters receives an\n\u003e `init` message with the `use_channel_announcement_filters` bit set, it\n\u003e sends back its channel filters.\n\u003e\n\u003e When a node that supports channel announcement filters receives\n\u003e a`channel_announcement_filters` message, it uses it to filter channel\n\u003e announcements (and, implicitly ,channel updates) before sending them.\n\u003e\n\u003e The filters we have in mind are simple:\n\u003e - Sort announcements by short channel id\n\u003e - Compute a marker height, which is `144 * ((now - 7 * 144) / 144)`\n\u003e (we round to multiples of 144 to make sync easier)\n\u003e - Group channel announcements that were created before this marker by\n\u003e groups of 144 blocks\n\u003e - Group channel announcements that were created after this marker by\n\u003e groups of 1 block\n\u003e - For each group, sort and concatenate all channel announcements short\n\u003e channel ids and hash the result (we could use sha256, or the first 16\n\u003e bytes of the sha256 hash)\n\u003e\n\u003e The new `channel_announcement_filters` would then be a list of\n\u003e (height, hash) pairs ordered by increasing heights.\n\u003e\n\u003e This implies that implementation can easily sort announcements by\n\u003e short channel id, which should not be very difficult.\n\u003e An additional step could be to send all short channel ids for all\n\u003e groups for which the group hash did not match. Alternatively we could\n\u003e use smarter filters\n\u003e\n\u003e The use case we have in mind is mobile nodes, or more generally nodes\n\u003e which are often offline and need to resync very often.\n\u003e\n\u003e Cheers,\n\u003e Fabrice\n\u003e _______________________________________________\n\u003e Lightning-dev mailing list\n\u003e Lightning-dev at lists.linuxfoundation.org\n\u003e https://lists.linuxfoundation.org/mailman/listinfo/lightning-dev\n\u003e\n-------------- next part --------------\nAn HTML attachment was scrubbed...\nURL: \u003chttp://lists.linuxfoundation.org/pipermail/lightning-dev/attachments/20180207/2ea76f0a/attachment-0001.html\u003e",
"sig": "0298fa618e5a58de94e835f6a564db7783795dc7d78fdcc2e3ecd8c4d854059bb5cb0cfb70ca3ace749411e17babbc3113de49e98a375f4791438e14573a8b5f"
}