kasperd / is on Nostr

Currently testing this platform to decide whether it's the future of social networking.

Curriculum Vitae:
PhD degree from Aarhus University
Worked at Google Zürich and London
Partner at Intempus Timeregistrering - now part of Visma
Operating [nat64.net/](https://nat64.net/ )

Last Notes

2024-09-02 15:58:30

- reply

I have often used MTU problems as an interview question. Good candidates would usually figure it out in 30 minutes. Not so good candidates would not figure it out in 60 minutes.
And then there was that one time I had a candidate who figured it out in 10 minutes. He was offered the position and then declined.

2024-08-26 21:46:53

- reply

One example from my own experience is with DNS. Back in 2015 we had a series of outages due to the provider who was hosting our DNS zones.
First the provider had a six hour outage during which we could do nothing, and we couldn't even get in touch with the provider.
Then we tried redundancy across two different DNS providers, which lead to another two outages when the first provider did not handle redundant setups correctly.
At that point I said we will never want to go through this again. And we decided to operate our own DNS servers with redundancy across four different hosting providers.
We have now used that setup for over nine years and not had a single outage of our DNS zone. We have of course had individual DNS servers be down for some time every now and then, but we never had all four go down.
Such a track record of course helps when I am making arguments for multi provider redundancy. I can point at this setup and say over nine years with 100% uptime, had we used Cloudflare we wouldn't have had such a high uptime.

2024-08-26 20:28:58

- reply

I am thinking one should never trust a single supplier no matter how big and trustworthy you think they are.
One opinion I have heard a few times goes like this: It's not a problem if we are completely dependent on CloudFlare, GCE, or Amazon. If they go down many other systems will go down as well, and customers won't blame us.
The problem with that line of thinking is, that the bigger the provider is, the less important each customer is to the provider. There have been stories about individual customers of big providers getting into trouble.

2024-08-25 14:25:21

- reply

But why did your router send destination unreachable errors?
Using tcpdump I saw `ICMP6, destination unreachable, unreachable address 2a05:87c6:1:1cc8::1, length 88` coming from 2a00:1098:2::7. And in the traceroute output above that is indicated as `!H`
That behavior from your router is one I would only expect if the next hop address did not respond to neighbor solicitation.
Did you verify that the address you pinged was identical to the next hop address currently in the routing table on that particular router?

2024-08-25 14:06:05

- reply

That's not what I saw. When I used traceroute I got destination unreachable errors from one of your routers. Here is the traceroute output I shared earlier:
```
traceroute to 2a05:87c6:1:1cc8::1 (2a05:87c6:1:1cc8::1), 30 hops max, 80 byte packets
1 2a00:1098:1::2d 0.284 ms 0.231 ms 0.217 ms
2 2a00:1098:2:ffff::9c 0.401 ms 0.387 ms 0.375 ms
3 2a00:1098:2:1::1c:1 0.437 ms 0.417 ms 0.453 ms
4 2a00:1098:2::7 0.323 ms 0.376 ms 0.365 ms
5 2a00:1098:2::7 3055.197 ms !H 3055.184 ms !H 3055.165 ms !H
```

2024-08-25 13:41:08

- reply

I agree that is one of the most likely explanations of the root cause. It is something which could have been automatically detected when receiving the advertisements from the route server. But I do not know if any BGP implementation has the logic to detect and filter such faulty advertisements.

2024-08-25 12:46:40

- reply

It does not look specific to A&A. One of the network paths confirmed to be affected did not go through A&A.
It's confirmed which exchange the issue lies at. But it's unclear to me if the issue is with the destination network or the exchange itself. I do not have a presence at that exchange myself, so the only information I have to work from is output from traceroute and similar tools.
It looks to me like one aspect of the BGP implementation you are using is a contributing factor. But that may very well be standard behavior. For all I know it could be that all other BGP implementations would respond the same in this corner case.

2024-08-25 11:29:47

- reply

A router server is indeed what I imagine as the most likely way to get the same failure mode as what I saw with OSPF.
It's technically possible to verify that neighbor discovery works for the next hop that you receive from a route server. How to actually implement it depends on the underlying platform and could be complicated.
What did you verify on the data path? The traceroute command I tried failed before the packet reached the data path from 2a00:1098:2::7 to the next hop.

2024-08-25 10:46:49

- reply

This reminds me of a failure mode I observed in OSPF. I don't know for certain if the same failure mode exists in BGP, but the symptoms sure look similar.
In OSPF the nodes connected to an Ethernet segment will elect a master. OSPF will verify that each node can communicate with the master, and this can be continuously verified with BFD.
OSPF will then assume that if two nodes can both communicate with the master, then they can also communicate directly with each other. If that assumption does not hold OSPF will configure routes which will always fail during the neighbor discovery step of the packet forwarding. I observed this behavior in two mainstream OSPF implementations.
@npub1wy5…64vc @npub1sv0…rq9x Does the BGP implementation that you are using accept routes for which the advertised next hop does not respond to neighbor solicitation?

2024-08-25 10:05:54

- reply

That hostname resolves to `2a05:87c6:1:1cc8::1` for me. I confirmed that the IP address is reachable from most networks I tried but it is not reachable from @npub1wy5…64vc or @npub1sv0…rq9x
The problem seems to affect both ssh and ping. With both protocols I get an ICMPv6 error message back:
```
ICMP6, destination unreachable, unreachable address 2a05:87c6:1:1cc8::1, length 88
```
The origin of the ICMPv6 error message is 2001:8b0:0:53::109 or 2a00:1098:2::7 depending on which provider I connect from.
Here is an example traceroute output:
```
root@nat64-london:~# traceroute -n -I 2a05:87c6:1:1cc8::1
traceroute to 2a05:87c6:1:1cc8::1 (2a05:87c6:1:1cc8::1), 30 hops max, 80 byte packets
1 2a00:1098:1::2d 0.284 ms 0.231 ms 0.217 ms
2 2a00:1098:2:ffff::9c 0.401 ms 0.387 ms 0.375 ms
3 2a00:1098:2:1::1c:1 0.437 ms 0.417 ms 0.453 ms
4 2a00:1098:2::7 0.323 ms 0.376 ms 0.365 ms
5 2a00:1098:2::7 3055.197 ms !H 3055.184 ms !H 3055.165 ms !H
root@nat64-london:~#
```
The last hop limit exceeded message came from the same IP as the unreachable address. The later error message was delayed by 3 seconds.
In my experience the 3 second delay is most often caused by a neighbor discovery timeout. That indicates that 2a00:1098:2::7 and 2001:8b0:0:53::109 have a route forward towards the destination, but the next hop is not responding to neighbor solicitation messages.