0

Anycast DNS outage (connection times out)

NextDNS stopped responding to all DNS queries since Thursday on one of my servers (hetzner).

All anycast IPs seems to fail after a few hops.

Diagnostic info provided below.

https://nextdns.io/diag/a558b0a0-2045-11ef-b3cd-4b0c5b3ab9a1

 

NOTE: I've also filed an issue on hetzner's end, and they're currently diagnosing the root cause.

7 replies

null
    • NextDNs
    • 5 mths ago
    • Reported - view

    We are investigating. Two totally different network paths blocking this IP is fishy. May I ask why the source IP is a hetzner IP? Are you using NextDNS from a server?

      • sefidel
      • 5 mths ago
      • Reported - view

      Yes, as mentioned in the description.

       

      I’ve been using NextDNS on my server to do some experiments. 

      • sefidel
      • 5 mths ago
      • Reported - view

       Update from Hetzner:

      It looks like the DNS server are blocking your server IP address. We are able to reach all IPs from Helsinki without any issues. We aren't able to solve this issue. 

      Please contact the owner of this service directly. 

      HOST: hel1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
       1.|-- static.88-198-**-**.clie  0.0%    20   18.4  17.4  10.2  27.6   4.4
       2.|-- ddos-mitigation.dc1.hel1.  0.0%    20    0.2   0.2   0.2   0.3   0.0
       3.|-- juniper4.dc1.hel1.hetzner  0.0%    20    0.6   0.4   0.4   0.6   0.1
       4.|-- ???                       100.0    20    0.0   0.0   0.0   0.0   0.0
       5.|-- ???                       100.0    20    0.0   0.0   0.0   0.0   0.0
       6.|-- po-31.lag.mow01.ru.misaka  0.0%    20   35.4  35.4  35.3  35.6   0.1
      7.|-- dns1.nextdns.io 0.0% 20 36.2 36.2 36.1 36.3 0.0

      HOST: hel1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
       1.|-- static.88-198-**-**.clie  0.0%    20   12.2  19.4   4.3  62.0  12.0
       2.|-- ddos-mitigation.dc1.hel1.  0.0%    20    0.4   0.3   0.2   0.4   0.1
      3.|-- core52.sto.hetzner.com 0.0% 20 6.7 6.7 6.5 7.0 0.1
      4.|-- core51.ams.hetzner.com 0.0% 20 27.2 27.1 27.0 27.4 0.1
      5.|-- core10.ams.hetzner.com 0.0% 20 27.1 27.2 27.1 27.3 0.1
      6.|-- b901f0cc.ptr.era-ix.net 0.0% 20 28.0 32.9 27.6 58.7 9.4
       7.|-- ???                       100.0    20    0.0   0.0   0.0   0.0   0.0
       8.|-- ???                       100.0    20    0.0   0.0   0.0   0.0   0.0
       9.|-- ???                       100.0    20    0.0   0.0   0.0   0.0   0.0
      10.|-- dns2.nextdns.io 0.0% 20 27.6 27.7 27.6 28.3 0.2

      Is there any rate-limiting in place?

      • sefidel
      • 5 mths ago
      • Reported - view

       Upon further investigation I've found that `dns.nextdns.io` is reachable via IPv4, but not IPv6.

       

      
      
      # mtr -s 1000 -r -c 1 dns.nextdns.io -4
      Start: 2024-06-02T09:47:48+0000
      HOST: cobalt                      Loss%   Snt   Last   Avg  Best  Wrst StDev
        1.|-- _gateway                   0.0%     1    0.7   0.7   0.7   0.7   0.0
        2.|-- core32.hel1.hetzner.com    0.0%     1    0.8   0.8   0.8   0.8   0.0
        3.|-- juniper4.dc1.hel1.hetzner  0.0%     1    0.6   0.6   0.6   0.6   0.0
        4.|-- 212.133.6.1                0.0%     1    1.6   1.6   1.6   1.6   0.0
        5.|-- ae1.3107.ear3.stk2.neo.co  0.0%     1    7.3   7.3   7.3   7.3   0.0
        6.|-- 212.133.1.90               0.0%     1    7.3   7.3   7.3   7.3   0.0
        7.|-- unn-156-146-32-149.cdn77.  0.0%     1    7.2   7.2   7.2   7.2   0.0
        8.|-- dns.nextdns.io             0.0%     1    7.2   7.2   7.2   7.2   0.0
      
      # mtr -s 1000 -r -c 1 dns.nextdns.io -6
      Start: 2024-06-02T09:48:00+0000
      HOST: cobalt                      Loss%   Snt   Last   Avg  Best  Wrst StDev
        1.|-- 2a01:4f9::a:2:b            0.0%     1    0.8   0.8   0.8   0.8   0.0
        2.|-- core31.hel1.hetzner.com    0.0%     1    0.7   0.7   0.7   0.7   0.0
        3.|-- core52.sto.hetzner.com     0.0%     1    7.3   7.3   7.3   7.3   0.0
        4.|-- core3.sto.hetzner.com      0.0%     1    7.2   7.2   7.2   7.2   0.0
        5.|-- netnod-ix-ge-a-sth-1500.a  0.0%     1    7.3   7.3   7.3   7.3   0.0
        6.|-- 2a00:11c0:2b:2::           0.0%     1    7.0   7.0   7.0   7.0   0.0
        7.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
      
      
      
      • sefidel
      • 5 mths ago
      • Reported - view

       Nevermind, both v4 and v6 is unreachable.

      I've tried `ping.nextdns.io` with a VPN connection to the server, and it seems like only anycast addresses are erroring out.

      • NextDNs
      • 5 mths ago
      • Reported - view

       this IP is not blocked on our side. 

      • sefidel
      • 5 mths ago
      • Reported - view

       I see. I’ll wait for the investigation to finish, then.

Content aside

  • 5 mths agoLast active
  • 7Replies
  • 187Views
  • 4 Following