Horrible DNS latencies since yesterday - family is not happy.

Hans Geiblinger   THANKS! I double checked and I already have DNSSEC, Rebind, and Forward unchecked.  I'll keep digging. 

Olivier Poitrey  Maybe I'm reading the tracert incorrectly.  I view all the hops with timeouts as a red-flag. Before a few weeks ago, I was not seeing any timeouts on either tracert reaching back to the nextdns infrastruture.  I do not recall the number of hops though.

Yes sir, no nextdns client. I setup NextDNS manually as I've done for the past year+.. in fact my renewal is coming up shortly. 

1) make sure dnsmasq.conf has the correct entries (note: IPV6 is disabled)

no-resolv
bogus-priv
strict-order
server=45.90.30.0  (btw, that's what the generated page says "0" but I know it's really "114" per an earlier issue where you said it would never be '0'.
server=45.90.28.0 (ditto above)
add-cpe-id=XXXXXXXXX

2) Alter stubby.yml to make sure it has the correct NextDNS entries:

Set -> round_robin_upstreams: 0

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
appdata_dir: "/var/lib/misc"
resolvconf: "/tmp/resolv.conf"
edns_client_subnet_private: 1
round_robin_upstreams: 0
idle_timeout: 9000
tls_connection_retries: 2
tls_backoff_time: 900
timeout: 3000
listen_addresses:
  - 127.0.1.1@53
upstream_recursive_servers:
  - address_data: 45.90.28.114
    tls_auth_name: "XXXXX.dns1.nextdns.io"
  - address_data: 45.90.30.114
    tls_auth_name: "XXXXX.dns2.nextdns.io"

> restart dnsmsgq..

I've restarted dnsmsgq service  few times just to make sure it's not lost and it never recovers until I drop the NextDNS entries and replace them with QUAD9 or Clouldflare or Google.   Then ususally no screams until I switch the router back to NextDNS and try again.

Thanks for taking a look!  Let me know what I'm missing.  I've been following the NextDNS discussion in the ASUS Merlin forums for more than a year.. so this is really a mystery. 

Hans Geiblinger   Hi - Correct.  Works perfectly.

#!/bin/sh
#
# Used by NextDNS to fix the /etc/stubby/stubby.yml AUTOMATICALLY to have "0" for round_robin_upstreams
#
CONFIG=$1
source /usr/sbin/helper.sh
pc_replace "round_robin_upstreams: 1" "round_robin_upstreams: 0" $CONFIG
#
# <EOF>

Olivier Poitrey  TY so much for the clarifications on the ICMP packets and "timeouts".  IDK that was the case with the tracert.   Still seems like an awful lot of hops to me.  As a performance guy, that # of hops would be a nightmare.

OK on 0 or 114 and yes I have the add-cpe-id option set properly.

As for using both dnsmasq and stubby, on ASUS Merlin, I run several other AMTM tools which leverage both stubby and dnsmasq (is my understanding) to implement those features:  skynet, diversion. 

I realize that diversion has some duplicates to NextDNS but it has useful features you do not such as experimental blocking of certain PITA sites.   I've run with both diversion ON and OFF and it does not seem to matter with these recent failures I've been reporting. For the months where I had no issues, I had diversion ON + NextDNS with zero issues.

In the very early threads where you were working with the Merlin developers, changes to both stubby.yml and dnsmasq.conf were listed by the SME on Merlin as required for the "manual" config - way before the client arrived.

I do not run the client b/c it does not integrate well with the other added Merlin AMTM tooling is the last things in those threads. 

Now that I know this is maybe not my ISP's many hops etc.. I'll keep experimenting with adding NextDNS back in.  For all I know it could be a problem caused by something in the entware updates too as they have been known to break things.

Thanks for your guidance! 

Hi Olivier:  Running most current ASUS Merlin

[00:24:38.751471] STUBBY: Stubby version: Stubby 0.3.0
[00:24:38.754325] STUBBY: Read config from file /etc/stubby/stubby.yml
[00:24:38.754632] STUBBY: DNSSEC Validation is OFF
[00:24:38.754664] STUBBY: Transport list is:
[00:24:38.754690] STUBBY:   - TLS
[00:24:38.754716] STUBBY: Privacy Usage Profile is Strict (Authentication required)

 I script change the round_robin_upstreams from 1 to 0 

... /stubby/stubby.yml

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
appdata_dir: "/var/lib/misc"
resolvconf: "/tmp/resolv.conf"
edns_client_subnet_private: 1
round_robin_upstreams: 0
idle_timeout: 9000
tls_connection_retries: 2
tls_backoff_time: 900
timeout: 3000
listen_addresses:
  - 127.0.1.1@53
upstream_recursive_servers:
  - address_data: 45.90.28.0
    tls_auth_name: "XXXXXXX.dns1.nextdns.io"
  - address_data: 45.90.30.0
    tls_auth_name: "XXXXXXX.dns2.nextdns.io"

Thanks for taking another look.  I monitor stubby -l now.. trying to catch whatever's causing my issues.

G. Mobley

Olivier Poitrey  Thank you for the guidance.   I have never tried RRU to 1 b/c the setup instructions state that value must be "0"   I will try "1" tomorrow AM.  I cannot play anymore tonight.  

WRT the NextDNS CLI, yes sir, I have ~ 6x IOT devices and cameras which do not function well when their DNS messed with so I list them on the "DNSFilter" page so they go directly to QUAD9.  They work fine when that is setup.    FWIW, when I have this exact same setup hitting 2 x QUAD9 hosts +  round_robin_upstreams:1  (or also  Clouldflare (used them both to test), the DNS worked for 3 weeks without a hiccup or blip of single DNS cannot be resolved.   I was watching it with "stubby -l"  I'll change tomorrow and see how it goes. 

I'm reviewing that stubby link you posted above.  They are talking a lot about the timeouts...   What's the best timeout for NextDNS?  Quote>  "I am wondering if it would be worthwhile adding a note in stubby.yml.example explaining that stubby will cycle servers when round_robin_upstreams: 0 is set and idle_timeout is set to a value longer than a given upstream server has configured on the backend."  The default idle_timeout in my current stubby.yml is 9000ms.   Could that be triggering the problem that post is referencing?   Thank you sir.

Olivier Poitrey   Morning!  ~ 24 hours with RRU:1 and I've not seen (nor have a I heard screams) about any DNS not resolving. TY!  This <IS> progress!   I'll keep the router running this way for a week, continue to monitor and report in again.    Do you have suggestions on the "proper idle_timeout" for NextDNS which is still set at 9000ms which is the default delivered in ASUS Merlin?  Many people cannot change that b/c that value is not exposed in the ASUS Merlin GUI.  I think they settled on 9000ms as good value for QUAD9, Cloudflare and others they built into the GUI to select for DoT.  You made my day! 

Olivier Poitrey  Morning sir.  Reporting on using above DNS issues on my ASUS/AX86U/Merlin/386.2_2.  When I changed the stubby.yml to use the default of RRU:1 vs RRU:0, the router has not lost DNS resolution - no family standing in my door!   I know that's not the configuration as stated but it's actually returned to being reliable. 

My gut says many ASUS/Merlin users simple fill in the WAN GUI page and never bother editing the correct stubby.yml files which defaults to stubby using -> RRU:1 on Merlin.   I'll continuing monitoring the logs. 

I also found your posting explaining when to use:  X.Y.Z.0 vs the X.Y.Z.### setups.  I'd never read or understood that before that post.  Maybe a good edition to the generated setups?    

See -> https://help.nextdns.io/t/p8htq2y/dnsmasq-setting-clarification-ipv4-address-and-strict-order  

At this point I feel the issue lies in stubby handling the RR setup on a RRU:0 setup - especially based on the link you posted earlier.  I'm betting it's not a common default or code-path. 

If I can, maybe I'll put QUAD9 back in, maually set stubby to RRU:0 to see if DNS starts crapping out with that too!   I've never tried that one.

Thank you sir!

Hi Olivier, Reporting back in.  My NextDNS + Merlin 384.2_2 has been very stable since the above change to use RR:1.   I too believe there is something wrong in DNSMSGQ when setup to use RR:0.   I'm keeping the config running this way and report back in another week.  Since January, 2021, the router has never stayed up more than about 2 days, really meaning DNS working and family not screaming,  with the same config and using RR:0.     Thanks!