0

nextdns stops responding on cli on asuswrt-merlin 386.1

After a while (not sure how long) the following happens when you try to use the cli:

admin@router:/tmp/home/root# /jffs/nextdns/nextdns status
Error: service nextdns status: fork/exec /sbin/service: cannot allocate memory:
admin@router:/tmp/home/root# /jffs/nextdns/nextdns restart
Error: service nextdns restart: fork/exec /sbin/service: cannot allocate memory: 

Only possibility to fix it is to reboot the router (at least the only solution I found so far). Any ideas why this is happening? DNS resolution is working all the time so far.

kr, chris

17replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • The error message suggests a memory leak?

    cannot allocate memory
    Like
  • Yes, but why? Hope the devs can check and fix it. :) 

    nextdns-cli is on v1.11.0 btw.

    Like
    • Chris At probably highly fragmented memory. A reboot of the router should fix. Does it happen repeatedly?

      Like
      • Chris At
      • Chris_At
      • 2 mths ago
      • Reported - view

      Olivier Poitrey 

      Yes, reboot fixes it. Can't tell how often or how soon after a reboot it happens, found it twice now and decided to post it here. Also I disabled "regularly flush cache" in asuswrt options so maybe this influences with it? Upgraded to 386.2beta3 today and will keep an eye on it.

      Like
  • admin@router:/tmp/home/root# /jffs/nextdns/nextdns status
    fatal error: runtime: out of memory
    
    runtime stack:
    runtime.throw(0x3b4dc8, 0x16)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/panic.go:1116 +0x5c fp=0xbef029e8 sp=0xbef029d4 pc=0x44de4
    runtime.sysMap(0x800000, 0x400000, 0x610518)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mem_linux.go:169 +0xa8 fp=0xbef02a0c sp=0xbef029e8 pc=0x27014
    runtime.(*linearAlloc).alloc(0x6010c0, 0x400000, 0x400000, 0x610518, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1447 +0x94 fp=0xbef02a24 sp=0xbef02a0c pc=0x1d430
    runtime.(*mheap).sysAlloc(0x600d68, 0x400000, 0x1, 0x1)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:633 +0x54 fp=0xbef02a80 sp=0xbef02a24 pc=0x1b53c
    runtime.(*mheap).grow(0x600d68, 0x1, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1344 +0x6c fp=0xbef02abc sp=0xbef02a80 pc=0x383d0
    runtime.(*mheap).allocSpan(0x600d68, 0x1, 0x11e00, 0x610528, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1160 +0x63c fp=0xbef02b0c sp=0xbef02abc pc=0x381a8
    runtime.(*mheap).alloc.func1()
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:907 +0x50 fp=0xbef02b34 sp=0xbef02b0c pc=0x7193c
    runtime.(*mheap).alloc(0x600d68, 0x1, 0x11e, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:901 +0x58 fp=0xbef02b58 sp=0xbef02b34 pc=0x377c8
    runtime.(*mcentral).grow(0x6023b8, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:506 +0x6c fp=0xbef02b7c sp=0xbef02b58 pc=0x26ab0
    runtime.(*mcentral).cacheSpan(0x6023b8, 0x11f4c)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:177 +0x448 fp=0xbef02bc4 sp=0xbef02b7c pc=0x267c8
    runtime.(*mcache).refill(0x40058088, 0x1e)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcache.go:142 +0x9c fp=0xbef02bd8 sp=0xbef02bc4 pc=0x26104
    runtime.(*mcache).nextFree(0x40058088, 0x5fc11e, 0x0, 0x1, 0x70724)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:880 +0x7c fp=0xbef02bf8 sp=0xbef02bd8 pc=0x1bd28
    runtime.mallocgc(0xe0, 0x3a6a18, 0x1, 0x1)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1061 +0x7c8 fp=0xbef02c60 sp=0xbef02bf8 pc=0x1c69c
    runtime.newobject(0x3a6a18, 0x491fc)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1195 +0x2c fp=0xbef02c74 sp=0xbef02c60 pc=0x1cae8
    runtime.malg(0x8000, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:3493 +0x1c fp=0xbef02c94 sp=0xbef02c74 pc=0x4fbe8
    runtime.mpreinit(0x5fcf00)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/os_linux.go:340 +0x1c fp=0xbef02ca0 sp=0xbef02c94 pc=0x42330
    runtime.mcommoninit(0x5fcf00, 0xffffffff, 0xffffffff)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:663 +0x10c fp=0xbef02cc8 sp=0xbef02ca0 pc=0x49350
    runtime.schedinit()
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:565 +0x84 fp=0xbef02cf0 sp=0xbef02cc8 pc=0x48ca8
    runtime.rt0_go(0xbef02e2f, 0xbef02e45, 0x0, 0xbef02e4c, 0xbef02e6e, 0xbef02e79, 0xbef02e81, 0xbef02e8c, 0xbef02e9f, 0xbef02eaf, ...)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/asm_arm.s:163 +0x7c fp=0xbef02d30 sp=0xbef02cf0 pc=0x7777c

    Uptime after reboot: 2 days 23 hour(s) 53 minute(s) 29 seconds

    Additional info: using dual-wan with usb-lte and have to "/jffs/nextdns/nextdns restart" everytime wan changes (done via wan-event script) because otherwise dns isn't working anymore, so maybe these restarts lead to that memory problem?

    Also, nothing to do with this bug, but the cli shortcut "nextdns ..." is only working after fresh install/reinstall of nextdns, after a reboot it is gone and you have to use "/jffs/nextdns/nextdns ...". Maybe this can be fixed also.

    kr, chris

    Like
  • And again after yesterdays reboot to fix it:

    ASUSWRT-Merlin RT-AC68U 386.2_beta3 Sun Mar 28 17:47:47 UTC 2021
    admin@router:/tmp/home/root# /jffs/nextdns/nextdns status
    fatal error: runtime: out of memory
    
    runtime stack:
    runtime.throw(0x3b4dc8, 0x16)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/panic.go:1116 +0x5c fp=0xbec499e8 sp=0xbec499d4 pc=0x44de4
    runtime.sysMap(0x800000, 0x400000, 0x610518)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mem_linux.go:169 +0xa8 fp=0xbec49a0c sp=0xbec499e8 pc=0x27014
    runtime.(*linearAlloc).alloc(0x6010c0, 0x400000, 0x400000, 0x610518, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1447 +0x94 fp=0xbec49a24 sp=0xbec49a0c pc=0x1d430
    runtime.(*mheap).sysAlloc(0x600d68, 0x400000, 0x1, 0x1)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:633 +0x54 fp=0xbec49a80 sp=0xbec49a24 pc=0x1b53c
    runtime.(*mheap).grow(0x600d68, 0x1, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1344 +0x6c fp=0xbec49abc sp=0xbec49a80 pc=0x383d0
    runtime.(*mheap).allocSpan(0x600d68, 0x1, 0x11e00, 0x610528, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1160 +0x63c fp=0xbec49b0c sp=0xbec49abc pc=0x381a8
    runtime.(*mheap).alloc.func1()
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:907 +0x50 fp=0xbec49b34 sp=0xbec49b0c pc=0x7193c
    runtime.(*mheap).alloc(0x600d68, 0x1, 0x11e, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:901 +0x58 fp=0xbec49b58 sp=0xbec49b34 pc=0x377c8
    runtime.(*mcentral).grow(0x6023b8, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:506 +0x6c fp=0xbec49b7c sp=0xbec49b58 pc=0x26ab0
    runtime.(*mcentral).cacheSpan(0x6023b8, 0x11f4c)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:177 +0x448 fp=0xbec49bc4 sp=0xbec49b7c pc=0x267c8
    runtime.(*mcache).refill(0x40087088, 0x1e)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcache.go:142 +0x9c fp=0xbec49bd8 sp=0xbec49bc4 pc=0x26104
    runtime.(*mcache).nextFree(0x40087088, 0x5fc11e, 0x0, 0x1, 0x70724)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:880 +0x7c fp=0xbec49bf8 sp=0xbec49bd8 pc=0x1bd28
    runtime.mallocgc(0xe0, 0x3a6a18, 0x1, 0x1)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1061 +0x7c8 fp=0xbec49c60 sp=0xbec49bf8 pc=0x1c69c
    runtime.newobject(0x3a6a18, 0x491fc)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1195 +0x2c fp=0xbec49c74 sp=0xbec49c60 pc=0x1cae8
    runtime.malg(0x8000, 0x0)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:3493 +0x1c fp=0xbec49c94 sp=0xbec49c74 pc=0x4fbe8
    runtime.mpreinit(0x5fcf00)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/os_linux.go:340 +0x1c fp=0xbec49ca0 sp=0xbec49c94 pc=0x42330
    runtime.mcommoninit(0x5fcf00, 0xffffffff, 0xffffffff)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:663 +0x10c fp=0xbec49cc8 sp=0xbec49ca0 pc=0x49350
    runtime.schedinit()
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:565 +0x84 fp=0xbec49cf0 sp=0xbec49cc8 pc=0x48ca8
    runtime.rt0_go(0xbec49e2f, 0xbec49e45, 0x0, 0xbec49e4c, 0xbec49e6e, 0xbec49e79, 0xbec49e81, 0xbec49e8c, 0xbec49e9f, 0xbec49eaf, ...)
            /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/asm_arm.s:163 +0x7c fp=0xbec49d30 sp=0xbec49cf0 pc=0x7777c
    
    
    Like
  • What size did you set for the cache?

    Like
      • Chris At
      • Chris_At
      • 2 mths ago
      • Reported - view

      Olivier Poitrey 10MB, config below (id removed):

      admin@router:/tmp/home/root# /jffs/nextdns/nextdns config
      control /var/run/nextdns.sock
      report-client-info true
      detect-captive-portals false
      hardened-privacy false
      listen localhost:53
      cache-max-age 0s
      max-ttl 5s
      bogus-priv true
      use-hosts true
      auto-activate false
      timeout 5s
      cache-size 10MB
      discovery-dns
      setup-router true
      log-queries false
      Like
    • Chris At try with 1M

      Like
      • Chris At
      • Chris_At
      • 2 mths ago
      • Reported - view

      Olivier Poitrey Ok, will try and keep you updated.

      Any idea how to fix the disappearing nextdns command shortcut I mentioned before?

      Like
    • Chris At the service is supposed to put it back when it starts: https://github.com/nextdns/nextdns/blob/master/host/service/merlin/service.go#L242

      Like
      • Chris At
      • Chris_At
      • 2 mths ago
      • Reported - view

      Olivier Poitrey same with 1M

      admin@router:/tmp/home/root# nextdns status

      fatal error: runtime: out of memory

       

      runtime stack:

      runtime.throw(0x3b4dc8, 0x16)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/panic.go

      :1116 +0x5c fp=0xbe99ba28 sp=0xbe99ba14 pc=0x44de4

      runtime.sysMap(0x800000, 0x400000, 0x610518)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mem_linu

      x.go:169 +0xa8 fp=0xbe99ba4c sp=0xbe99ba28 pc=0x27014

      runtime.(*linearAlloc).alloc(0x6010c0, 0x400000, 0x400000, 0x61

      0518, 0x0)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g

      o:1447 +0x94 fp=0xbe99ba64 sp=0xbe99ba4c pc=0x1d430

      runtime.(*mheap).sysAlloc(0x600d68, 0x400000, 0x1, 0x1)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g

      o:633 +0x54 fp=0xbe99bac0 sp=0xbe99ba64 pc=0x1b53c

      runtime.(*mheap).grow(0x600d68, 0x1, 0x0)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go

      :1344 +0x6c fp=0xbe99bafc sp=0xbe99bac0 pc=0x383d0

      runtime.(*mheap).allocSpan(0x600d68, 0x1, 0x11e00, 0x610528, 0x

      0)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go

      :1160 +0x63c fp=0xbe99bb4c sp=0xbe99bafc pc=0x381a8

      runtime.(*mheap).alloc.func1()

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go

      :907 +0x50 fp=0xbe99bb74 sp=0xbe99bb4c pc=0x7193c

      runtime.(*mheap).alloc(0x600d68, 0x1, 0x11e, 0x0)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go

      :901 +0x58 fp=0xbe99bb98 sp=0xbe99bb74 pc=0x377c8

      runtime.(*mcentral).grow(0x6023b8, 0x0)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral

      .go:506 +0x6c fp=0xbe99bbbc sp=0xbe99bb98 pc=0x26ab0

      runtime.(*mcentral).cacheSpan(0x6023b8, 0x11f4c)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral

      .go:177 +0x448 fp=0xbe99bc04 sp=0xbe99bbbc pc=0x267c8

      runtime.(*mcache).refill(0x400ff088, 0x1e)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcache.g

      o:142 +0x9c fp=0xbe99bc18 sp=0xbe99bc04 pc=0x26104

      runtime.(*mcache).nextFree(0x400ff088, 0x5fc11e, 0x0, 0x1, 0x70

      724)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g

      o:880 +0x7c fp=0xbe99bc38 sp=0xbe99bc18 pc=0x1bd28

      runtime.mallocgc(0xe0, 0x3a6a18, 0x1, 0x1)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g

      o:1061 +0x7c8 fp=0xbe99bca0 sp=0xbe99bc38 pc=0x1c69c

      runtime.newobject(0x3a6a18, 0x491fc)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g

      o:1195 +0x2c fp=0xbe99bcb4 sp=0xbe99bca0 pc=0x1cae8

      runtime.malg(0x8000, 0x0)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:

      3493 +0x1c fp=0xbe99bcd4 sp=0xbe99bcb4 pc=0x4fbe8

      runtime.mpreinit(0x5fcf00)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/os_linux

      .go:340 +0x1c fp=0xbe99bce0 sp=0xbe99bcd4 pc=0x42330

      runtime.mcommoninit(0x5fcf00, 0xffffffff, 0xffffffff)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:

      663 +0x10c fp=0xbe99bd08 sp=0xbe99bce0 pc=0x49350

      runtime.schedinit()

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:

      565 +0x84 fp=0xbe99bd30 sp=0xbe99bd08 pc=0x48ca8

      runtime.rt0_go(0xbe99be6b, 0xbe99be73, 0x0, 0xbe99be7a, 0xbe99b

      e9c, 0xbe99bea7, 0xbe99beaf, 0xbe99beba, 0xbe99becd, 0xbe99bedd

      , ...)

      /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/asm_arm.

      s:163 +0x7c fp=0xbe99bd70 sp=0xbe99bd30 pc=0x7777c

      admin@router:/tmp/home/root#

       

      Like
      • Chris At
      • Chris_At
      • 2 mths ago
      • Reported - view

      Olivier Poitrey Addon: I could restart it using "sh /jffs/nextdns/nextdns.init restart". A simple "nextdns restart" brought the same memory failure as above.

      Like
      • Chris At
      • Chris_At
      • 2 mths ago
      • Reported - view

      Olivier Poitrey Regarding putting back when it starts it seems not to work every time (after router reboot):

      ASUSWRT-Merlin RT-AC68U 386.2_0 Fri Apr 2 02:49:55 UTC 2021

      admin@router:/tmp/home/root# nextdns status

      -sh: nextdns: not found

      admin@router:/tmp/home/root# /jffs/nextdns/nextdns status

      running

      admin@router:/tmp/home/root#

      Like
  • I am having a similar issue. The service just stops running or responding to commands...I'm not seeing any logs indicating what's going on. If I run the debug command, it states:
    INFO: OS: asuswrt-merlin
    INFO: GOARCH: armv7
    INFO: GOOS: linux
    INFO: NEXTDNS_BIN: /jffs/nextdns/nextdns
    INFO: LATEST_RELEASE: 1.11.0
    DEBUG: Start install loop with CURRENT_RELEASE=
    DEBUG: NextDNS is not installed
    i) Install NextDNS
    e) Exit

    If I reinstall, it works again for a few days.

    Like
  • I'm seeing similar behavior on Asus RT-AC86U model running Merlin.  Router auto reboots weekly and by about 5 days in, DNS drops off and I run /jffs/nextdns/nextdns status and it says "not running".

    I start it up and its fine. I just changed caching to 1M as per above suggestion and will let you know.

    NextDNS version if 1.32.1.  I dont have any memory allocation errors in /jffs/syslog.log  but right when I run the "nextdns status" the syslog.log records that its stopped.

    May  7 16:37:02 dropbear[8366]: Password auth succeeded for '<admin user>' from <IP:src port>
    May  7 16:37:04 nextdns.init: Stopped

    (edit)

    I saw via 'top' that memory was consumed with having VPN-Server w/ IPSEC turned on, so I just turned it off and we'll see how it goes w/ that and 1MB cache.

    Like
      • Brian P
      • brianp9906
      • 3 wk ago
      • Reported - view

      Brian P It has not stopped running since the above changes of reducing memory usage on the ASUS Merlin RT-AC86U and then setting NextDNS cache to 1MB.  Been running steady ever since.

      Like
Like Follow
  • 3 wk agoLast active
  • 17Replies
  • 179Views
  • 5 Following