0

nextdns stops responding on cli on asuswrt-merlin 386.1

After a while (not sure how long) the following happens when you try to use the cli:

admin@router:/tmp/home/root# /jffs/nextdns/nextdns status
Error: service nextdns status: fork/exec /sbin/service: cannot allocate memory:
admin@router:/tmp/home/root# /jffs/nextdns/nextdns restart
Error: service nextdns restart: fork/exec /sbin/service: cannot allocate memory:

Only possibility to fix it is to reboot the router (at least the only solution I found so far). Any ideas why this is happening? DNS resolution is working all the time so far.

kr, chris

17 replies

null
    • iOS Developer
    • Rob
    • 3 yrs ago
    • Reported - view

    The error message suggests a memory leak? 

    cannot allocate memory
    • Chris_At
    • 3 yrs ago
    • Reported - view

    Yes, but why? Hope the devs can check and fix it. :) 

    nextdns-cli is on v1.11.0 btw.

      • olivier
      • 3 yrs ago
      • Reported - view

      Chris At probably highly fragmented memory. A reboot of the router should fix. Does it happen repeatedly?

      • Chris_At
      • 3 yrs ago
      • Reported - view

      Olivier Poitrey 

      Yes, reboot fixes it. Can't tell how often or how soon after a reboot it happens, found it twice now and decided to post it here. Also I disabled "regularly flush cache" in asuswrt options so maybe this influences with it? Upgraded to 386.2beta3 today and will keep an eye on it.

    • Chris_At
    • 3 yrs ago
    • Reported - view
    admin@router:/tmp/home/root# /jffs/nextdns/nextdns status
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x3b4dc8, 0x16)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/panic.go:1116 +0x5c fp=0xbef029e8 sp=0xbef029d4 pc=0x44de4
runtime.sysMap(0x800000, 0x400000, 0x610518)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mem_linux.go:169 +0xa8 fp=0xbef02a0c sp=0xbef029e8 pc=0x27014
runtime.(*linearAlloc).alloc(0x6010c0, 0x400000, 0x400000, 0x610518, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1447 +0x94 fp=0xbef02a24 sp=0xbef02a0c pc=0x1d430
runtime.(*mheap).sysAlloc(0x600d68, 0x400000, 0x1, 0x1)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:633 +0x54 fp=0xbef02a80 sp=0xbef02a24 pc=0x1b53c
runtime.(*mheap).grow(0x600d68, 0x1, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1344 +0x6c fp=0xbef02abc sp=0xbef02a80 pc=0x383d0
runtime.(*mheap).allocSpan(0x600d68, 0x1, 0x11e00, 0x610528, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1160 +0x63c fp=0xbef02b0c sp=0xbef02abc pc=0x381a8
runtime.(*mheap).alloc.func1()
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:907 +0x50 fp=0xbef02b34 sp=0xbef02b0c pc=0x7193c
runtime.(*mheap).alloc(0x600d68, 0x1, 0x11e, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:901 +0x58 fp=0xbef02b58 sp=0xbef02b34 pc=0x377c8
runtime.(*mcentral).grow(0x6023b8, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:506 +0x6c fp=0xbef02b7c sp=0xbef02b58 pc=0x26ab0
runtime.(*mcentral).cacheSpan(0x6023b8, 0x11f4c)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:177 +0x448 fp=0xbef02bc4 sp=0xbef02b7c pc=0x267c8
runtime.(*mcache).refill(0x40058088, 0x1e)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcache.go:142 +0x9c fp=0xbef02bd8 sp=0xbef02bc4 pc=0x26104
runtime.(*mcache).nextFree(0x40058088, 0x5fc11e, 0x0, 0x1, 0x70724)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:880 +0x7c fp=0xbef02bf8 sp=0xbef02bd8 pc=0x1bd28
runtime.mallocgc(0xe0, 0x3a6a18, 0x1, 0x1)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1061 +0x7c8 fp=0xbef02c60 sp=0xbef02bf8 pc=0x1c69c
runtime.newobject(0x3a6a18, 0x491fc)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1195 +0x2c fp=0xbef02c74 sp=0xbef02c60 pc=0x1cae8
runtime.malg(0x8000, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:3493 +0x1c fp=0xbef02c94 sp=0xbef02c74 pc=0x4fbe8
runtime.mpreinit(0x5fcf00)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/os_linux.go:340 +0x1c fp=0xbef02ca0 sp=0xbef02c94 pc=0x42330
runtime.mcommoninit(0x5fcf00, 0xffffffff, 0xffffffff)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:663 +0x10c fp=0xbef02cc8 sp=0xbef02ca0 pc=0x49350
runtime.schedinit()
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:565 +0x84 fp=0xbef02cf0 sp=0xbef02cc8 pc=0x48ca8
runtime.rt0_go(0xbef02e2f, 0xbef02e45, 0x0, 0xbef02e4c, 0xbef02e6e, 0xbef02e79, 0xbef02e81, 0xbef02e8c, 0xbef02e9f, 0xbef02eaf, ...)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/asm_arm.s:163 +0x7c fp=0xbef02d30 sp=0xbef02cf0 pc=0x7777c

    Uptime after reboot: 2 days 23 hour(s) 53 minute(s) 29 seconds

    Additional info: using dual-wan with usb-lte and have to "/jffs/nextdns/nextdns restart" everytime wan changes (done via wan-event script) because otherwise dns isn't working anymore, so maybe these restarts lead to that memory problem?

    Also, nothing to do with this bug, but the cli shortcut "nextdns ..." is only working after fresh install/reinstall of nextdns, after a reboot it is gone and you have to use "/jffs/nextdns/nextdns ...". Maybe this can be fixed also.

    kr, chris

    • Chris_At
    • 3 yrs ago
    • Reported - view

    And again after yesterdays reboot to fix it: 

    ASUSWRT-Merlin RT-AC68U 386.2_beta3 Sun Mar 28 17:47:47 UTC 2021
admin@router:/tmp/home/root# /jffs/nextdns/nextdns status
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x3b4dc8, 0x16)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/panic.go:1116 +0x5c fp=0xbec499e8 sp=0xbec499d4 pc=0x44de4
runtime.sysMap(0x800000, 0x400000, 0x610518)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mem_linux.go:169 +0xa8 fp=0xbec49a0c sp=0xbec499e8 pc=0x27014
runtime.(*linearAlloc).alloc(0x6010c0, 0x400000, 0x400000, 0x610518, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1447 +0x94 fp=0xbec49a24 sp=0xbec49a0c pc=0x1d430
runtime.(*mheap).sysAlloc(0x600d68, 0x400000, 0x1, 0x1)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:633 +0x54 fp=0xbec49a80 sp=0xbec49a24 pc=0x1b53c
runtime.(*mheap).grow(0x600d68, 0x1, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1344 +0x6c fp=0xbec49abc sp=0xbec49a80 pc=0x383d0
runtime.(*mheap).allocSpan(0x600d68, 0x1, 0x11e00, 0x610528, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:1160 +0x63c fp=0xbec49b0c sp=0xbec49abc pc=0x381a8
runtime.(*mheap).alloc.func1()
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:907 +0x50 fp=0xbec49b34 sp=0xbec49b0c pc=0x7193c
runtime.(*mheap).alloc(0x600d68, 0x1, 0x11e, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go:901 +0x58 fp=0xbec49b58 sp=0xbec49b34 pc=0x377c8
runtime.(*mcentral).grow(0x6023b8, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:506 +0x6c fp=0xbec49b7c sp=0xbec49b58 pc=0x26ab0
runtime.(*mcentral).cacheSpan(0x6023b8, 0x11f4c)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral.go:177 +0x448 fp=0xbec49bc4 sp=0xbec49b7c pc=0x267c8
runtime.(*mcache).refill(0x40087088, 0x1e)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcache.go:142 +0x9c fp=0xbec49bd8 sp=0xbec49bc4 pc=0x26104
runtime.(*mcache).nextFree(0x40087088, 0x5fc11e, 0x0, 0x1, 0x70724)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:880 +0x7c fp=0xbec49bf8 sp=0xbec49bd8 pc=0x1bd28
runtime.mallocgc(0xe0, 0x3a6a18, 0x1, 0x1)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1061 +0x7c8 fp=0xbec49c60 sp=0xbec49bf8 pc=0x1c69c
runtime.newobject(0x3a6a18, 0x491fc)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.go:1195 +0x2c fp=0xbec49c74 sp=0xbec49c60 pc=0x1cae8
runtime.malg(0x8000, 0x0)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:3493 +0x1c fp=0xbec49c94 sp=0xbec49c74 pc=0x4fbe8
runtime.mpreinit(0x5fcf00)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/os_linux.go:340 +0x1c fp=0xbec49ca0 sp=0xbec49c94 pc=0x42330
runtime.mcommoninit(0x5fcf00, 0xffffffff, 0xffffffff)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:663 +0x10c fp=0xbec49cc8 sp=0xbec49ca0 pc=0x49350
runtime.schedinit()
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:565 +0x84 fp=0xbec49cf0 sp=0xbec49cc8 pc=0x48ca8
runtime.rt0_go(0xbec49e2f, 0xbec49e45, 0x0, 0xbec49e4c, 0xbec49e6e, 0xbec49e79, 0xbec49e81, 0xbec49e8c, 0xbec49e9f, 0xbec49eaf, ...)
        /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/asm_arm.s:163 +0x7c fp=0xbec49d30 sp=0xbec49cf0 pc=0x7777c
    • olivier
    • 3 yrs ago
    • Reported - view

    What size did you set for the cache?

      • Chris_At
      • 3 yrs ago
      • Reported - view

      Olivier Poitrey 10MB, config below (id removed): 

      admin@router:/tmp/home/root# /jffs/nextdns/nextdns config
control /var/run/nextdns.sock
report-client-info true
detect-captive-portals false
hardened-privacy false
listen localhost:53
cache-max-age 0s
max-ttl 5s
bogus-priv true
use-hosts true
auto-activate false
timeout 5s
cache-size 10MB
discovery-dns
setup-router true
log-queries false
      • olivier
      • 3 yrs ago
      • Reported - view

      Chris At try with 1M

      • Chris_At
      • 3 yrs ago
      • Reported - view

      Olivier Poitrey Ok, will try and keep you updated.

      Any idea how to fix the disappearing nextdns command shortcut I mentioned before?

      • olivier
      • 3 yrs ago
      • Reported - view

      Chris At the service is supposed to put it back when it starts: https://github.com/nextdns/nextdns/blob/master/host/service/merlin/service.go#L242

      • Chris_At
      • 3 yrs ago
      • Reported - view

      Olivier Poitrey same with 1M 

      admin@router:/tmp/home/root# nextdns status
      fatal error: runtime: out of memory
       
      


      runtime stack:
      runtime.throw(0x3b4dc8, 0x16)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/panic.go
      :1116 +0x5c fp=0xbe99ba28 sp=0xbe99ba14 pc=0x44de4
      runtime.sysMap(0x800000, 0x400000, 0x610518)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mem_linu
      x.go:169 +0xa8 fp=0xbe99ba4c sp=0xbe99ba28 pc=0x27014
      runtime.(*linearAlloc).alloc(0x6010c0, 0x400000, 0x400000, 0x61
      0518, 0x0)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g
      o:1447 +0x94 fp=0xbe99ba64 sp=0xbe99ba4c pc=0x1d430
      runtime.(*mheap).sysAlloc(0x600d68, 0x400000, 0x1, 0x1)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g
      o:633 +0x54 fp=0xbe99bac0 sp=0xbe99ba64 pc=0x1b53c
      runtime.(*mheap).grow(0x600d68, 0x1, 0x0)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go
      :1344 +0x6c fp=0xbe99bafc sp=0xbe99bac0 pc=0x383d0
      runtime.(*mheap).allocSpan(0x600d68, 0x1, 0x11e00, 0x610528, 0x
      0)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go
      :1160 +0x63c fp=0xbe99bb4c sp=0xbe99bafc pc=0x381a8
      runtime.(*mheap).alloc.func1()
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go
      :907 +0x50 fp=0xbe99bb74 sp=0xbe99bb4c pc=0x7193c
      runtime.(*mheap).alloc(0x600d68, 0x1, 0x11e, 0x0)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mheap.go
      :901 +0x58 fp=0xbe99bb98 sp=0xbe99bb74 pc=0x377c8
      runtime.(*mcentral).grow(0x6023b8, 0x0)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral
      .go:506 +0x6c fp=0xbe99bbbc sp=0xbe99bb98 pc=0x26ab0
      runtime.(*mcentral).cacheSpan(0x6023b8, 0x11f4c)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcentral
      .go:177 +0x448 fp=0xbe99bc04 sp=0xbe99bbbc pc=0x267c8
      runtime.(*mcache).refill(0x400ff088, 0x1e)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/mcache.g
      o:142 +0x9c fp=0xbe99bc18 sp=0xbe99bc04 pc=0x26104
      runtime.(*mcache).nextFree(0x400ff088, 0x5fc11e, 0x0, 0x1, 0x70
      724)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g
      o:880 +0x7c fp=0xbe99bc38 sp=0xbe99bc18 pc=0x1bd28
      runtime.mallocgc(0xe0, 0x3a6a18, 0x1, 0x1)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g
      o:1061 +0x7c8 fp=0xbe99bca0 sp=0xbe99bc38 pc=0x1c69c
      runtime.newobject(0x3a6a18, 0x491fc)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/malloc.g
      o:1195 +0x2c fp=0xbe99bcb4 sp=0xbe99bca0 pc=0x1cae8
      runtime.malg(0x8000, 0x0)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:
      3493 +0x1c fp=0xbe99bcd4 sp=0xbe99bcb4 pc=0x4fbe8
      runtime.mpreinit(0x5fcf00)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/os_linux
      .go:340 +0x1c fp=0xbe99bce0 sp=0xbe99bcd4 pc=0x42330
      runtime.mcommoninit(0x5fcf00, 0xffffffff, 0xffffffff)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:
      663 +0x10c fp=0xbe99bd08 sp=0xbe99bce0 pc=0x49350
      runtime.schedinit()
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/proc.go:
      565 +0x84 fp=0xbe99bd30 sp=0xbe99bd08 pc=0x48ca8
      runtime.rt0_go(0xbe99be6b, 0xbe99be73, 0x0, 0xbe99be7a, 0xbe99b
      e9c, 0xbe99bea7, 0xbe99beaf, 0xbe99beba, 0xbe99becd, 0xbe99bedd
      , ...)
              /opt/hostedtoolcache/go/1.15.1/x64/src/runtime/asm_arm.
      s:163 +0x7c fp=0xbe99bd70 sp=0xbe99bd30 pc=0x7777c
      admin@router:/tmp/home/root#

       
      • Chris_At
      • 3 yrs ago
      • Reported - view

      Olivier Poitrey Addon: I could restart it using "sh /jffs/nextdns/nextdns.init restart". A simple "nextdns restart" brought the same memory failure as above.

      • Chris_At
      • 3 yrs ago
      • Reported - view

      Olivier Poitrey Regarding putting back when it starts it seems not to work every time (after router reboot):

      ASUSWRT-Merlin RT-AC68U 386.2_0 Fri Apr 2 02:49:55 UTC 2021

      admin@router:/tmp/home/root# nextdns status

      -sh: nextdns: not found

      admin@router:/tmp/home/root# /jffs/nextdns/nextdns status

      running

      admin@router:/tmp/home/root#

    • Craig_Heimericks
    • 3 yrs ago
    • Reported - view

    I am having a similar issue. The service just stops running or responding to commands...I'm not seeing any logs indicating what's going on. If I run the debug command, it states:
    INFO: OS: asuswrt-merlin
    INFO: GOARCH: armv7
    INFO: GOOS: linux
    INFO: NEXTDNS_BIN: /jffs/nextdns/nextdns
    INFO: LATEST_RELEASE: 1.11.0
    DEBUG: Start install loop with CURRENT_RELEASE=
    DEBUG: NextDNS is not installed
    i) Install NextDNS
    e) Exit

    If I reinstall, it works again for a few days.

    • brianp9906
    • 3 yrs ago
    • Reported - view

    I'm seeing similar behavior on Asus RT-AC86U model running Merlin.  Router auto reboots weekly and by about 5 days in, DNS drops off and I run /jffs/nextdns/nextdns status and it says "not running".

    I start it up and its fine. I just changed caching to 1M as per above suggestion and will let you know.

    NextDNS version if 1.32.1.  I dont have any memory allocation errors in /jffs/syslog.log  but right when I run the "nextdns status" the syslog.log records that its stopped.

    May  7 16:37:02 dropbear[8366]: Password auth succeeded for '<admin user>' from <IP:src port>
    May  7 16:37:04 nextdns.init: Stopped

    (edit)

    I saw via 'top' that memory was consumed with having VPN-Server w/ IPSEC turned on, so I just turned it off and we'll see how it goes w/ that and 1MB cache.

      • brianp9906
      • 3 yrs ago
      • Reported - view

      Brian P It has not stopped running since the above changes of reducing memory usage on the ASUS Merlin RT-AC86U and then setting NextDNS cache to 1MB.  Been running steady ever since.

Login to reply

Content aside

  • 3 yrs agoLast active
  • 17Replies
  • 420Views
  • 5 Following