3

NextDNS App for Splunk

Hi NextDNS community,

I'm a huge fan of this tool and in my day job I'm a Splunk professional. I've written a series of dashboards for use with NextDNS logs, this is being published to Splunkbase or you can download it from my Git Repo:

https://github.com/jameswintermute/NextDNS_APP

It is unofficial and a community built app, any suggestions or feedback gratefully received. This is fully compatible with a free Splunk licence and no purchase of any kind necessary.

11 replies

null
    • James_W
    • 1 yr ago
    • Reported - view

    The App has now been approved by Splunk and is publically listed on the Splunkbase website:

    https://splunkbase.splunk.com/apps?keyword=NextDNS

    If anybody at NextDNS would like to get in touch with me I'd be pleased to update and improve it and collaborate with you. NextDNS is great and I want more people to know about it and the logging output to improve!

    • Robert.1
    • 1 yr ago
    • Reported - view

    Hi.

    I really love these applications but I do have a big problem and I have been struggling with this.
    Every time I do a curl command (the same you provided) I do get duplicates in my index. So lets say I have 1000 DNS queries and I run the curl command via crontab 10 times I suddenly have 10000 or so search results. 

    I have been experimenting with crcSalt, initCrcLength and a lot of other things but still everything is duplicated. Think I have depleted internet of answers.. 

    Also I am using the example input Stanza from the TA app. 

    Can you please advise? 

    Br Robert 

    • Robert.1
    • 1 yr ago
    • Reported - view

    I found a wa for this issue so everything works fine now but this was not fixed on Splunk side.

    I have 3 cronjobs that are handling this instead.

    55 0-23/1 * * * cat foo.log > bar.log

    0 0-23/1 * * * curl -X GET -H "X-Api-Key: <value>" -s -L https://api.nextdns.io/profiles/xxxxxx/logs/download > /var/log/nextdns/foo.log

    5 0-23/1 * * * grep -xvFf bar.log foo.log > nextdns.log

    Basically what this does is creating a copy of the downloaded file "bar.log" then the Nextdns log is being curled (foo.log)

    last step is a comparison between these 2 files and the new lines will be sent to nextdns.log that the TA should monitor. This way only new log entries will be sent to Splunk Index. 

    Now all duplicate issues are gone. 

    If someone else has a good fix for this please let me know. :)

     

    Br Robert
     

      • James_W
      • 1 yr ago
      • Reported - view

      Thank you very much for contributing to this discussion. I saw you reply a week or two ago but we just had a new baby. Anyway...

      You are right, the log rotate and duplication issue is annoying and I fear makes NextDNS Prosumer rather than fully Professional grade; I'd love to work with your team @NextDNS!

      I will recreate your script and update GIT / Splunkbase

      • NextDNs
      • 1 yr ago
      • Reported - view

      have you tried the streaming api? https://nextdns.github.io/api/#streaming

      • James_W
      • 1 yr ago
      • Reported - view

      Thanks for getting in touch. Yes, I have tried Streaming, however the log format is entirely different. Whilst I can convert that in Splunk using props and transforms it is not ideal. Also you would need to setup a systemctl service to start the streaming service and if the platform restarts then the possibility for missing logs is likely.

      Would the NextDNS team be open to having a discussion at some point regarding logging concerns? I am a certified Splunk Consultant and if we could just make it that bit more reliable it would be amazing and good for your product

      • Robert.1
      • 1 yr ago
      • Reported - view

       

      Congrats to the newborn!

      • NextDNs
      • 1 yr ago
      • Reported - view

       SSE is meant to solve that. Each event as an id, if you pass that id to the next streaming request as a parameter, it resumes the stream at that id.

      Regarding the log format, can you please give us more info on what is not ideal and how it could be improved?

      • James_W
      • 8 mths ago
      • Reported - view

       

      Hi, I would like to resurrect this thread, is it possible have a chat with one of your team? As before I'm a certified Splunk professional and I've written an app and TA for use on SplunkBase for NextDNS, the problem myself and others still have is unique ID ref so that we do not get duplicate events in the index between pulls. Can you make time to speak with me? I'm based in the UK and generally free to meet 0900-1700 during the week

      • NextDNs
      • 8 mths ago
      • Reported - view

       could you please help me understand how SSE event ids aren’t solving your problem?

      • Robert.1
      • 5 mths ago
      • Reported - view

      Hi, have you had any time to work further on this?

Login to reply

Content aside

  • 3 Likes
  • 5 mths agoLast active
  • 11Replies
  • 280Views
  • 3 Following