HaGeZi Ultimate - TIF list and whitelist python script

Hey everyone, with the assistance of AI I have put together a python script that will import a custom denylist and whitelist that basically imports the missing 800k+ TIF entries in HaGeZi Ultimate. Attached is the files that inlude a comprehensive whitelist, and the missing tif domain file. These were both deduplicated against the existing domains. So no overlaps. Enjoy. You need all files in order to run the script. They must also all be in the same target folder. 

Instructions.

Overview This guide explains how to install Python, set up the environment, and use the provided script to upload domains to your NextDNS Denylist and Allowlist. The script adds domains from missing_tif_domains.txt to the Denylist and comprehensive_whitelist.txt to the Allowlist, enhancing your NextDNS configuration with additional Threat Intelligence Feeds (TIF) and a whitelist for functionality.

Prerequisites

• A NextDNS account with a Config ID (found in your NextDNS dashboard under "Setup").

• A NextDNS API key (generated in your account settings under "API").

• Two text files: missing_tif_domains.txt (domains to block) and comprehensive_whitelist.txt (domains to allow), each with one domain per line.

Step 1: Install Python Python is required to run the script. Follow these steps based on your operating system:

Windows

1. Download Python:

   • Go to python.org/downloads

   • Click "Download Python 3.x.x" (latest version, e.g., 3.11.6 as of Feb 2025).

2. Run the Installer:

   • Open the downloaded .exe file.

   • Check "Add Python to PATH" at the bottom of the installer window.

   • Click "Install Now" and wait for completion.

3. Verify Installation:

   • Open Command Prompt (Win + R, type cmd, press Enter).

   • Type python --version or python3 --version. You should see something like Python 3.11.6.

   • If it fails, try restarting your computer or reinstalling with the PATH option checked.

macOS

1. Check if Python is Installed:

   • Open Terminal (Applications > Utilities > Terminal).

   • Type python3 --version. If it shows a version (e.g., 3.9.x), skip to Step 3.

2. Install Python:

   • Download from python.org/downloads or use Homebrew:

     • Install Homebrew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

     • Install Python: brew install python

3. Verify Installation:

   • In Terminal, type python3 --version. Confirm it works.

Linux (Ubuntu/Debian)

1. Update Package List:

   • Open Terminal.

   • Run sudo apt update

2. Install Python:

   • Run sudo apt install python3 python3-pip

3. Verify Installation:

   • Type python3 --version. You should see the version number.

Step 2: Install Required Library The script uses the requests library for API calls.

1. Open your terminal/command prompt.

2. Run: pip install requests

   • On some systems, use pip3 install requests

3. Verify: Run python -c "import requests" (or python3). No error means it’s installed.

Step 3: Prepare Your Files

1. Get Your NextDNS Credentials:

   • Log into my.nextdns.io

   • Find your Config ID under "Setup" (e.g., 1e74f1).

   • Generate an API Key under "Account" > "API" (e.g., a long string like b5bd23...).

2. Edit the Script:

   • Open the script in a text editor (e.g., Notepad, VS Code).

   • Replace "YOUR_API_KEY_HERE" with your API key.

   • Replace "YOUR_CONFIG_ID_HERE" with your Config ID.

   • Save the file as nextdns_update.py

3. Prepare Domain Files:

   • Create missing_tif_domains.txt with domains to block (one per line).

   • Create comprehensive_whitelist.txt with domains to allow (one per line).

   • Place both files in the same directory as nextdns_update.py

Step 4: Run the Script

1. Navigate to Directory:

   • Open your terminal/command prompt.

   • Change to the script’s directory:

     • Windows: cd path\to\directory (e.g., cd C:\Users\You\Documents\NextDNS)

     • macOS/Linux: cd /path/to/directory (e.g., cd ~/Documents/NextDNS)

2. Execute the Script:

   • Run: python nextdns_update.py

     • On macOS/Linux, you might need python3 nextdns_update.py

3. Monitor Output:

   • You’ll see progress like: Starting NextDNS update script... Testing API connectivity... API connectivity verified Loaded progress: 0 Denylist, 0 Allowlist entries Reading file: comprehensive_whitelist.txt Read X domains from comprehensive_whitelist.txt Reading file: missing_tif_domains.txt Read Y domains from missing_tif_domains.txt Starting Allowlist update... Adding domain1.com to Allowlist (1/X, ~0h remaining)... ... Starting Denylist update... Adding domainA.com to Denylist (1/Y, ~Zh remaining)... ... Allowlist: X added, 0 failed Denylist: Y added, 0 failed

   • Check nextdns_update.log for detailed logs if issues arise.

Step 5: Verify Results

1. Check NextDNS Dashboard:

   • Log into my.nextdns.io

   • Go to your Config ID’s "Denylist" and "Allowlist" tabs.

   • Confirm the domains from your files appear.

2. Test Functionality:

   • Use your internet to ensure blocked domains are stopped and whitelisted ones work.

Troubleshooting

• No Output: Ensure Python and requests are installed. Check file paths.

• API Errors:

  • 401 Unauthorized: Wrong API key or Config ID.

  • 429 Too Many Requests: Script auto-retries after 60 seconds; if persistent, increase REQUEST_DELAY.

• File Errors: Verify missing_tif_domains.txt and comprehensive_whitelist.txt exist and aren’t empty.

• Progress Resumption: If interrupted, nextdns_progress.json tracks completed domains—delete it to start fresh.

Notes

• Runtime: At 1 second per domain, 273,000 TIF domains take 76 hours (3 days), plus whitelist time. Retries may extend this.

• Customization: Adjust REQUEST_DELAY (e.g., 0.75) if you test and find it stable, but 1.0 is safest.

This script and guide should work for anyone with a NextDNS account—enjoy your enhanced blocking!