pfsense with 2 WAN (2 public IPs) loadbalanced

Hi,

I am trying to search and though this appeared on a few post, I still can't find the solution.

Setup:
pfsense gateway with loadbalanced 2 WAN connection

(IP are for reference)

ISP 1 has IP 9.9.9.9

ISP 2 has IP 5.5.5.5

To activate loadbalancing on pfsense, the two WAN connections are both on TIER 1.

With that setup, connections from clients are distributed between the two WAN connection.

If I setup 2 networks on nextdns for each of my ISP and linking the IP to it, the nextdns IP for the two network are different.

Issue:
If I use only one set from the 2 network, (e.g. nextdns IP for ISP 1), when the connection is routed to ISP 2, filtering doesn't work.

Sam goes of course if I reverse the setup.

Question:
How can I make this work that I would only need one network on nextdns for the two different WAN connections I have.

Additional info:
ISP 1 supports IPv6
ISP 2 does not support IPv6

Thanks,