OpenWrt Forum Archive

Topic: mwan3; multi-wan policy routing (general topic)

The content of this topic has been archived between 22 May 2013 and 6 May 2018. Unfortunately there are posts – most likely complete pages – missing.

Thomymaster wrote:

OK i havent tried this yet, what i am concerned about is if there is an error in the package (look at the error message above).


What i forget to say is that the error appeared after upgrading from 12.01rc1 to 12.09 release

I don't see how there could be an error in the package if it worked fine for me.

I usually just SCP files to my router. The first router my PC connects to gets my working files via SCP. Currently the same files on the github as I haven't done any work since the last commit.

(Last edited by arfett on 5 May 2013, 19:56)

hi all. I try increase speed, weight is balancing but speed is not increase. when i put 2 usb modem 3G, it run as once modem. I download by IDM with 16 piece, it have many connect

I'm getting more and more signals that load-balancing does not work correctly with multiple 3G devices. As i don't have them, i can't test it.

Maybe there is someone with this setup, which i can use to test and maybe fix the issue?

Adze, the luci page is 100% done. I'm not changing anything else unless someone has a great idea for a new feature or you add something to mwan3. Maybe the first post should reflect that.

Hello,

The following issue jumped when I wanted to restart mwan3:

/etc/init.d/mwan3 restart
iptables v1.4.10: can't initialize iptables table `mangle': Bad file descriptor
Perhaps iptables or your kernel needs to be upgraded.
iptables v1.4.10: Couldn't load target `mwan3_rules':File not found

Try `iptables -h' or 'iptables --help' for more information.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.
iptables: No chain/target/match by that name.

And I think there are some other "holes". With previous versions I had "doubled" download speeds using torrents, but now the max speed never exeeds the fastest wan connection. If it is necessary, I can provide any additional info.

(Last edited by headless.cross on 7 May 2013, 23:22)

Hi... any mwan3 packages for barrier breaker ? i cannot find it on repo..

headless.cross wrote:

And I think there are some other "holes". With previous versions I had "doubled" download speeds using torrents, but now the max speed never exeeds the fastest wan connection. If it is necessary, I can provide any additional info.

Yes please! I bet that you don't have mangle tables module correctly installed..

Adze wrote:

Yes please! I bet that you don't have mangle tables module correctly installed..

I think you're right. Very strange though. I'm running ATTITUDE ADJUSTMENT (12.09, r36088). I have removed the iptables-mod-ipopt with -force-depends and reinstalled it through LuCI (for unknown reasons I couldn't install it from cli). After that the "/etc/init.d/mwan3 restart" went smooth and fast (the previous behaviour was kinda strange, no errors during restart.. but it was very slow and never ended, I had to run "ctrl-c" to stop it somehow). This afternoon I will make some speed tests and report back.

After many tests and different configuration combinations, I couldn't archive high speeds on torrents.
Shall I post my configs to gain any help?

(Last edited by headless.cross on 8 May 2013, 21:11)

headless.cross wrote:

After many tests and different configuration combinations, I couldn't archive high speeds on torrents.
Shall I post my configs to gain any help?

Just FYI the latest troubleshooting page has everything on one auto-refreshing html based page so you can copy paste all relevant information into a single forum codeblock all at once.

arfett wrote:

Just FYI the latest troubleshooting page has everything on one auto-refreshing html based page so you can copy paste all relevant information into a single forum codeblock all at once.

I have the latest luci-app-mwan3 binary (thank you for your effort) and my question was more... rhetorical.
Here comes the debug data:

Output of "ip route show:

default via 127.0.0.1 dev lo  src 10.0.2.1  metric 5 
default via 10.0.0.1 dev eth0.2  proto static  metric 10 
default via 192.168.0.1 dev wlan0  proto static  metric 20 
8.23.224.0/24 via 192.168.0.1 dev wlan0  proto static  metric 2 
10.0.0.0/29 dev eth0.2  proto static  scope link  metric 10 
10.0.2.0/28 dev eth0.1  proto kernel  scope link  src 10.0.2.1 
192.168.0.0/24 dev wlan0  proto static  scope link  metric 20 
192.168.1.0/24 via 10.0.0.1 dev eth0.2  proto static  metric 2

Output of "ip rule show":

0:    from all lookup local 
1001:    from all fwmark 0x100/0xff00 lookup 1001 
1002:    from all fwmark 0x200/0xff00 lookup 1002 
1016:    from all fwmark 0x1000/0xff00 lookup 1016 
1017:    from all fwmark 0x1100/0xff00 lookup 1017 
1018:    from all fwmark 0x1200/0xff00 lookup 1018 
1019:    from all fwmark 0x1300/0xff00 lookup 1019 
1020:    from all fwmark 0x1400/0xff00 lookup 1020 
32766:    from all lookup main 
32767:    from all lookup default

Output of "ip route list table 1001-1027":

1001
default via 10.0.0.1 dev eth0.2 

1002
default via 192.168.0.1 dev wlan0 

1016
default via 10.0.0.1 dev eth0.2  metric 1 

1017
default via 192.168.0.1 dev wlan0  metric 1 

1018
default  metric 1 
    nexthop via 192.168.0.1  dev wlan0 weight 3
    nexthop via 10.0.0.1  dev eth0.2 weight 4

1019
default via 10.0.0.1 dev eth0.2  metric 1 
default via 192.168.0.1 dev wlan0  metric 2 

1020
default via 192.168.0.1 dev wlan0  metric 1 
default via 10.0.0.1 dev eth0.2  metric 2

Output of "iptables -L -t mangle -v -n":

Chain PREROUTING (policy ACCEPT 744K packets, 454M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 745K  454M mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 29209 packets, 2629K bytes)
 pkts bytes target     prot opt in     out     source               destination         
29877 2684K mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 714K packets, 452M bytes)
 pkts bytes target     prot opt in     out     source               destination         
1104K  691M zone_wwan_MSSFIX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
1104K  691M zone_wan_MSSFIX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 30316 packets, 3161K bytes)
 pkts bytes target     prot opt in     out     source               destination         
31716 3262K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 744K packets, 455M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 746K  455M mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_default (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  126 19273 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3         mark match !0x8000/0x8000 MARK or 0x8000 
 5917  484K MARK       all  --  *      *       0.0.0.0/0            10.0.0.0/29         mark match !0x8000/0x8000 MARK or 0x8000 
 4650  931K MARK       all  --  *      *       0.0.0.0/0            10.0.2.0/28         mark match !0x8000/0x8000 MARK or 0x8000 
 3426  287K MARK       all  --  *      *       0.0.0.0/0            192.168.0.0/24      mark match !0x8000/0x8000 MARK or 0x8000 

Chain mwan3_post (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 239K   35M MARK       all  --  *      wlan0   0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 MARK xset 0x200/0xff00 
 147K   65M MARK       all  --  *      eth0.2  0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 MARK xset 0x100/0xff00 
 386K  357M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x8000/0x8000 MARK and 0xffff7fff 
 776K  457M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK save mask 0xff00 

Chain mwan3_pre (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 776K  458M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore mask 0xff00 
 235K  251M MARK       all  --  wlan0  *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8200/0xff00 
 137K  105M MARK       all  --  eth0.2 *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8100/0xff00 
 404K  102M mwan3_default  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 
28666 2416K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   95  5700 MARK       all  --  *      *       0.0.0.0/0            8.23.224.0/24       mark match 0x0/0xff00 MARK xset 0x8000/0xff00 
   18  1023 MARK       all  --  *      *       0.0.0.0/0            192.168.1.0/24      mark match 0x0/0xff00 MARK xset 0x8000/0xff00 
  237 11376 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport sports 0:65535 multiport dports 443,8443 mark match 0x0/0xff00 MARK xset 0x1100/0xff00 
27145 2324K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 MARK xset 0x1200/0xff00 

Chain zone_wan_MSSFIX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 8772  421K TCPMSS     tcp  --  *      eth0.2  0.0.0.0/0            0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU 

Chain zone_wwan_MSSFIX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 5006  240K TCPMSS     tcp  --  *      wlan0   0.0.0.0/0            0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU 

Output of "ifconfig":

eth0      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:762729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:760084 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:307356163 (293.1 MiB)  TX bytes:642078082 (612.3 MiB)
          Interrupt:4 

eth0.1    Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx  
          inet addr:10.0.2.1  Bcast:10.0.2.15  Mask:255.255.255.240
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:565316 errors:0 dropped:0 overruns:0 frame:0
          TX packets:540242 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:134434136 (128.2 MiB)  TX bytes:559124059 (533.2 MiB)

eth0.2    Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx  
          inet addr:10.0.0.2  Bcast:10.0.0.7  Mask:255.255.255.248
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:197127 errors:0 dropped:0 overruns:0 frame:0
          TX packets:219841 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:159169985 (151.7 MiB)  TX bytes:79913575 (76.2 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:59679 errors:0 dropped:0 overruns:0 frame:0
          TX packets:59679 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:10641731 (10.1 MiB)  TX bytes:10641731 (10.1 MiB)

wlan0     Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx  
          inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:369606 errors:0 dropped:0 overruns:0 frame:0
          TX packets:376378 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:399068791 (380.5 MiB)  TX bytes:72206179 (68.8 MiB)

Output of "cat /etc/config/mwan3":

config rule 'noip_updater'
    option use_policy 'default'
    option dest_ip '8.23.224.0/24'

config rule 'router'
    option use_policy 'default'
    option dest_ip '192.168.1.0/24'

config rule 'https'
    option proto 'tcp'
    option use_policy 'wwan_only'
    option dest_port '443,8443'
    option dest_ip '0.0.0.0/0'

config rule 'internet'
    option use_policy 'wan_wwan_loadbalanced'
    option proto 'all'
    option 'equalize' '1'
    option dest_ip '0.0.0.0/0'

config interface 'wan'
    option enabled '1'
    list track_ip '10.0.0.1'
    list track_ip 'xx.xx.xx.1'
    list track_ip 'xx.xx.xx.2'
    option timeout '5'
    option down '5'
    option up '3'
    option reliability '3'
    option count '3'
    option interval '30'

config interface 'wwan'
    option enabled '1'
    list track_ip '192.168.0.1'
    list track_ip 'xx.xx.xx.1'
    list track_ip 'xx.xx.xx.2'
    option timeout '5'
    option down '5'
    option up '3'
    option reliability '3'
    option count '3'
    option interval '30'

config member 'wan_m1_w4'
    option interface 'wan'
    option metric '1'
    option weight '4'

config member 'wan_m2_w4'
    option interface 'wan'
    option metric '2'
    option weight '4'

config member 'wwan_m1_w3'
    option interface 'wwan'
    option metric '1'
    option weight '3'

config member 'wwan_m2_w3'
    option interface 'wwan'
    option metric '2'
    option weight '3'

config policy 'wan_only'
    list use_member 'wan_m1_w4'

config policy 'wwan_only'
    list use_member 'wwan_m1_w3'

config policy 'wan_wwan_loadbalanced'
    list use_member 'wan_m1_w4'
    list use_member 'wwan_m1_w3'

config policy 'wan_pri_wwan_sec'
    list use_member 'wan_m1_w4'
    list use_member 'wwan_m2_w3'

config policy 'wwan_pri_wan_sec'
    list use_member 'wan_m2_w4'
    list use_member 'wwan_m1_w3'

Output of "cat /etc/config/network":

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'lan'
    option ifname 'eth0.1'
    option proto 'static'
    option ipaddr '10.0.2.1'
    option netmask '255.255.255.240'

config interface 'wan'
    option ifname 'eth0.2'
    option proto 'static'
    option ipaddr '10.0.0.2'
    option dns 'xx.xx.xx.1 xx.xx.xx.2'
    option gateway '10.0.0.1'
    option netmask '255.255.255.248'
    option broadcast '10.0.0.7'
    option metric '10'

config switch
    option name 'eth0'
    option reset '1'
    option enable_vlan '1'

config switch_vlan
    option device 'eth0'
    option vlan '1'
    option ports '0t 2 3 4 5'

config switch_vlan
    option device 'eth0'
    option vlan '2'
    option ports '0t 1'

config interface 'wwan'
    option proto 'static'
    option ipaddr '192.168.0.2'
    option netmask '255.255.255.0'
    option gateway '192.168.0.1'
    option broadcast '192.168.0.255'
    option dns 'xx.xx.xx.1 xx.xx.xx.2'
    option metric '20'

config route
    option interface 'wwan'
    option netmask '255.255.255.0'
    option gateway '192.168.0.1'
    option target '8.23.224.0'
    option metric '2'

config route
    option interface 'wan'
    option target '192.168.1.0'
    option netmask '255.255.255.0'
    option gateway '10.0.0.1'
    option metric '2'

(Last edited by headless.cross on 9 May 2013, 12:44)

headless.cross wrote:

I have the latest luci-app-mwan3 binary

Still not the latest files. I remade the troubleshooting page so it's possible to copy everything at once as to keep single posts from eating up 3 solid pages by having everything in a single code block.

arfett wrote:

Still not the latest files. I remade the troubleshooting page so it's possible to copy everything at once as to keep single posts from eating up 3 solid pages by having everything in a single code block.

I have edited the previous post instead of posting the new data, because for the moment I have no access to my router. Thanks in advance.

Edit: Indeed I haven't the latest binaries. My apologies.

(Last edited by headless.cross on 9 May 2013, 12:50)

headless.cross wrote:
arfett wrote:

Still not the latest files. I remade the troubleshooting page so it's possible to copy everything at once as to keep single posts from eating up 3 solid pages by having everything in a single code block.

I have edited the previous post instead of posting the new data, because for the moment I have no access to my router. Thanks in advance.

Edit: Indeed I haven't the latest binaries. My apologies.

No worries. Not a big deal really I just realized after people were putting their troubleshooting dumps into this thread it was taking up insane amounts of space lol.

I don't see any glaring issues with the config. Maybe Adze will spot something.

(Last edited by arfett on 9 May 2013, 22:55)

Hi headless.cross


I have checked and checked... your config looks fine and your output looks fine. But then i spotted something strange.

Although you defined "option 'equalize' '1'" on your last rule, the mwan3 script did not pick up on this. I don't know why?!?

Could you please try it without the qoutes around equalize please? Like:

option equalize '1'

Thnx!

Adze wrote:

Could you please try it without the qoutes around equalize please?

Of course, here's the output:

Firewall default output policy (must be ACCEPT) :

ACCEPT

Output of "ip route show" :

default via 127.0.0.1 dev lo  src 10.0.2.1  metric 5 
default via 10.0.0.1 dev eth0.2  proto static  metric 10 
default via 192.168.0.1 dev wlan0  proto static  metric 20 
8.23.224.0/24 via 192.168.0.1 dev wlan0  proto static  metric 2 
10.0.0.0/29 dev eth0.2  proto static  scope link  metric 10 
10.0.2.0/28 dev eth0.1  proto kernel  scope link  src 10.0.2.1 
192.168.0.0/24 dev wlan0  proto static  scope link  metric 20 
192.168.1.0/24 via 10.0.0.1 dev eth0.2  proto static  metric 2 

Output of "ip rule show" :

0:    from all lookup local 
1001:    from all fwmark 0x100/0xff00 lookup 1001 
1002:    from all fwmark 0x200/0xff00 lookup 1002 
1016:    from all fwmark 0x1000/0xff00 lookup 1016 
1017:    from all fwmark 0x1100/0xff00 lookup 1017 
1018:    from all fwmark 0x1200/0xff00 lookup 1018 
1019:    from all fwmark 0x1300/0xff00 lookup 1019 
1020:    from all fwmark 0x1400/0xff00 lookup 1020 
32766:    from all lookup main 
32767:    from all lookup default 

Output of "ip route list table 1001-1027" :

1001
default via 10.0.0.1 dev eth0.2 
1002
default via 192.168.0.1 dev wlan0 
1016
default via 10.0.0.1 dev eth0.2  metric 1 
1017
default via 192.168.0.1 dev wlan0  metric 1 
1018
default  metric 1 
    nexthop via 192.168.0.1  dev wlan0 weight 3
    nexthop via 10.0.0.1  dev eth0.2 weight 4
1019
default via 10.0.0.1 dev eth0.2  metric 1 
default via 192.168.0.1 dev wlan0  metric 2 
1020
default via 192.168.0.1 dev wlan0  metric 1 
default via 10.0.0.1 dev eth0.2  metric 2 

Output of "iptables -L -t mangle -v -n | awk '/mwan3/' RS=" :

Chain PREROUTING (policy ACCEPT 444K packets, 270M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 449K  270M mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 44883 packets, 4058K bytes)
 pkts bytes target     prot opt in     out     source               destination         
50221 4403K mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 48777 packets, 5250K bytes)
 pkts bytes target     prot opt in     out     source               destination         
54995 5648K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 447K packets, 271M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 453K  271M mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_default (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   73 11029 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3         mark match !0x8000/0x8000 MARK or 0x8000 
15803 1565K MARK       all  --  *      *       0.0.0.0/0            10.0.0.0/29         mark match !0x8000/0x8000 MARK or 0x8000 
10382 1961K MARK       all  --  *      *       0.0.0.0/0            10.0.2.0/28         mark match !0x8000/0x8000 MARK or 0x8000 
 4781  402K MARK       all  --  *      *       0.0.0.0/0            192.168.0.0/24      mark match !0x8000/0x8000 MARK or 0x8000 

Chain mwan3_post (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 119K   12M MARK       all  --  *      wlan0   0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 MARK xset 0x200/0xff00 
77900 7360K MARK       all  --  *      eth0.2  0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 MARK xset 0x100/0xff00 
 292K  255M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x8000/0x8000 MARK and 0xffff7fff 
 504K  276M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK save mask 0xff00 

Chain mwan3_pre (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 504K  276M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore mask 0xff00 
 158K  165M MARK       all  --  wlan0  *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8200/0xff00 
 103K   87M MARK       all  --  eth0.2 *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8100/0xff00 
 243K   24M mwan3_default  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 
39275 2908K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  137  8220 MARK       all  --  *      *       0.0.0.0/0            8.23.224.0/24       mark match 0x0/0xff00 MARK xset 0x8000/0xff00 
   48  2739 MARK       all  --  *      *       0.0.0.0/0            192.168.1.0/24      mark match 0x0/0xff00 MARK xset 0x8000/0xff00 
  540 26408 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport sports 0:65535 multiport dports 443,8443 mark match 0x0/0xff00 MARK xset 0x1100/0xff00 
15849 1246K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 statistic mode random probability 0.571000 MARK xset 0x100/0xff00 
12061  944K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 statistic mode random probability 1.000000 MARK xset 0x200/0xff00 

Output of "ifconfig" :

eth0      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:321022 errors:0 dropped:0 overruns:0 frame:0
          TX packets:367383 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:120688215 (115.0 MiB)  TX bytes:287293217 (273.9 MiB)
          Interrupt:4 

eth0.1    Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:10.0.2.1  Bcast:10.0.2.15  Mask:255.255.255.240
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:202727 errors:0 dropped:0 overruns:0 frame:0
          TX packets:258858 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:21730067 (20.7 MiB)  TX bytes:274385648 (261.6 MiB)

eth0.2    Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:10.0.0.2  Bcast:10.0.0.7  Mask:255.255.255.248
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:118283 errors:0 dropped:0 overruns:0 frame:0
          TX packets:108525 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:93178043 (88.8 MiB)  TX bytes:11438037 (10.9 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:12427 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12427 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:894589 (873.6 KiB)  TX bytes:894589 (873.6 KiB)

wlan0     Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:184056 errors:0 dropped:0 overruns:0 frame:0
          TX packets:147176 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:176377814 (168.2 MiB)  TX bytes:20117641 (19.1 MiB)

Output of "cat /etc/config/mwan3" :

config rule 'noip_updater'
    option use_policy 'default'
    option dest_ip '8.23.224.0/24'

config rule 'zte'
    option use_policy 'default'
    option dest_ip '192.168.1.0/24'

config rule 'https'
    option proto 'tcp'
    option use_policy 'wwan_only'
    option dest_port '443,8443'
    option dest_ip '0.0.0.0/0'

config rule 'internet'
    option use_policy 'wan_wwan_loadbalanced'
    option proto 'all'
    option dest_ip '0.0.0.0/0'
    option equalize '1'

config interface 'wan'
    option enabled '1'
    list track_ip '10.0.0.1'
    list track_ip 'xx.xx.xx.1'
    list track_ip 'xx.xx.xx.2'
    option timeout '5'
    option down '5'
    option up '3'
    option reliability '3'
    option count '3'
    option interval '30'

config interface 'wwan'
    option enabled '1'
    list track_ip '192.168.0.1'
    list track_ip 'xx.xx.xx.1'
    list track_ip 'xx.xx.xx.2'
    option timeout '5'
    option down '5'
    option up '3'
    option reliability '3'
    option count '3'
    option interval '30'

config member 'wan_m1_w4'
    option interface 'wan'
    option metric '1'
    option weight '4'

config member 'wan_m2_w4'
    option interface 'wan'
    option metric '2'
    option weight '4'

config member 'wwan_m1_w3'
    option interface 'wwan'
    option metric '1'
    option weight '3'

config member 'wwan_m2_w3'
    option interface 'wwan'
    option metric '2'
    option weight '3'

config policy 'wan_only'
    list use_member 'wan_m1_w4'

config policy 'wwan_only'
    list use_member 'wwan_m1_w3'

config policy 'wan_wwan_loadbalanced'
    list use_member 'wan_m1_w4'
    list use_member 'wwan_m1_w3'

config policy 'wan_pri_wwan_sec'
    list use_member 'wan_m1_w4'
    list use_member 'wwan_m2_w3'

config policy 'wwan_pri_wan_sec'
    list use_member 'wan_m2_w4'
    list use_member 'wwan_m1_w3'

Output of "cat /etc/config/network" :

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'lan'
    option ifname 'eth0.1'
    option proto 'static'
    option ipaddr '10.0.2.1'
    option netmask '255.255.255.240'

config interface 'wan'
    option ifname 'eth0.2'
    option proto 'static'
    option ipaddr '10.0.0.2'
    option dns 'xx.xx.xx.1 xx.xx.xx.2'
    option gateway '10.0.0.1'
    option netmask '255.255.255.248'
    option broadcast '10.0.0.7'
    option metric '10'

config switch
    option name 'eth0'
    option reset '1'
    option enable_vlan '1'

config switch_vlan
    option device 'eth0'
    option vlan '1'
    option ports '0t 2 3 4 5'

config switch_vlan
    option device 'eth0'
    option vlan '2'
    option ports '0t 1'

config interface 'wwan'
    option proto 'static'
    option ipaddr '192.168.0.2'
    option netmask '255.255.255.0'
    option gateway '192.168.0.1'
    option broadcast '192.168.0.255'
    option dns 'xx.xx.xx.1 xx.xx.xx.2'
    option metric '20'

config route
    option interface 'wwan'
    option netmask '255.255.255.0'
    option gateway '192.168.0.1'
    option target '8.23.224.0'
    option metric '2'

config route
    option interface 'wan'
    option target '192.168.1.0'
    option netmask '255.255.255.0'
    option gateway '10.0.0.1'
    option metric '2'

Some hints:
1) When I check my external IP, e.g. on cmyip.com, the IP is changing fine.
2) The "traceroute" from the router itself works great, it uses different gateways

root@dw:~# traceroute google.com
traceroute to google.com (173.194.34.96), 30 hops max, 38 byte packets
 1  10.0.0.1 (10.0.0.1)  0.732 ms  0.542 ms  0.629 ms
root@dw:~# traceroute google.com
traceroute to google.com (173.194.34.110), 30 hops max, 38 byte packets
 1  192.168.0.1 (192.168.0.1)  2.227 ms  10.0.0.1 (10.0.0.1)  0.741 ms  192.168.0.1 (192.168.0.1)  1.960 ms

Some issues:
1) As you can see, I have a static route to no-ip, so the ddns script request will travel through the wwan interface. The "traceroute" from the router itself routes the request correctly all the time BUT my no-ip host often assigns the ip address from the wan interface (unwanted behaviour). That's really odd and it's driving me crazy. <---- issue solved
The "Source of IP address" was set to URL and by misconfigure the provided URL was wrong, so the ddns script had a last resort url set (hardcoded) to checkip.dyndns.org and my external ip was changing all the time.

2) I can't still archive "doubled" speeds using torrents (for this issue, I think I have a clue what is going on, but I have to make some tests)

Thanks in advance.

(Last edited by headless.cross on 11 May 2013, 00:46)

This is fixed but not deleted!

Hi all!!

At first let me say, this is the best extension idea i have seen over the years for openwrt!
This way i can have multiple vpn connections and fallback / loadballance them! Great!

I have a problem setting it up right and need a little push in the right direction.

When i set the default policy to e1_only, which is a pptp Connection, i cant ping anything:

root@Openwrt:~# ping -c3 -w 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

root@Openwrt:~# ping -c3 -w 3 8.8.8.8 -I eth0.2   #Cableconnection
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=49 time=29.975 ms
64 bytes from 8.8.8.8: seq=1 ttl=49 time=26.910 ms
64 bytes from 8.8.8.8: seq=2 ttl=49 time=27.646 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 26.910/28.177/29.975 ms

root@Openwrt:~# ping -c3 -w 3 8.8.8.8 -I pptp-e1   #pptp Vpn
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=49 time=96.570 ms
64 bytes from 8.8.8.8: seq=1 ttl=49 time=92.487 ms
64 bytes from 8.8.8.8: seq=2 ttl=49 time=93.068 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 92.487/94.041/96.570 ms

This is what my Debug shows:   

cant find anything wrong here!! But i am fairly sure you will wink

MWAN3 Multi-WAN troubleshooting

Firewall default output policy (must be ACCEPT) :

ACCEPT

Output of "ip route show" :

default via 77.64.166.1 dev eth0.2  proto static  metric 5 
default via 172.16.36.1 dev pptp-e1  proto static  metric 10 
77.64.166.0/24 dev eth0.2  proto static  scope link  metric 5 
93.115.84.194 via 77.64.166.1 dev eth0.2  proto static  metric 5 
172.16.36.1 dev pptp-e1  proto kernel  scope link  src 172.16.36.103 
192.168.1.0/24 via 192.168.3.2 dev br-lan  proto static 
192.168.2.0/24 via 192.168.3.2 dev br-lan  proto static 
192.168.3.0/24 dev br-lan  proto kernel  scope link  src 192.168.3.1 

Output of "ip rule show" :

0:    from all lookup local 
1001:    from all fwmark 0x100/0xff00 lookup 1001 
1002:    from all fwmark 0x200/0xff00 lookup 1002 
1016:    from all fwmark 0x1000/0xff00 lookup 1016 
1017:    from all fwmark 0x1100/0xff00 lookup 1017 
32766:    from all lookup main 
32767:    from all lookup default 

Output of "ip route list table 1001-1027" :

1001
default via 77.64.166.1 dev eth0.2 
1002
default via 172.16.36.1 dev pptp-e1 
1016
default via 172.16.36.1 dev pptp-e1  metric 1 
default via 172.16.36.1 dev pptp-e1  metric 2 
1017
default via 77.64.166.1 dev eth0.2  metric 1 

Output of "iptables -L -t mangle -v -n | awk '/mwan3/' RS=" :

Chain PREROUTING (policy ACCEPT 11113 packets, 847K bytes)
 pkts bytes target     prot opt in     out     source               destination         
11137  850K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
11113  847K fwmark     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 4014 packets, 421K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 4030  423K mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 11790 packets, 1109K bytes)
 pkts bytes target     prot opt in     out     source               destination         
11821 1111K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 11823 packets, 1111K bytes)
 pkts bytes target     prot opt in     out     source               destination         
11850 1113K mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_default (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3          mark match ! 0x8000/0x8000 MARK or 0x8000
    0     0 MARK       all  --  *      *       0.0.0.0/0            77.64.166.0/24       mark match ! 0x8000/0x8000 MARK or 0x8000
    0     0 MARK       all  --  *      *       0.0.0.0/0            172.16.36.1          mark match ! 0x8000/0x8000 MARK or 0x8000
 9478  731K MARK       all  --  *      *       0.0.0.0/0            192.168.3.0/24       mark match ! 0x8000/0x8000 MARK or 0x8000

Chain mwan3_post (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  717 57612 MARK       all  --  *      pptp-e1  0.0.0.0/0            0.0.0.0/0            mark match ! 0x8000/0x8000 MARK xset 0x200/0xff00
 2833  546K MARK       all  --  *      eth0.2  0.0.0.0/0            0.0.0.0/0            mark match ! 0x8000/0x8000 MARK xset 0x100/0xff00
12305  930K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x8000/0x8000 MARK and 0xffff7fff
15880 1536K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff00

Chain mwan3_pre (2 references)
 pkts bytes target     prot opt in     out     source               destination         
22958 1961K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0xff00
  132 11754 MARK       all  --  pptp-e1 *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x8200/0xff00
 2692  187K MARK       all  --  eth0.2 *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x8100/0xff00
20134 1762K mwan3_default  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x8000/0x8000
 7602  468K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 7570  465K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 MARK xset 0x1000/0xff00


Output of "ifconfig" :

br-lan    Link encap:Ethernet  HWaddr A0:F3:C1:FA:C6:2B  
          inet addr:192.168.3.1  Bcast:192.168.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13711 errors:0 dropped:20 overruns:0 frame:0
          TX packets:13551 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:837826 (818.1 KiB)  TX bytes:751330 (733.7 KiB)

eth0      Link encap:Ethernet  HWaddr 00:04:9F:EF:01:01  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:20166 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20584 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2084596 (1.9 MiB)  TX bytes:2656232 (2.5 MiB)
          Base address:0x4000 

eth0.1    Link encap:Ethernet  HWaddr 00:04:9F:EF:01:01  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13580 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13558 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:822535 (803.2 KiB)  TX bytes:753210 (735.5 KiB)

eth0.2    Link encap:Ethernet  HWaddr A0:F3:C1:FA:C6:2E  
          inet addr:77.64.166.181  Bcast:77.64.166.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6559 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7026 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:655965 (640.5 KiB)  TX bytes:1820686 (1.7 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:6101 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1232923 (1.1 MiB)  TX bytes:1232923 (1.1 MiB)

pptp-e1   Link encap:Point-to-Point Protocol  
          inet addr:172.16.36.103  P-t-P:172.16.36.1  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:253 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1438 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3 
          RX bytes:21542 (21.0 KiB)  TX bytes:108184 (105.6 KiB)

wlan0     Link encap:Ethernet  HWaddr A0:F3:C1:FA:C6:2D  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:1688 (1.6 KiB)

wlan1     Link encap:Ethernet  HWaddr A0:F3:C1:FA:C6:2C  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:65 errors:0 dropped:0 overruns:0 frame:0
          TX packets:64 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:7939 (7.7 KiB)  TX bytes:9942 (9.7 KiB)

Output of "cat /etc/config/mwan3" :

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'e1'
    option enabled '1'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'e1_m2_w3'
    option interface 'e1'
    option weight '3'
    option metric '1'

config policy 'e1_only'
    list use_member 'e1_m2_w3'

config policy 'wan_only'
    list use_member 'wan_m1_w3'

config rule 'vpn'
    option proto 'all'
    option use_policy 'e1_only'

Output of "cat /etc/config/network" :

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config globals 'globals'
    option ula_prefix 'fd5f:7ded:706d::/48'

config interface 'lan'
    option ifname 'eth0.1'
    option type 'bridge'
    option proto 'static'
    option netmask '255.255.255.0'
    option ip6assign '60'
    option macaddr 'a0:f3:c1:fa:c6:2b'
    option ipaddr '192.168.3.1'

config interface 'wan'
    option ifname 'eth0.2'
    option proto 'dhcp'
    option macaddr 'a0:f3:c1:fa:c6:2e'
    option metric '5'

config switch
    option name 'switch0'
    option reset '1'
    option enable_vlan '1'

config switch_vlan
    option device 'switch0'
    option vlan '1'
    option ports '0t 2 3 4 5'

config switch_vlan
    option device 'switch0'
    option vlan '2'
    option ports '0t 1'

config route
    option interface 'lan'
    option target '192.168.2.0'
    option netmask '255.255.255.0'
    option gateway '192.168.3.2'

config route
    option interface 'lan'
    option target '192.168.1.0'
    option netmask '255.255.255.0'
    option gateway '192.168.3.2'

config interface 'e1'
    option proto 'pptp'
    option server '** MY VPN HOST **'
    option username '** MY VPN USER **'
    option password '** MY VPN PASSWORD **'
    option metric '10'

But when i Switch back to wan_only, which is my Cableconnection, everything is fine:

root@Openwrt:~# ping -c3 -w 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=49 time=29.397 ms
64 bytes from 8.8.8.8: seq=1 ttl=49 time=28.186 ms
64 bytes from 8.8.8.8: seq=2 ttl=49 time=26.059 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 26.059/27.880/29.397 ms
root@Openwrt:~# ping -c3 -w 3 8.8.8.8 -I eth0.2
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=49 time=29.936 ms
64 bytes from 8.8.8.8: seq=1 ttl=49 time=26.420 ms
64 bytes from 8.8.8.8: seq=2 ttl=49 time=27.521 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 26.420/27.959/29.936 ms
root@Openwrt:~# ping -c3 -w 3 8.8.8.8 -I pptp-e1
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=49 time=94.381 ms
64 bytes from 8.8.8.8: seq=1 ttl=49 time=93.822 ms
64 bytes from 8.8.8.8: seq=2 ttl=49 time=90.036 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 90.036/92.746/94.381 ms
root@Openwrt:~# traceroute www.google.de
traceroute to www.google.de (173.194.69.94), 30 hops max, 38 byte packets
 1  10.180.0.1 (10.180.0.1)  9.953 ms  10.978 ms  11.634 ms
 2  172.16.0.113 (172.16.0.113)  17.666 ms  19.349 ms  14.653 ms
 3  172.16.16.5 (172.16.16.5)  18.487 ms  18.185 ms  13.297 ms
 4  217.69.226.54 (217.69.226.54)  22.535 ms  26.838 ms  13.353 ms
 5  209.85.249.184 (209.85.249.184)  35.438 ms  31.888 ms  209.85.249.182 (209.85.249.182)  33.506 ms
 6  66.249.95.67 (66.249.95.67)  32.791 ms  33.082 ms  66.249.95.219 (66.249.95.219)  28.864 ms
 7  64.233.174.53 (64.233.174.53)  30.732 ms  32.305 ms  64.233.174.29 (64.233.174.29)  38.499 ms
 8  *  *  *
 9  bk-in-f94.1e100.net (173.194.69.94)  32.231 ms  27.162 ms  25.218 ms

EDIT!!
Fixed it on my own. Maybe someone has the same quirk.
The solution was to ad a forwarding rule in openwrt firewall to
the "vpnzone"

(Last edited by derdigge on 17 May 2013, 01:53)

hi Adze

I dont know why. I load balancing 2 line 3G. After 5 minutes download, 1 line will not load balancing. Only 1 line download, i remove line working, line remain work ( that is no line drop ). I think maybe my usb hub, but i change other hub ( 2 kind, and many hub ) nothing change.
/ps: i cant remove ip server ping in interface

(Last edited by khanh3t on 17 May 2013, 06:02)

khanh3t wrote:

hi Adze

I dont know why. I load balancing 2 line 3G. After 5 minutes download, 1 line will not load balancing. Only 1 line download, i remove line working, line remain work ( that is no line drop ). I think maybe my usb hub, but i change other hub ( 2 kind, and many hub ) nothing change.
/ps: i cant remove ip server ping in interface

I'm experiencing the same behaviour. After a while, the traffic flows through one interface and the load-balancing fades away.
Stopping the download procces fixes things, e.g. my external IP changes correctly.

headless.cross wrote:
khanh3t wrote:

hi Adze

I dont know why. I load balancing 2 line 3G. After 5 minutes download, 1 line will not load balancing. Only 1 line download, i remove line working, line remain work ( that is no line drop ). I think maybe my usb hub, but i change other hub ( 2 kind, and many hub ) nothing change.
/ps: i cant remove ip server ping in interface

I'm experiencing the same behaviour. After a while, the traffic flows through one interface and the load-balancing fades away.
Stopping the download procces fixes things, e.g. my external IP changes correctly.


Could you guys please paste me the output of "cat /proc/net/nf_conntrack" please? If possible 2 times (when load-balancing is fine and when load-balancing grinds to a halt). Thnx!

Hi all,

It looks like since Openwrt trunk got iproute2 updated to version 3.9.0 on May 14th, the table handling of mwan3 does not work anymore. Since this update it is not possible to use tables numbers greater than 255.

for example:

ip route add 192.168.10.0/24 dev eth0.2 table 1001
ip: invalid argument '1001' to 'table'

but what works is:
ip route add 192.168.10.0/24 dev eth0.2 table 101

So is this an iproute2 upstream issue or should tha max number of iproute2 tables be always smaller thatn 255 ?

Bye Bluse

Bluse-Blue, that error string does not belong to iproute2, it seems to belong to the BusyBox ip applet.
This applet does not handle TIDs > 255 as it is not supporting the newer netlink attribute needed for 32bit TIDs.

Hi Bluse-Blue

As jow already replied, you're using the busybox applet of iproute. I have iproute 3.9.0 installed and no problem here.

Adze wrote:

Could you guys please paste me the output of "cat /proc/net/nf_conntrack" please? If possible 2 times (when load-balancing is fine and when load-balancing grinds to a halt). Thnx!

Hello Adze, I will post the "after" results of the "cat /proc/net/nf_conntrack" because I can't catch the max speed peak, the speed from the start of the download is wwan's max speed, but when surfing, my external ip changes (this proves that the load-balancing works), very strange behaviour.

Here's the output:

ipv4     2 tcp      6 50 TIME_WAIT src=10.0.2.2 dst=46.252.112.34 sport=2414 dport=36027 packets=9 bytes=1059 src=46.252.112.34 dst=10.0.0.2 sport=36027 dport=2414 packets=8 bytes=971 [ASSURED] mark=256 use=2
ipv4     2 udp      17 127 src=10.0.2.2 dst=84.202.169.202 sport=15551 dport=44125 packets=5 bytes=655 src=84.202.169.202 dst=192.168.0.2 sport=44125 dport=15551 packets=3 bytes=836 [ASSURED] mark=512 use=2
ipv4     2 udp      17 27 src=10.0.2.2 dst=24.92.90.240 sport=15551 dport=24654 packets=1 bytes=131 [UNREPLIED] src=24.92.90.240 dst=10.0.0.2 sport=24654 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 178 src=10.0.2.2 dst=71.89.123.239 sport=15551 dport=11779 packets=7376 bytes=368978 src=71.89.123.239 dst=10.0.0.2 sport=11779 dport=15551 packets=7192 bytes=10533009 [ASSURED] mark=256 use=2
ipv4     2 udp      17 6 src=10.0.2.2 dst=176.37.113.92 sport=15551 dport=15422 packets=4 bytes=269 [UNREPLIED] src=176.37.113.92 dst=10.0.0.2 sport=15422 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 12 src=10.0.2.2 dst=87.143.233.91 sport=15551 dport=50718 packets=1 bytes=95 [UNREPLIED] src=87.143.233.91 dst=192.168.0.2 sport=50718 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 152 src=10.0.2.2 dst=174.71.19.2 sport=15551 dport=37509 packets=5 bytes=789 src=174.71.19.2 dst=10.0.0.2 sport=37509 dport=15551 packets=3 bytes=655 [ASSURED] mark=256 use=2
ipv4     2 udp      17 156 src=10.0.2.2 dst=78.109.177.133 sport=15551 dport=64223 packets=1 bytes=131 src=78.109.177.133 dst=10.0.0.2 sport=64223 dport=15551 packets=5 bytes=655 [ASSURED] mark=256 use=2
ipv4     2 udp      17 56 src=10.0.2.2 dst=222.8.33.13 sport=15551 dport=56206 packets=1 bytes=131 src=222.8.33.13 dst=10.0.0.2 sport=56206 dport=15551 packets=1 bytes=305 mark=256 use=2
ipv4     2 udp      17 46 src=10.0.2.2 dst=89.93.48.129 sport=15551 dport=6881 packets=1 bytes=131 [UNREPLIED] src=89.93.48.129 dst=192.168.0.2 sport=6881 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 tcp      6 44 SYN_SENT src=10.0.2.2 dst=83.243.65.171 sport=2303 dport=18183 packets=3 bytes=144 [UNREPLIED] src=83.243.65.171 dst=192.168.0.2 sport=18183 dport=2303 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 56 src=10.0.2.2 dst=41.77.2.249 sport=15551 dport=32022 packets=1 bytes=95 [UNREPLIED] src=41.77.2.249 dst=10.0.0.2 sport=32022 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 101 SYN_SENT src=10.0.2.2 dst=5.158.233.74 sport=2865 dport=60877 packets=3 bytes=144 [UNREPLIED] src=5.158.233.74 dst=192.168.0.2 sport=60877 dport=2865 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 12 src=10.0.2.2 dst=60.226.129.89 sport=15551 dport=49776 packets=1 bytes=134 [UNREPLIED] src=60.226.129.89 dst=10.0.0.2 sport=49776 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 55 src=10.0.2.2 dst=85.220.97.95 sport=15551 dport=50746 packets=1 bytes=95 [UNREPLIED] src=85.220.97.95 dst=10.0.0.2 sport=50746 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 92 src=10.0.2.2 dst=80.251.122.164 sport=15551 dport=13026 packets=20 bytes=2529 src=80.251.122.164 dst=10.0.0.2 sport=13026 dport=15551 packets=21 bytes=3571 [ASSURED] mark=256 use=2
ipv4     2 udp      17 59 src=10.0.2.2 dst=92.230.192.109 sport=15551 dport=51413 packets=1 bytes=58 [UNREPLIED] src=92.230.192.109 dst=192.168.0.2 sport=51413 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 59 src=10.0.2.2 dst=46.38.2.93 sport=15551 dport=51413 packets=1 bytes=134 src=46.38.2.93 dst=192.168.0.2 sport=51413 dport=15551 packets=1 bytes=313 mark=512 use=2
ipv4     2 udp      17 4 src=10.0.2.2 dst=108.180.64.251 sport=15551 dport=51413 packets=1 bytes=134 src=108.180.64.251 dst=10.0.0.2 sport=51413 dport=15551 packets=1 bytes=331 mark=256 use=2
ipv4     2 udp      17 172 src=10.0.2.2 dst=121.96.63.227 sport=15551 dport=10994 packets=17 bytes=5228 src=121.96.63.227 dst=10.0.0.2 sport=10994 dport=15551 packets=16 bytes=2153 [ASSURED] mark=256 use=2
ipv4     2 udp      17 23 src=10.0.2.2 dst=95.233.65.228 sport=15551 dport=27983 packets=4 bytes=269 [UNREPLIED] src=95.233.65.228 dst=10.0.0.2 sport=27983 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 0 src=10.0.2.2 dst=85.17.73.196 sport=15551 dport=25411 packets=3 bytes=174 [UNREPLIED] src=85.17.73.196 dst=10.0.0.2 sport=25411 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 117 src=10.0.2.2 dst=178.162.47.79 sport=15551 dport=12366 packets=8 bytes=813 src=178.162.47.79 dst=192.168.0.2 sport=12366 dport=15551 packets=7 bytes=1296 [ASSURED] mark=512 use=2
ipv4     2 tcp      6 62 SYN_SENT src=10.0.2.2 dst=77.50.101.19 sport=2477 dport=40963 packets=3 bytes=144 [UNREPLIED] src=77.50.101.19 dst=10.0.0.2 sport=40963 dport=2477 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 34 src=10.0.2.2 dst=207.38.136.170 sport=15551 dport=15719 packets=1 bytes=95 [UNREPLIED] src=207.38.136.170 dst=10.0.0.2 sport=15719 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 19 src=10.0.2.2 dst=94.109.62.190 sport=15551 dport=53672 packets=1 bytes=131 [UNREPLIED] src=94.109.62.190 dst=192.168.0.2 sport=53672 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 22 src=10.0.2.2 dst=5.43.69.68 sport=15551 dport=59278 packets=4 bytes=269 [UNREPLIED] src=5.43.69.68 dst=192.168.0.2 sport=59278 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 175 src=10.0.2.2 dst=122.176.143.54 sport=15551 dport=32519 packets=12 bytes=1622 src=122.176.143.54 dst=192.168.0.2 sport=32519 dport=15551 packets=13 bytes=1513 [ASSURED] mark=512 use=2
ipv4     2 udp      17 49 src=10.0.2.2 dst=88.186.140.131 sport=15551 dport=51413 packets=1 bytes=131 [UNREPLIED] src=88.186.140.131 dst=10.0.0.2 sport=51413 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 59 SYN_SENT src=10.0.2.2 dst=31.31.100.220 sport=2451 dport=21343 packets=3 bytes=144 [UNREPLIED] src=31.31.100.220 dst=192.168.0.2 sport=21343 dport=2451 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 169 src=10.0.2.2 dst=95.68.83.205 sport=15551 dport=63511 packets=14 bytes=1659 src=95.68.83.205 dst=192.168.0.2 sport=63511 dport=15551 packets=15 bytes=1608 [ASSURED] mark=512 use=2
ipv4     2 udp      17 37 src=10.2.2.1 dst=155.207.1.8 sport=44304 dport=123 packets=1 bytes=76 src=155.207.1.8 dst=10.0.0.2 sport=123 dport=44304 packets=1 bytes=76 mark=256 use=2
ipv4     2 udp      17 31 src=10.0.2.2 dst=177.3.193.2 sport=15551 dport=37896 packets=1 bytes=95 [UNREPLIED] src=177.3.193.2 dst=192.168.0.2 sport=37896 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 165 src=10.0.2.2 dst=86.212.96.184 sport=15551 dport=28052 packets=2 bytes=376 src=86.212.96.184 dst=10.0.0.2 sport=28052 dport=15551 packets=2 bytes=568 [ASSURED] mark=256 use=2
ipv4     2 udp      17 50 src=10.0.2.2 dst=151.32.246.102 sport=15551 dport=58880 packets=1 bytes=131 src=151.32.246.102 dst=192.168.0.2 sport=58880 dport=15551 packets=1 bytes=315 mark=512 use=2
ipv4     2 udp      17 121 src=10.0.2.2 dst=77.34.113.212 sport=15551 dport=6881 packets=2 bytes=262 src=77.34.113.212 dst=10.0.0.2 sport=6881 dport=15551 packets=2 bytes=610 [ASSURED] mark=256 use=2
ipv4     2 tcp      6 118 TIME_WAIT src=10.0.2.2 dst=122.176.143.54 sport=3070 dport=32519 packets=8 bytes=822 src=122.176.143.54 dst=192.168.0.2 sport=32519 dport=3070 packets=6 bytes=597 [ASSURED] mark=512 use=2
ipv4     2 udp      17 45 src=10.0.2.2 dst=109.229.19.6 sport=15551 dport=11789 packets=1 bytes=131 src=109.229.19.6 dst=192.168.0.2 sport=11789 dport=15551 packets=1 bytes=315 mark=512 use=2
ipv4     2 udp      17 118 src=10.0.2.2 dst=46.160.81.215 sport=15551 dport=49001 packets=2 bytes=262 src=46.160.81.215 dst=10.0.0.2 sport=49001 dport=15551 packets=2 bytes=610 [ASSURED] mark=256 use=2
ipv4     2 udp      17 4 src=10.0.2.2 dst=184.75.213.106 sport=15551 dport=61824 packets=1 bytes=131 [UNREPLIED] src=184.75.213.106 dst=192.168.0.2 sport=61824 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 108 src=10.0.2.2 dst=115.212.89.197 sport=15551 dport=4041 packets=2 bytes=262 src=115.212.89.197 dst=10.0.0.2 sport=4041 dport=15551 packets=1 bytes=326 [ASSURED] mark=256 use=2
ipv4     2 udp      17 23 src=10.0.2.2 dst=82.26.161.89 sport=15551 dport=17226 packets=1 bytes=134 [UNREPLIED] src=82.26.161.89 dst=192.168.0.2 sport=17226 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 1 src=10.0.2.2 dst=10.0.2.1 sport=58604 dport=53 packets=1 bytes=73 src=10.0.2.1 dst=10.0.2.2 sport=53 dport=58604 packets=1 bytes=89 mark=0 use=2
ipv4     2 udp      17 57 src=10.0.2.2 dst=85.47.113.114 sport=15551 dport=24874 packets=3 bytes=211 [UNREPLIED] src=85.47.113.114 dst=192.168.0.2 sport=24874 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 11 src=10.0.2.2 dst=146.66.47.206 sport=15551 dport=51755 packets=1 bytes=145 [UNREPLIED] src=146.66.47.206 dst=10.0.0.2 sport=51755 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 0 src=10.0.2.2 dst=78.173.156.69 sport=15551 dport=1030 packets=1 bytes=131 [UNREPLIED] src=78.173.156.69 dst=10.0.0.2 sport=1030 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 111 src=10.0.2.2 dst=78.90.230.6 sport=15551 dport=6881 packets=2 bytes=262 src=78.90.230.6 dst=192.168.0.2 sport=6881 dport=15551 packets=2 bytes=592 [ASSURED] mark=512 use=2
ipv4     2 tcp      6 53 SYN_SENT src=10.0.2.2 dst=204.83.195.200 sport=2387 dport=58484 packets=3 bytes=144 [UNREPLIED] src=204.83.195.200 dst=10.0.0.2 sport=58484 dport=2387 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 50 src=10.0.2.2 dst=222.127.85.13 sport=15551 dport=41657 packets=1 bytes=95 [UNREPLIED] src=222.127.85.13 dst=192.168.0.2 sport=41657 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 57 src=10.0.2.2 dst=84.212.203.229 sport=15551 dport=47026 packets=1 bytes=131 src=84.212.203.229 dst=10.0.0.2 sport=47026 dport=15551 packets=1 bytes=296 mark=256 use=2
ipv4     2 udp      17 6 src=10.0.2.2 dst=121.54.40.51 sport=15551 dport=10914 packets=2 bytes=262 [UNREPLIED] src=121.54.40.51 dst=192.168.0.2 sport=10914 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 28 src=10.0.2.2 dst=89.164.193.7 sport=15551 dport=40581 packets=1 bytes=95 [UNREPLIED] src=89.164.193.7 dst=192.168.0.2 sport=40581 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 58 src=10.0.2.2 dst=186.32.25.208 sport=15551 dport=59359 packets=1 bytes=95 [UNREPLIED] src=186.32.25.208 dst=10.0.0.2 sport=59359 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 51 src=10.0.2.2 dst=79.116.217.80 sport=15551 dport=17389 packets=1 bytes=131 [UNREPLIED] src=79.116.217.80 dst=192.168.0.2 sport=17389 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 26 src=10.0.2.2 dst=118.171.156.229 sport=15551 dport=1385 packets=1 bytes=131 [UNREPLIED] src=118.171.156.229 dst=192.168.0.2 sport=1385 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 tcp      6 25 SYN_SENT src=10.0.2.2 dst=79.167.37.230 sport=2138 dport=15551 packets=2 bytes=96 [UNREPLIED] src=79.167.37.230 dst=10.0.0.2 sport=15551 dport=2138 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 14 src=10.0.2.2 dst=197.153.16.187 sport=15551 dport=15985 packets=1 bytes=95 [UNREPLIED] src=197.153.16.187 dst=10.0.0.2 sport=15985 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 115 src=10.0.2.2 dst=58.69.94.111 sport=15551 dport=22683 packets=3 bytes=527 src=58.69.94.111 dst=192.168.0.2 sport=22683 dport=15551 packets=1 bytes=513 [ASSURED] mark=512 use=2
ipv4     2 udp      17 126 src=10.0.2.2 dst=174.95.115.2 sport=15551 dport=30787 packets=2 bytes=379 src=174.95.115.2 dst=192.168.0.2 sport=30787 dport=15551 packets=1 bytes=473 [ASSURED] mark=512 use=2
ipv4     2 tcp      6 8 CLOSE src=10.0.2.2 dst=81.129.62.230 sport=3080 dport=6881 packets=6 bytes=888 src=81.129.62.230 dst=192.168.0.2 sport=6881 dport=3080 packets=6 bytes=556 [ASSURED] mark=512 use=2
ipv4     2 udp      17 11 src=10.0.2.2 dst=41.140.225.62 sport=15551 dport=47800 packets=1 bytes=95 [UNREPLIED] src=41.140.225.62 dst=10.0.0.2 sport=47800 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 50 src=10.0.2.2 dst=188.244.137.248 sport=15551 dport=33815 packets=1 bytes=95 [UNREPLIED] src=188.244.137.248 dst=10.0.0.2 sport=33815 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 173 src=10.0.2.2 dst=79.167.37.230 sport=15551 dport=15551 packets=35 bytes=3924 src=79.167.37.230 dst=10.0.0.2 sport=15551 dport=15551 packets=15 bytes=2385 [ASSURED] mark=256 use=2
ipv4     2 udp      17 50 src=10.0.2.2 dst=115.133.211.6 sport=15551 dport=32411 packets=1 bytes=131 [UNREPLIED] src=115.133.211.6 dst=10.0.0.2 sport=32411 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 105 SYN_SENT src=10.0.2.2 dst=92.99.27.244 sport=2905 dport=29814 packets=3 bytes=144 [UNREPLIED] src=92.99.27.244 dst=10.0.0.2 sport=29814 dport=2905 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 97 src=10.0.2.2 dst=79.176.101.247 sport=15551 dport=40301 packets=2 bytes=262 src=79.176.101.247 dst=10.0.0.2 sport=40301 dport=15551 packets=1 bytes=315 [ASSURED] mark=256 use=2
ipv4     2 udp      17 87 src=10.0.2.2 dst=92.47.206.100 sport=15551 dport=40140 packets=6 bytes=774 src=92.47.206.100 dst=10.0.0.2 sport=40140 dport=15551 packets=4 bytes=805 [ASSURED] mark=256 use=2
ipv4     2 udp      17 121 src=10.0.2.2 dst=95.26.7.5 sport=15551 dport=29746 packets=24 bytes=2727 src=95.26.7.5 dst=10.0.0.2 sport=29746 dport=15551 packets=22 bytes=2618 [ASSURED] mark=256 use=2
ipv4     2 udp      17 99 src=10.0.2.2 dst=207.216.203.254 sport=15551 dport=50090 packets=2 bytes=262 src=207.216.203.254 dst=192.168.0.2 sport=50090 dport=15551 packets=2 bytes=630 [ASSURED] mark=512 use=2
ipv4     2 tcp      6 24 TIME_WAIT src=10.0.2.2 dst=10.0.2.1 sport=2195 dport=80 packets=7 bytes=749 src=10.0.2.1 dst=10.0.2.2 sport=80 dport=2195 packets=9 bytes=3691 [ASSURED] mark=0 use=2
ipv4     2 udp      17 59 src=10.0.2.2 dst=123.201.96.97 sport=15551 dport=32536 packets=1 bytes=134 [UNREPLIED] src=123.201.96.97 dst=10.0.0.2 sport=32536 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 39 src=10.0.2.2 dst=70.29.219.205 sport=15551 dport=6881 packets=1 bytes=131 src=70.29.219.205 dst=10.0.0.2 sport=6881 dport=15551 packets=1 bytes=315 mark=256 use=2
ipv4     2 udp      17 37 src=10.0.2.2 dst=188.251.214.173 sport=15551 dport=64093 packets=1 bytes=131 [UNREPLIED] src=188.251.214.173 dst=192.168.0.2 sport=64093 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 7 src=10.0.2.2 dst=109.174.52.41 sport=15551 dport=7090 packets=3 bytes=174 [UNREPLIED] src=109.174.52.41 dst=10.0.0.2 sport=7090 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 48 src=10.0.2.2 dst=78.184.184.38 sport=15551 dport=16610 packets=1 bytes=131 src=78.184.184.38 dst=192.168.0.2 sport=16610 dport=15551 packets=1 bytes=315 mark=512 use=2
ipv4     2 udp      17 2 src=10.0.2.2 dst=77.50.101.19 sport=15551 dport=40963 packets=3 bytes=174 [UNREPLIED] src=77.50.101.19 dst=192.168.0.2 sport=40963 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 53 src=10.0.2.2 dst=96.48.205.84 sport=15551 dport=41635 packets=1 bytes=131 src=96.48.205.84 dst=192.168.0.2 sport=41635 dport=15551 packets=1 bytes=315 mark=512 use=2
ipv4     2 udp      17 33 src=10.0.2.2 dst=50.51.15.61 sport=15551 dport=38167 packets=1 bytes=131 [UNREPLIED] src=50.51.15.61 dst=10.0.0.2 sport=38167 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 40 src=10.0.2.2 dst=109.230.160.98 sport=15551 dport=11607 packets=1 bytes=131 src=109.230.160.98 dst=10.0.0.2 sport=11607 dport=15551 packets=1 bytes=315 mark=256 use=2
ipv4     2 tcp      6 94 TIME_WAIT src=10.0.2.2 dst=10.0.2.1 sport=2879 dport=80 packets=7 bytes=748 src=10.0.2.1 dst=10.0.2.2 sport=80 dport=2879 packets=9 bytes=3690 [ASSURED] mark=0 use=2
ipv4     2 udp      17 19 src=10.0.2.2 dst=31.18.41.240 sport=15551 dport=6881 packets=1 bytes=95 [UNREPLIED] src=31.18.41.240 dst=10.0.0.2 sport=6881 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 31 SYN_SENT src=10.0.2.2 dst=95.46.80.23 sport=2169 dport=47673 packets=3 bytes=144 [UNREPLIED] src=95.46.80.23 dst=192.168.0.2 sport=47673 dport=2169 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 50 src=10.0.2.2 dst=82.226.214.26 sport=15551 dport=6881 packets=1 bytes=131 src=82.226.214.26 dst=10.0.0.2 sport=6881 dport=15551 packets=1 bytes=296 mark=256 use=2
ipv4     2 tcp      6 119 SYN_SENT src=10.0.2.2 dst=108.180.252.88 sport=3035 dport=25125 packets=3 bytes=144 [UNREPLIED] src=108.180.252.88 dst=10.0.0.2 sport=25125 dport=3035 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 24 src=10.0.2.2 dst=99.141.59.97 sport=15551 dport=55224 packets=1 bytes=131 src=99.141.59.97 dst=192.168.0.2 sport=55224 dport=15551 packets=1 bytes=315 mark=512 use=2
ipv4     2 udp      17 8 src=10.0.2.2 dst=31.23.217.255 sport=15551 dport=12928 packets=3 bytes=393 [UNREPLIED] src=31.23.217.255 dst=10.0.0.2 sport=12928 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 43 src=10.0.2.2 dst=88.120.150.103 sport=15551 dport=51413 packets=1 bytes=131 [UNREPLIED] src=88.120.150.103 dst=192.168.0.2 sport=51413 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 tcp      6 89 SYN_SENT src=10.0.2.2 dst=94.70.5.52 sport=2739 dport=57392 packets=3 bytes=144 [UNREPLIED] src=94.70.5.52 dst=10.0.0.2 sport=57392 dport=2739 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 22 TIME_WAIT src=10.0.2.2 dst=89.248.162.149 sport=2173 dport=80 packets=5 bytes=562 src=89.248.162.149 dst=10.0.0.2 sport=80 dport=2173 packets=5 bytes=738 [ASSURED] mark=256 use=2
ipv4     2 udp      17 49 src=10.0.2.2 dst=82.235.103.72 sport=15551 dport=51413 packets=1 bytes=131 [UNREPLIED] src=82.235.103.72 dst=10.0.0.2 sport=51413 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 58 src=10.0.2.2 dst=70.75.207.85 sport=15551 dport=24874 packets=2 bytes=153 [UNREPLIED] src=70.75.207.85 dst=192.168.0.2 sport=24874 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 162 src=10.0.2.2 dst=89.173.20.35 sport=15551 dport=53504 packets=17 bytes=2284 src=89.173.20.35 dst=10.0.0.2 sport=53504 dport=15551 packets=12 bytes=1052 [ASSURED] mark=256 use=2
ipv4     2 tcp      6 75 SYN_SENT src=10.0.2.2 dst=86.28.150.50 sport=2614 dport=34444 packets=3 bytes=144 [UNREPLIED] src=86.28.150.50 dst=10.0.0.2 sport=34444 dport=2614 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 4 src=10.0.2.2 dst=10.0.2.1 sport=58024 dport=53 packets=1 bytes=60 src=10.0.2.1 dst=10.0.2.2 sport=53 dport=58024 packets=1 bytes=76 mark=0 use=2
ipv4     2 tcp      6 44 SYN_SENT src=10.0.2.2 dst=84.51.116.55 sport=2305 dport=24018 packets=3 bytes=144 [UNREPLIED] src=84.51.116.55 dst=10.0.0.2 sport=24018 dport=2305 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 59 src=10.0.2.2 dst=96.242.135.111 sport=15551 dport=20500 packets=1 bytes=95 [UNREPLIED] src=96.242.135.111 dst=10.0.0.2 sport=20500 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 77 SYN_SENT src=10.0.2.2 dst=108.207.27.41 sport=2631 dport=47270 packets=3 bytes=144 [UNREPLIED] src=108.207.27.41 dst=192.168.0.2 sport=47270 dport=2631 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 169 src=10.0.2.2 dst=187.140.96.143 sport=15551 dport=19992 packets=15 bytes=4054 src=187.140.96.143 dst=192.168.0.2 sport=19992 dport=15551 packets=16 bytes=1852 [ASSURED] mark=512 use=2
ipv4     2 udp      17 107 src=10.0.2.2 dst=176.109.241.133 sport=15551 dport=11789 packets=2 bytes=262 src=176.109.241.133 dst=10.0.0.2 sport=11789 dport=15551 packets=2 bytes=630 [ASSURED] mark=256 use=2
ipv4     2 tcp      6 69 TIME_WAIT src=10.0.2.2 dst=10.0.2.1 sport=2635 dport=80 packets=7 bytes=747 src=10.0.2.1 dst=10.0.2.2 sport=80 dport=2635 packets=9 bytes=3689 [ASSURED] mark=0 use=2
ipv4     2 udp      17 50 src=10.0.2.2 dst=93.96.214.28 sport=15551 dport=62525 packets=3 bytes=174 [UNREPLIED] src=93.96.214.28 dst=10.0.0.2 sport=62525 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 12 src=10.0.2.2 dst=46.35.165.227 sport=15551 dport=20245 packets=1 bytes=134 [UNREPLIED] src=46.35.165.227 dst=10.0.0.2 sport=20245 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 51 src=10.0.2.2 dst=88.172.11.97 sport=15551 dport=61288 packets=1 bytes=131 src=88.172.11.97 dst=10.0.0.2 sport=61288 dport=15551 packets=1 bytes=315 mark=256 use=2
ipv4     2 udp      17 2 src=10.0.2.2 dst=176.195.40.36 sport=15551 dport=34281 packets=1 bytes=131 [UNREPLIED] src=176.195.40.36 dst=10.0.0.2 sport=34281 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 165 src=10.0.2.2 dst=58.106.79.49 sport=15551 dport=37485 packets=2 bytes=339 src=58.106.79.49 dst=10.0.0.2 sport=37485 dport=15551 packets=2 bytes=1251 [ASSURED] mark=256 use=2
ipv4     2 tcp      6 76 SYN_SENT src=10.0.2.2 dst=83.60.5.246 sport=2624 dport=26266 packets=3 bytes=144 [UNREPLIED] src=83.60.5.246 dst=192.168.0.2 sport=26266 dport=2624 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 85 src=10.0.2.2 dst=37.146.222.198 sport=15551 dport=53642 packets=2 bytes=262 src=37.146.222.198 dst=10.0.0.2 sport=53642 dport=15551 packets=2 bytes=630 [ASSURED] mark=256 use=2
ipv4     2 tcp      6 23 SYN_SENT src=10.0.2.2 dst=94.158.169.9 sport=2094 dport=48697 packets=3 bytes=144 [UNREPLIED] src=94.158.169.9 dst=10.0.0.2 sport=48697 dport=2094 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 34 src=10.0.2.2 dst=82.243.45.93 sport=15551 dport=51413 packets=1 bytes=131 [UNREPLIED] src=82.243.45.93 dst=192.168.0.2 sport=51413 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 125 src=10.0.2.2 dst=189.55.91.36 sport=15551 dport=12310 packets=1 bytes=131 src=189.55.91.36 dst=10.0.0.2 sport=12310 dport=15551 packets=2 bytes=253 [ASSURED] mark=256 use=2
ipv4     2 tcp      6 50 TIME_WAIT src=10.0.2.2 dst=37.214.107.91 sport=2449 dport=41012 packets=5 bytes=276 src=37.214.107.91 dst=192.168.0.2 sport=41012 dport=2449 packets=3 bytes=128 [ASSURED] mark=512 use=2
ipv4     2 udp      17 15 src=10.0.2.2 dst=49.136.94.152 sport=15551 dport=30594 packets=1 bytes=145 [UNREPLIED] src=49.136.94.152 dst=10.0.0.2 sport=30594 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 tcp      6 40 TIME_WAIT src=10.0.2.2 dst=80.202.242.47 sport=2326 dport=33158 packets=9 bytes=799 src=80.202.242.47 dst=192.168.0.2 sport=33158 dport=2326 packets=9 bytes=1196 [ASSURED] mark=512 use=2
ipv4     2 udp      17 169 src=10.0.2.2 dst=141.237.55.98 sport=15551 dport=47755 packets=15 bytes=3978 src=141.237.55.98 dst=192.168.0.2 sport=47755 dport=15551 packets=16 bytes=1717 [ASSURED] mark=512 use=2
ipv4     2 udp      17 44 src=10.0.2.2 dst=109.191.26.32 sport=15551 dport=28691 packets=1 bytes=131 src=109.191.26.32 dst=192.168.0.2 sport=28691 dport=15551 packets=1 bytes=305 mark=512 use=2
ipv4     2 udp      17 19 src=10.0.2.2 dst=212.253.181.191 sport=15551 dport=28396 packets=1 bytes=134 [UNREPLIED] src=212.253.181.191 dst=10.0.0.2 sport=28396 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 7 src=10.0.2.2 dst=184.75.214.170 sport=15551 dport=57558 packets=1 bytes=131 [UNREPLIED] src=184.75.214.170 dst=192.168.0.2 sport=57558 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 tcp      6 34 TIME_WAIT src=10.0.2.2 dst=10.0.2.1 sport=2287 dport=80 packets=7 bytes=748 src=10.0.2.1 dst=10.0.2.2 sport=80 dport=2287 packets=9 bytes=3690 [ASSURED] mark=0 use=2
ipv4     2 tcp      6 4 TIME_WAIT src=10.0.2.2 dst=10.0.2.1 sport=1988 dport=80 packets=7 bytes=748 src=10.0.2.1 dst=10.0.2.2 sport=80 dport=1988 packets=9 bytes=3690 [ASSURED] mark=0 use=2
ipv4     2 udp      17 101 src=10.0.2.2 dst=178.122.217.252 sport=15551 dport=49001 packets=2 bytes=262 src=178.122.217.252 dst=10.0.0.2 sport=49001 dport=15551 packets=2 bytes=610 [ASSURED] mark=256 use=2
ipv4     2 udp      17 1 src=10.0.2.2 dst=88.243.9.6 sport=15551 dport=5863 packets=1 bytes=131 [UNREPLIED] src=88.243.9.6 dst=10.0.0.2 sport=5863 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 97 src=10.0.2.2 dst=114.173.145.5 sport=15551 dport=47245 packets=1 bytes=131 src=114.173.145.5 dst=10.0.0.2 sport=47245 dport=15551 packets=3 bytes=393 [ASSURED] mark=256 use=2
ipv4     2 udp      17 29 src=10.0.2.2 dst=88.168.136.11 sport=15551 dport=51413 packets=1 bytes=131 [UNREPLIED] src=88.168.136.11 dst=192.168.0.2 sport=51413 dport=15551 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 32 src=10.0.2.2 dst=92.100.2.26 sport=15551 dport=22915 packets=1 bytes=95 [UNREPLIED] src=92.100.2.26 dst=10.0.0.2 sport=22915 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 108 src=10.0.2.2 dst=59.11.215.148 sport=15551 dport=22112 packets=2 bytes=262 src=59.11.215.148 dst=192.168.0.2 sport=22112 dport=15551 packets=2 bytes=620 [ASSURED] mark=512 use=2
ipv4     2 udp      17 175 src=10.0.2.2 dst=203.237.76.166 sport=15551 dport=56578 packets=10 bytes=1524 src=203.237.76.166 dst=192.168.0.2 sport=56578 dport=15551 packets=19 bytes=1951 [ASSURED] mark=512 use=2
ipv4     2 udp      17 42 src=10.0.2.2 dst=194.228.32.99 sport=15551 dport=42197 packets=1 bytes=131 [UNREPLIED] src=194.228.32.99 dst=10.0.0.2 sport=42197 dport=15551 packets=0 bytes=0 mark=256 use=2
ipv4     2 udp      17 6 src=10.0.2.2 dst=86.174.120.79 sport=15551 dport=31406 packets=1 bytes=131 src=86.174.120.79 dst=192.168.0.2 sport=31406 dport=15551 packets=1 bytes=305 mark=512 use=2
ipv4     2 tcp      6 85 SYN_SENT src=10.0.2.2 dst=195.78.246.11 sport=2707 dport=62488 packets=3 bytes=144 [UNREPLIED] src=195.78.246.11 dst=192.168.0.2 sport=62488 dport=2707 packets=0 bytes=0 mark=512 use=2
ipv4     2 udp      17 179 src=10.0.2.2 dst=117.199.67.26 sport=15551 dport=62348 packets=60 bytes=4029 src=117.199.67.26 dst=192.168.0.2 sport=62348 dport=15551 packets=55 bytes=20269 [ASSURED] mark=512 use=2

I would like to use mwan3 to increase the reliability of my connection to my PBX  /  VoIP Server.

1) Can I define a pool of OpenVPN tun interfaces to be my Wan connections?

2) What is the fastest time I can set for detecting a fail-over? sub-second? (I am trying not to drop a call)

3) can I setup a rule to only route my VoIP phones over this link? based on vlan ID? based on source IP?

Thanks,
Simeon