Hi all,

I added a wiki config section on controlling the external interface and IP used when sending out traffic. In my case, I have two WAN interfaces with dynamic IPs used to load-balance outgoing traffic for all normal clients, and a third WAN interface with a static IP block used for outgoing traffic from servers. I needed to map particular servers to particular static IP addresses on the wan3 interface.

http://wiki.openwrt.org/doc/howto/mwan3 … interfaces

Thanks for mwan3!

Regards,
Tim Miller Dyck

I'm a bit confused

eth0.1 is the LAN interface
eth0.2 is WAN2
eth1 is WAN

Where would I find the settings to configure the dhcp per interface?

(Last edited by jigglywiggly on 30 Oct 2013, 10:24)

Never mind got it. Uninstalled mwan 1.19.

Deleted the config file at /etc/config/mwan3

Installed 1.3 and the gui and I went to the interfaces tab, and had to setup the interfaces properly.
It's case sensitive for some reason. wan must be called wan, it cannot be called WAN, and WAN2 cannot be called wan2 in my case.

https://dl.dropboxusercontent.com/u/462 … s-mwan.PNG

had to change the rules in member configuration too

https://dl.dropboxusercontent.com/u/462 … ration.PNG

to match up with the interfaces

and then finally my twitch rules

https://dl.dropboxusercontent.com/u/462 … rules2.PNG

works great now, thanks for your hard work

mwan3 config

config member 'wan1_m1_w3'
    option metric '1'
    option weight '3'
    option interface 'wan'

config member 'wan1_m2_w3'
    option metric '2'
    option weight '3'
    option interface 'wan'

config member 'wan2_m1_w2'
    option metric '1'
    option weight '2'
    option interface 'WAN2'

config member 'wan2_m2_w2'
    option metric '2'
    option weight '2'
    option interface 'WAN2'

config policy 'wan1_only'
    list use_member 'wan1_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'wan1_wan2_loadbalanced'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m1_w2'

config policy 'wan1_pri_wan2_sec'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_pri_wan1_sec'
    list use_member 'wan1_m2_w3'
    list use_member 'wan2_m1_w2'

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '8.8.4.4'
    list track_ip '4.2.2.1'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '1'

config interface 'WAN2'
    option enabled '1'
    list track_ip '4.2.2.2'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'
    option reroute '1'

config rule 'twitch'
    option dest_ip '0.0.0.0/0'
    option dest_port '1935'
    option proto 'tcp'
    option use_policy 'wan2_only'
    option equalize '1'

config rule 'twitch2'
    option dest_ip '0.0.0.0/0'
    option dest_port '1935'
    option proto 'udp'
    option use_policy 'wan2_only'
    option equalize '1'

config rule 'loadover'
    option dest_ip '0.0.0.0/0'
    option proto 'all'
    option use_policy 'wan1_pri_wan2_sec'
    option equalize '1'

(Last edited by jigglywiggly on 30 Oct 2013, 16:26)

Hi Adze, can you help me out?
After installing mwan3, my ip rule is very strange:

Output of "ip rule show" :

0:    from all lookup local
1001:    from all fwmark 0x100/0xff00 lookup 1001
1002:    from all fwmark 0x200/0xff00 lookup 1002
1016:    from all fwmark 0x1000/0xff00 lookup 1016
1017:    from all fwmark 0x1100/0xff00 lookup 1017
1018:    from all fwmark 0x1200/0xff00 lookup 1018
1019:    from all fwmark 0x1300/0xff00 lookup 1019
1020:    from all fwmark 0x1400/0xff00 lookup 1020
32766:    from all lookup main
32767:    from all lookup default
80000:    from 192.168.1.1 nop
80000:    from 127.0.0.1 nop
80000:    from 192.168.16.108 nop
80000:    from 172.24.210.132 nop
90000:    from 192.168.1.1/24 nop
90000:    from 127.0.0.1/8 nop
90000:    from 192.168.16.108/24 nop
90000:    from 172.24.210.132/23 nop
-------------------------------------------------------------------------------------------
I have two wans, one is 192.168.16.1; another is 172.24.210.1.
I dont know why mwan3 cannot use the two gateways as default gateway.

more information you may need as flow. Is there anything I missed?
---------------------------------------------------------------------------------------------
Software versions :

OpenWrt - OpenWrt Barrier Breaker r38578 (r38578)
mwan3 - 1.3-0
luci-app-mwan3 - 1.1-13

Firewall default output policy (must be ACCEPT) :

ACCEPT

Output of "ip route show" :

default via 172.24.210.1 dev eth1  proto static  metric 10
172.24.210.0/23 dev eth1  proto static  scope link  metric 10
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
192.168.16.0/24 dev wlan0  proto static  scope link  metric 20

Output of "ip rule show" :

0:    from all lookup local
1001:    from all fwmark 0x100/0xff00 lookup 1001
1002:    from all fwmark 0x200/0xff00 lookup 1002
1016:    from all fwmark 0x1000/0xff00 lookup 1016
1017:    from all fwmark 0x1100/0xff00 lookup 1017
1018:    from all fwmark 0x1200/0xff00 lookup 1018
1019:    from all fwmark 0x1300/0xff00 lookup 1019
1020:    from all fwmark 0x1400/0xff00 lookup 1020
32766:    from all lookup main
32767:    from all lookup default
80000:    from 192.168.1.1 nop
80000:    from 127.0.0.1 nop
80000:    from 192.168.16.108 nop
80000:    from 172.24.210.132 nop
90000:    from 192.168.1.1/24 nop
90000:    from 127.0.0.1/8 nop
90000:    from 192.168.16.108/24 nop
90000:    from 172.24.210.132/23 nop

Output of "ip route list table 1001-1099" (1001-1015 = interface tables, 1016-1099 = policy tables) :

1001
default via 172.24.210.1 dev eth1
1002
default via 192.168.16.1 dev wlan0
1016
default via 172.24.210.1 dev eth1  metric 1
1017
default via 192.168.16.1 dev wlan0  metric 1
1018
default  metric 1
    nexthop via 172.24.210.1  dev eth1 weight 3
    nexthop via 192.168.16.1  dev wlan0 weight 2
1019
default via 172.24.210.1 dev eth1  metric 1
default via 192.168.16.1 dev wlan0  metric 2
1020
default via 192.168.16.1 dev wlan0  metric 1
default via 172.24.210.1 dev eth1  metric 2

Output of "iptables -L -t mangle -v -n | awk '/mwan3/' RS=" :

Chain PREROUTING (policy ACCEPT 15264 packets, 7085K bytes)
pkts bytes target     prot opt in     out     source               destination         
1779K 1110M mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
15264 7085K fwmark     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 1392 packets, 163K bytes)
pkts bytes target     prot opt in     out     source               destination         
141K   19M mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 488 packets, 151K bytes)
pkts bytes target     prot opt in     out     source               destination         
37573 4926K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 14253 packets, 7046K bytes)
pkts bytes target     prot opt in     out     source               destination         
1660K 1093M mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_default (1 references)
pkts bytes target     prot opt in     out     source               destination         
1454  269K MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3          mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            127.0.0.0/8          mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            127.0.0.1            mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
  696 73552 MARK       all  --  *      *       0.0.0.0/0            172.24.210.0/23      mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
   49  4154 MARK       all  --  *      *       0.0.0.0/0            192.168.1.0/24       mark match 0x0/0xff00 MARK xset 0x7f00/0xff00
   13  1146 MARK       all  --  *      *       0.0.0.0/0            192.168.16.0/24      mark match 0x0/0xff00 MARK xset 0x7f00/0xff00

Chain mwan3_interfaces (1 references)
pkts bytes target     prot opt in     out     source               destination         
55989 3963K mwan3_wan2  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
1478 95430 mwan3_wan1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_post (2 references)
pkts bytes target     prot opt in     out     source               destination         
5006 1099K MARK       all  --  *      eth1    0.0.0.0/0           !172.24.210.0/23      mark match ! 0x7f00/0xff00 MARK xset 0x100/0xff00
269K   45M MARK       all  --  *      wlan0   0.0.0.0/0           !192.168.16.0/24      mark match ! 0x7f00/0xff00 MARK xset 0x200/0xff00
1058K  985M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x8000/0x8000 MARK and 0xffff7fff
1802K 1112M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff00

Chain mwan3_pre (2 references)
pkts bytes target     prot opt in     out     source               destination         
1817K 1115M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0xff00
4510 3265K MARK       all  --  eth1   *      !172.24.210.0/23      0.0.0.0/0            mark match ! 0x7f00/0xff00 MARK xset 0x8100/0xff00
505K  495M MARK       all  --  wlan0  *      !192.168.16.0/24      0.0.0.0/0            mark match ! 0x7f00/0xff00 MARK xset 0x8200/0xff00
160K   21M mwan3_default  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
56739 4063K mwan3_interfaces  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
40399 2759K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
  519 81954 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 MARK xset 0x7f00/0xff00

Chain mwan3_rules (1 references)
pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  *      *       192.168.21.0/24      0.0.0.0/0            multiport sports 0:65535 multiport dports 563 mark match 0x0/0xff00 MARK xset 0x1100/0xff00
    0     0 MARK       tcp  --  *      *       192.168.21.0/24      0.0.0.0/0            multiport sports 0:65535 multiport dports 995 mark match 0x0/0xff00 MARK xset 0x1000/0xff00
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            88.154.0.0/16        multiport sports 0:65535 multiport dports 1024:65535 mark match 0x0/0xff00 statistic mode random probability 0.39999999991 MARK xset 0x200/0xff00
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            88.154.0.0/16        multiport sports 0:65535 multiport dports 1024:65535 mark match 0x0/0xff00 statistic mode random probability 0.99999999953 MARK xset 0x100/0xff00
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            77.11.41.0/24        multiport sports 0:65535 multiport dports 1024:65535 mark match 0x0/0xff00 MARK xset 0x1300/0xff00
    0     0 MARK       udp  --  *      *       0.0.0.0/0            112.136.0.0/16       multiport sports 0:65535 multiport dports 5352 mark match 0x0/0xff00 MARK xset 0x1400/0xff00
1047 61742 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 MARK xset 0x1200/0xff00

Chain mwan3_wan1 (1 references)
pkts bytes target     prot opt in     out     source               destination         
  317 24271 MARK       all  --  *      *       172.24.210.132       0.0.0.0/0            MARK xset 0x100/0xff00

Chain mwan3_wan2 (1 references)
pkts bytes target     prot opt in     out     source               destination         
5383  432K MARK       all  --  *      *       192.168.16.108       0.0.0.0/0            MARK xset 0x200/0xff00

Output of "ifconfig" :

br-lan    Link encap:Ethernet  HWaddr C6:3D:C7:8B:94:E7 
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::c43d:c7ff:fe8b:94e7/64 Scope:Link
          inet6 addr: fda1:b0d1:92a3::1/60 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:635711 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1062892 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:109757465 (104.6 MiB)  TX bytes:1003781703 (957.2 MiB)

eth0      Link encap:Ethernet  HWaddr C6:3D:C7:8B:94:E7 
          inet6 addr: fe80::c43d:c7ff:fe8b:94e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:636325 errors:0 dropped:117 overruns:13 frame:0
          TX packets:1062905 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:121257077 (115.6 MiB)  TX bytes:1008035460 (961.3 MiB)
          Interrupt:4

eth0.1    Link encap:Ethernet  HWaddr C6:3D:C7:8B:94:E7 
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:635729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1062892 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:109758527 (104.6 MiB)  TX bytes:1003781703 (957.2 MiB)

eth1      Link encap:Ethernet  HWaddr C4:3D:C7:8B:94:E8 
          inet addr:172.24.210.132  Bcast:172.24.211.255  Mask:255.255.254.0
          inet6 addr: fe80::c63d:c7ff:fe8b:94e8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:756036 errors:0 dropped:0 overruns:0 frame:0
          TX packets:342951 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:526600193 (502.2 MiB)  TX bytes:65248388 (62.2 MiB)
          Interrupt:5

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:21 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2044 (1.9 KiB)  TX bytes:2044 (1.9 KiB)

wlan0     Link encap:Ethernet  HWaddr C4:3D:C7:8B:94:E7 
          inet addr:192.168.16.108  Bcast:192.168.16.255  Mask:255.255.255.0
          inet6 addr: fe80::c63d:c7ff:fe8b:94e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:515197 errors:0 dropped:0 overruns:0 frame:0
          TX packets:272354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:505230319 (481.8 MiB)  TX bytes:54832891 (52.2 MiB)

Output of "cat /etc/config/mwan3" :

# This is a mwan3 example config. For mwan3 to work you will need at least:
#
# - 2 interfaces
# - 2 members
# - 1 policy
# - 1 rule
#
# First define all your wan interfaces. Interface name must match with the
# name used in your network configuration:

config 'interface' 'wan1'
    option 'enabled' '1'
    list 'track_ip' '8.8.4.4'
    list 'track_ip' '8.8.8.8'
    list 'track_ip' '208.67.222.222'
    list 'track_ip' '208.67.220.220'
    option 'reliability' '2'
    option 'count' '1'
    option 'timeout' '2'
    option 'interval' '5'
    option 'down' '3'
    option 'up' '8'
    option 'reroute' '0'

config 'interface' 'wan2'
    option 'enabled' '1'
    list 'track_ip' '8.8.8.8'
    list 'track_ip' '208.67.220.220'
    option 'reliability' '1'
    option 'count' '1'
    option 'timeout' '2'
    option 'interval' '5'
    option 'down' '3'
    option 'up' '8'
    option 'reroute' '0'

# Next define a member and configure metric and weight values for this member.
# Each interface can have multiple member definitions. Give each member a correct
# name (A-Z, a-z, 0-9, "_" and no spaces).

config 'member' 'wan1_m1_w3'
    option 'interface' 'wan1'
    option 'metric' '1'
    option 'weight' '3'

config 'member' 'wan1_m2_w3'
    option 'interface' 'wan1'
    option 'metric' '2'
    option 'weight' '3'

config 'member' 'wan2_m1_w2'
    option 'interface' 'wan2'
    option 'metric' '1'
    option 'weight' '2'

config 'member' 'wan2_m2_w2'
    option 'interface' 'wan2'
    option 'metric' '2'
    option 'weight' '2'

# After that create a routing policy. A routing policy consist of one or more
# members. Give each policy a correct name (A-Z, a-z, 0-9, "_" and no spaces). You
# can create multiple policies, so that it is possible for different traffic to
# have different primary and/or backup interfaces.

config 'policy' 'wan1_only'
    list 'use_member' 'wan1_m1_w3'

config 'policy' 'wan2_only'
    list 'use_member' 'wan2_m1_w2'

config 'policy' 'wan1_wan2_loadbalanced'
    list 'use_member' 'wan1_m1_w3'
    list 'use_member' 'wan2_m1_w2'

config 'policy' 'wan1_pri_wan2_sec'
    list 'use_member' 'wan1_m1_w3'
    list 'use_member' 'wan2_m2_w2'

config 'policy' 'wan2_pri_wan1_sec'
    list 'use_member' 'wan1_m2_w3'
    list 'use_member' 'wan2_m1_w2'

# And to finish the config define your traffic rules. Rules are matched in top to
# bottom order. If you define a rule and it matches, all following rules are ignored.
#
# If the option equalize is set, mwan3 will load-balance each new session to the same
# host. If not set, it will load-balance based on destination.

config 'rule' 'rule1'
    option 'src_ip' '192.168.21.0/24'
    option 'proto' 'tcp'
    option 'dest_port' '563'
    option 'use_policy' 'wan2_only'

config 'rule' 'rule2'
    option 'src_ip' '192.168.21.0/24'
    option 'proto' 'tcp'
    option 'dest_port' '995'
    option 'use_policy' 'wan1_only'

config 'rule' 'rule3'
    option 'dest_ip' '88.154.0.0/16'
    option 'proto' 'tcp'
    option 'dest_port' '1024:65535'
    option 'equalize' '1'
    option 'use_policy' 'wan1_wan2_loadbalanced'

config 'rule' 'rule4'
    option 'dest_ip' '77.11.41.0/24'
    option 'proto' 'tcp'
    option 'dest_port' '1024:65535'
    option 'use_policy' 'wan1_pri_wan2_sec'

config 'rule' 'rule5'
    option 'dest_ip' '112.136.0.0/16'
    option 'proto' 'udp'
    option 'dest_port' '5352'
    option 'use_policy' 'wan2_pri_wan1_sec'

config 'rule' 'rule6'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'wan1_wan2_loadbalanced'

Output of "cat /etc/config/network" :

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'lan'
    option ifname 'eth0.1'
    option type 'bridge'
    option proto 'static'
    option ipaddr '192.168.1.1'
    option netmask '255.255.255.0'
    option ip6assign '60'

config interface 'wan1'
    option ifname 'eth1'
    option proto 'dhcp'
    option metric '10'

config globals 'globals'
    option ula_prefix 'fda1:b0d1:92a3::/48'

config switch
    option name 'rtl8366s'
    option reset '1'
    option enable_vlan '1'
    option blinkrate '2'

config switch_vlan
    option device 'rtl8366s'
    option vlan '1'
    option ports '0 1 2 3 5t'

config switch_port
    option device 'rtl8366s'
    option port '1'
    option led '6'

config switch_port
    option device 'rtl8366s'
    option port '2'
    option led '9'

config switch_port
    option device 'rtl8366s'
    option port '5'
    option led '2'

config interface 'wan2'
    option proto 'dhcp'
    option metric '20'

(Last edited by yanggis on 31 Oct 2013, 07:52)

yanggis,

You have no wan2 interface configured...

I mean in your network config have you defined a wan2 interface section, but it has no ifname option. Please correct your network config first before even trying to start with mwan3. It is not a mwan3 error, but a general network setup error.

Good luck!

Hi Adze, many thanks for your fast response. I've corrected network config already.

Initially, it works well. and the output of "ip rule show" :
0:    from all lookup local
1001:    from all fwmark 0x100/0xff00 lookup 1001
1002:    from all fwmark 0x200/0xff00 lookup 1002
1016:    from all fwmark 0x1000/0xff00 lookup 1016
1017:    from all fwmark 0x1100/0xff00 lookup 1017
1018:    from all fwmark 0x1200/0xff00 lookup 1018
1019:    from all fwmark 0x1300/0xff00 lookup 1019
1020:    from all fwmark 0x1400/0xff00 lookup 1020
32766:    from all lookup main
32767:    from all lookup default
80000:    from 192.168.1.1 nop
80000:    from 127.0.0.1 nop
80000:    from 192.168.16.108 nop
80000:    from 172.24.210.132 nop
90000:    from 192.168.1.1/24 nop
90000:    from 127.0.0.1/8 nop
90000:    from 192.168.16.108/24 nop
90000:    from 172.24.210.132/23 nop
---------------------------------------------------------------------------
However, after about 20 mins, the first ip row is missing.  First default route is also missing. the output of "ip rule show" is as follow:
--------------------------------------------------------------------------
Output of "ip rule show" :

0:    from all lookup local
1002:    from all fwmark 0x200/0xff00 lookup 1002
1017:    from all fwmark 0x1100/0xff00 lookup 1017
1018:    from all fwmark 0x1200/0xff00 lookup 1018
1019:    from all fwmark 0x1300/0xff00 lookup 1019
1020:    from all fwmark 0x1400/0xff00 lookup 1020
32766:    from all lookup main
32767:    from all lookup default
80000:    from 192.168.1.1 nop
80000:    from 127.0.0.1 nop
80000:    from 192.168.16.108 nop
80000:    from 172.24.211.190 nop
90000:    from 192.168.1.1/24 nop
90000:    from 127.0.0.1/8 nop
90000:    from 192.168.16.108/24 nop
90000:    from 172.24.211.190/23 nop

Output of "ip route list table 1001-1099" (1001-1015 = interface tables, 1016-1099 = policy tables) :

1002
default via 192.168.16.1 dev wlan0
1017
default via 192.168.16.1 dev wlan0  metric 1
1018
default via 192.168.16.1 dev wlan0  metric 1
1019
default via 192.168.16.1 dev wlan0  metric 2
1020
default via 192.168.16.1 dev wlan0  metric 1

----------------------------------------------------------------------------------
Is there anyhing wrong? more config info. as follow:
--------------------------------------------------------------------------------

/etc/config/mwan3

config interface 'wan1'
    option enabled '1'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option reroute '1'
    list track_ip '8.8.8.8'

config interface 'wan2'
    option enabled '1'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option reroute '1'
    list track_ip '8.8.4.4'

config member 'wan1_m1_w3'
    option interface 'wan1'
    option metric '1'
    option weight '3'

config member 'wan1_m2_w3'
    option interface 'wan1'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan1_only'
    list use_member 'wan1_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'wan1_wan2_loadbalanced'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m1_w2'

config policy 'wan1_pri_wan2_sec'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_pri_wan1_sec'
    list use_member 'wan1_m2_w3'
    list use_member 'wan2_m1_w2'

config rule 'rule6'
    option dest_ip '0.0.0.0/0'
    option use_policy 'wan1_wan2_loadbalanced'

/etc/config/network
config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'lan'
    option ifname 'eth0.1'
    option type 'bridge'
    option proto 'static'
    option ipaddr '192.168.1.1'
    option netmask '255.255.255.0'
    option ip6assign '60'

config interface 'wan1'
    option ifname 'eth1'
    option proto 'dhcp'
    option metric '10'
   

config globals 'globals'
    option ula_prefix 'fda1:b0d1:92a3::/48'

config switch
    option name 'rtl8366s'
    option reset '1'
    option enable_vlan '1'
    option blinkrate '2'

config switch_vlan
    option device 'rtl8366s'
    option vlan '1'
    option ports '0 1 2 3 5t'

config switch_port
    option device 'rtl8366s'
    option port '1'
    option led '6'

config switch_port
    option device 'rtl8366s'
    option port '2'
    option led '9'

config switch_port
    option device 'rtl8366s'
    option port '5'
    option led '2'

config interface 'wan2'
    option proto 'dhcp'
    option metric '20'
    option ifname 'wlan0'

Thanks for you kindness!

Hi yanggis,

I still believe your network config is not 100% correct. Could you please paste me the main routing table entries? Do you see two default routes? Can you ping google.com from every wan interface as descriped in the wiki?

Thnx!

Hi Adze. You are right.
When I try to ping google.com from both wan interface. Every wan interface work well.

/etc/config$  ping -c 1 -I eth1 www.google.com
PING www.google.com (202.51.247.181): 56 data bytes
64 bytes from 202.51.247.181: seq=0 ttl=55 time=1.521 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.521/1.521/1.521 ms
/etc/config$  ping -c 1 -I eth1 www.google.com
PING www.google.com (202.51.247.155): 56 data bytes
64 bytes from 202.51.247.155: seq=0 ttl=55 time=7.836 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 7.836/7.836/7.836 ms
/etc/config$  ping -c 1 -I wlan0 www.google.com
PING www.google.com (202.51.247.181): 56 data bytes
64 bytes from 202.51.247.181: seq=0 ttl=54 time=8.134 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 8.134/8.134/8.134 ms
/etc/config$  ping -c 1 -I wlan0 www.google.com
PING www.google.com (202.51.247.154): 56 data bytes
64 bytes from 202.51.247.154: seq=0 ttl=54 time=3.098 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 3.098/3.098/3.098 ms

However, when I login my router, mwan3 display eth1 (wan1) is offline.

I am 100% sure wan1 interface works well. However, usually, mwan3 will shows that eth1(wan1) is offline after around 20 mins after resboot my router.

Hi yanggis,

What does logging say after wan1 went offline? Have you tried it without the tracking option?

Adze

Hi Adze,

I'm happily using mwan3 in failover setup for a while now.
However, when I try to increase wan2 ping interval from default 5 to 30 seconds,
my 3g connection gives up on me after a while:

Nov  1 13:55:24 mwan3track: Interface wan2 (3g-wan2) is offline
Nov  1 13:55:24 mwan3: ifdown interface wan2 (3g-wan2)
Nov  1 13:55:26 firewall: removing wan2 (3g-wan2) from zone wan

Could you help me understand what could be wrong?
I have posted my config files below.

Thanks,
Ferry

+++++

# route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    10     0        0 eth0
0.0.0.0         10.64.64.64     0.0.0.0         UG    20     0        0 3g-wan2
10.64.64.64     0.0.0.0         255.255.255.255 UH    0      0        0 3g-wan2
10.111.28.0     192.168.169.1   255.255.255.0   UG    0      0        0 tun0
10.111.29.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan
10.111.31.0     192.168.169.1   255.255.255.0   UG    0      0        0 tun0
192.168.1.0     0.0.0.0         255.255.255.0   U     10     0        0 eth0
192.168.169.0   0.0.0.0         255.255.255.0   U     0      0        0 tun0

+++++

/etc/config/network

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'lan'
    option ifname 'eth1'
    option type 'bridge'
    option proto 'static'
    option netmask '255.255.255.0'
    option ipaddr '10.111.29.254'

config interface 'wan1'
    option ifname 'eth0'
    option proto 'dhcp'
    option defaultroute '1'
    option metric '10'

config interface 'wan2'
    option device '/dev/ttyUSB2'
    option service 'umts'
    option proto '3g'
    option apn 'office.vodafone.nl'
    option pincode '1234'
    option username 'vodafone'
    option password 'vodafone'
    option defaultroute '1'
    option metric '20'
    option peerdns '1'

config interface 'vpn'
    option proto 'none'
    option ifname 'tun0'

+++++

/etc/config/mwan3:

config interface 'wan1'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option reroute '1'

config interface 'wan2'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option down '3'
    option up '8'
    option reroute '1'
    option interval '30'

config member 'wan1_m1_w3'
    option interface 'wan1'
    option metric '1'
    option weight '3'

config member 'wan1_m2_w3'
    option interface 'wan1'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan1_only'
    list use_member 'wan1_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'wan1_wan2_loadbalanced'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m1_w2'

config policy 'wan1_pri_wan2_sec'
    list use_member 'wan1_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_pri_wan1_sec'
    list use_member 'wan1_m2_w3'
    list use_member 'wan2_m1_w2'

config rule 'rule1'
    option dest_ip '0.0.0.0/0'
    option use_policy 'wan1_pri_wan2_sec'

+++++

/etc/config/firewall

root@client44:~# cat /etc/config/firewall

config defaults
    option syn_flood '1'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'REJECT'

config zone
    option name 'lan'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'REJECT'
    option network 'lan'
    option conntrack '1'

config zone
    option name 'wan'
    option input 'REJECT'
    option output 'ACCEPT'
    option forward 'REJECT'
    option masq '1'
    option mtu_fix '1'
    option network 'wan1 wan2'
    option conntrack '1'

config forwarding
    option src 'lan'
    option dest 'wan'

config rule
    option name 'Allow-DHCP-Renew'
    option src 'wan'
    option proto 'udp'
    option dest_port '68'
    option target 'ACCEPT'
    option family 'ipv4'

config rule
    option name 'Allow-Ping'
    option src 'wan'
    option proto 'icmp'
    option icmp_type 'echo-request'
    option family 'ipv4'
    option target 'ACCEPT'

config rule
    option name 'Allow-DHCPv6'
    option src 'wan'
    option proto 'udp'
    option src_ip 'fe80::/10'
    option src_port '547'
    option dest_ip 'fe80::/10'
    option dest_port '546'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-ICMPv6-Input'
    option src 'wan'
    option proto 'icmp'
    list icmp_type 'echo-request'
    list icmp_type 'echo-reply'
    list icmp_type 'destination-unreachable'
    list icmp_type 'packet-too-big'
    list icmp_type 'time-exceeded'
    list icmp_type 'bad-header'
    list icmp_type 'unknown-header-type'
    list icmp_type 'router-solicitation'
    list icmp_type 'neighbour-solicitation'
    list icmp_type 'router-advertisement'
    list icmp_type 'neighbour-advertisement'
    option limit '1000/sec'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-ICMPv6-Forward'
    option src 'wan'
    option dest '*'
    option proto 'icmp'
    list icmp_type 'echo-request'
    list icmp_type 'echo-reply'
    list icmp_type 'destination-unreachable'
    list icmp_type 'packet-too-big'
    list icmp_type 'time-exceeded'
    list icmp_type 'bad-header'
    list icmp_type 'unknown-header-type'
    option limit '1000/sec'
    option family 'ipv6'
    option target 'ACCEPT'

config include
    option path '/etc/firewall.user'

config zone
    option name 'vpn'
    option input 'ACCEPT'
    option forward 'REJECT'
    option output 'ACCEPT'
    option network 'vpn'
    option conntrack '1'

config forwarding
    option dest 'lan'
    option src 'vpn'

config forwarding
    option dest 'vpn'
    option src 'lan'

Hi Adze,

I see mwan3_1.3-0_all.ipk is posted. Is this release the recommended version now for production use, or is it still being tested? I saw in the git notes that you were working on a new way of handing outgoing traffic from the router itself.

Thanks,
Tim

It looks like mwan3track is not getting ping replies three times in a row. Can you confirm that with a tcpdump please?

On a side note, as you use wan2 only as a backup interface, you can remove track_ip options entirely. When you remove the track_ip option, mwan3 assumes that that interface is always up. If wan1 fails, the router will try to send traffic over wan2, even if it is also down.

All versions of mwan3 are test versions . I would not recommend it for business critical applications. But i do recommend the 1.3 version over any previous version.

If you were to use version 1.3, then it is advisable:

- not to use the reroute option anymore
- remove any static routes set to loopback interfaces (loopback trick on previous mwan3 versions)
- add an ip address to the loopback interface in your network config

Hi Adze, thanks. To spell out details:

1.

One should set option reroute '0' for all WAN interfaces

Will traffic originating from the router still fail-over if the primary WAN interface is down, with this set to 0?

2.

This is the default for OpenWrt regarding the loopback interface:

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

Do you mean give the loopback interface an IP address other than 127.0.0.1 through an additional interface alias definition?

3. With ver. 1.3, should server programs on the router such as OpenVPN that need to listen on all WAN interfaces still be configured to listen on an inside interface (such as a LAN interface or loopback interface) and have port-forwards set up, as described in http://wiki.openwrt.org/doc/howto/mwan3?s[]=mwan3#example.1have.openvpn.server.be.accessible.through.multiple.wan.interfaces ?

Thanks,
Tim

That works, but you can leave it out, as 0 is the default value.

Yes, if configured so and you added a loopback ip address.

Interface alias is deprecated according to wiki. You can just add a new interface to your network config:

config interface 'localhost'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '192.168.88.99'
        option netmask '255.255.255.255'

Yes, i would advise so, but is not always necessary.

Why OpenVPN needs this is because of udp, which is connectionless. When you have multiple interfaces, each with their own listener, a packet which originated from interface x, but sent out via interface y will get a reply back on interface y. As udp is connectionless and interface y also has a service listening, it will be handled by that listener. In some cases this can lead to unexpected results.

If you setup OpenVPN to use loopback interface, all packets will be sent and received on the same interface.

(Last edited by Adze on 3 Nov 2013, 15:54)

Any plans to add this package + luci app to OpenWRT officially?
I like to include it in my squashfs.

I'm updating the luci app to remove the reroute option.

Eventually I'll get around to figuring out how to make things centered on the bootstrap skin. It looks a little funny sometimes unless you use the old openwrt.org theme on the GUI.

Hi Adze, thanks for the reply.

So, with v1.3, to get source traffic from the router (e.g. a ping out from the router) to follow mwan3 failover rules, one should:

- leave reroute at 0, the default
- add a new interface bound to the "lo" interface but with a non-127.0.0.1 IP

config interface 'localhost'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '192.168.88.99'
        option netmask '255.255.255.255'

- DON'T add a static route override directing outgoing router traffic to route first to this IP then out from there (the previous loopback interface trick)
- add a mwan3 rule for traffic from 192.168.88.99 (using the above example) to fail-over / load-balance as desired (is this needed?)

Does the OS automatically use the new 192.168.88.99 interface as its source IP when sending out traffic? I through it would normally use the wan interface with the lowest metric if no loopback static route workaround existed?

Thanks,
Tim

Why should you not use the reroute option?

Maybe unrelated, but I noticed when I have two interfaces with the same gateway, one of them will stop working.

e.g If I have WAN2 up and running, and then plug WAN3 in, WAN2 goes offline.

(Last edited by jigglywiggly on 4 Nov 2013, 08:22)

Hi arfett, thanks for your work on the LuCI app for mwan3!

A couple of UI suggestions for your consideration:
- Adze mentioned in another post that if the test IP infrastructure is not defined for an interface, mwan3 will always assume the interface is up. This would be handy to have an option in the LuCI UI to define this setting for an interface.
- For interfaces that are just for emergency use purposes and are metered by packet count (e.g. some cellular data connections), it would be helpful to be able to increase the ping interval to longer than 5 minutes. 10 and 15 minutes would be a nice add.

Thanks,
Tim

Hi,

I discovered bug in new version of mwan3 v mwan3_1.3-0

The openvpn Gargoyle v 1.1.5.11 and mwan3 v mwan3_1.3-0 loadbalancer
rule does not work properly. OpenVpn server on router has not allowed
connection for clients
When I moving back old version mwan3-1.2-20 - file /etc/hotplug.d/iface/15-mwan3
mwan3 loadbalancer and openvpn server working properly.

My system:
Gargoyle PL 1.5.11 (git 5980bc3)                               |
OpenWrt Attitude Adjustment 12.09.1 (r38462)

below there is a log from logread:

Nov  4 21:19:11 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: 213.158.217.85:24865 TLS Error: TLS key
negotiation failed to occur within 60 seconds (check your network
connectivity)
Nov  4 21:19:11 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: 213.158.217.85:24865 TLS Error: TLS
handshake failed
Nov  4 21:19:11 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: 213.158.217.85:24865
SIGUSR1[soft,tls-error] received, client-instance restarting
Nov  4 21:19:12 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: read UDPv4 [ECONNREFUSED]: Connection
refused (code=146)
Nov  4 21:19:15 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: MULTI: multi_create_instance called
Nov  4 21:19:15 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335 Re-using SSL/TLS
context
Nov  4 21:19:15 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335 LZO compression
initialized
Nov  4 21:19:15 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335 Control Channel MTU
parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Nov  4 21:19:15 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335 Data Channel MTU
parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Nov  4 21:19:15 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335 TLS: Initial packet
from xx.xx.xx.xx:61335, sid=d9fcc23f 31f30e8c
Nov  4 21:19:24 router_glowny_64m_16flash authpriv.info
dropbear[4156]: Child connection from 192.168.200.201:2364
Nov  4 21:19:32 router_glowny_64m_16flash authpriv.notice
dropbear[4156]: Password auth succeeded for 'root' from
192.168.200.201:2364
Nov  4 21:19:42 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:52608 TLS Error: TLS key
negotiation failed to occur within 60 seconds (check your network
connectivity)
Nov  4 21:19:42 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:52608 TLS Error: TLS
handshake failed
Nov  4 21:19:42 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:52608
SIGUSR1[soft,tls-error] received, client-instance restarting
Nov  4 21:19:57 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:58853 TLS Error: TLS key
negotiation failed to occur within 60 seconds (check your network
connectivity)
Nov  4 21:19:57 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:58853 TLS Error: TLS
handshake failed
Nov  4 21:19:57 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:58853
SIGUSR1[soft,tls-error] received, client-instance restarting
Nov  4 21:20:09 router_glowny_64m_16flash daemon.warn
dnsmasq-dhcp[2574]: no address range available for DHCP request via
eth0.3
Nov  4 21:20:15 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335 TLS Error: TLS key
negotiation failed to occur within 60 seconds (check your network
connectivity)
Nov  4 21:20:15 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335 TLS Error: TLS
handshake failed
Nov  4 21:20:15 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:61335
SIGUSR1[soft,tls-error] received, client-instance restarting
Nov  4 21:20:17 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: MULTI: multi_create_instance called
Nov  4 21:20:17 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869 Re-using SSL/TLS
context
Nov  4 21:20:17 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869 LZO compression
initialized
Nov  4 21:20:17 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869 Control Channel MTU
parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Nov  4 21:20:17 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869 Data Channel MTU
parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Nov  4 21:20:17 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869 TLS: Initial packet
from xx.xx.xx.xx:52869, sid=c2ac3ecd 0082a654
Nov  4 21:21:06 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: MULTI: multi_create_instance called
Nov  4 21:21:06 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926 Re-using SSL/TLS
context
Nov  4 21:21:06 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926 LZO compression
initialized
Nov  4 21:21:06 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926 Control Channel MTU
parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Nov  4 21:21:06 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926 Data Channel MTU
parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Nov  4 21:21:06 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926 TLS: Initial packet
from xx.xx.xx.xx:56926, sid=9b1f9ae6 f2534bd0
Nov  4 21:21:17 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869 TLS Error: TLS key
negotiation failed to occur within 60 seconds (check your network
connectivity)
Nov  4 21:21:17 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869 TLS Error: TLS
handshake failed
Nov  4 21:21:17 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:52869
SIGUSR1[soft,tls-error] received, client-instance restarting
Nov  4 21:22:06 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926 TLS Error: TLS key
negotiation failed to occur within 60 seconds (check your network
connectivity)
Nov  4 21:22:06 router_glowny_64m_16flash daemon.err
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926 TLS Error: TLS
handshake failed
Nov  4 21:22:06 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:56926
SIGUSR1[soft,tls-error] received, client-instance restarting
Nov  4 21:22:07 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: MULTI: multi_create_instance called
Nov  4 21:22:07 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:54518 Re-using SSL/TLS
context
Nov  4 21:22:07 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:54518 LZO compression
initialized
Nov  4 21:22:07 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:54518 Control Channel MTU
parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Nov  4 21:22:07 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:54518 Data Channel MTU
parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Nov  4 21:22:07 router_glowny_64m_16flash daemon.notice
openvpn(custom_config)[10008]: xx.xx.xx.xx:54518 TLS: Initial packet
from xx.xx.xx.xx:54518, sid=3e7c47a4 ae32142c
root@router_glowny_64m_16flash:~#

there are a logs for openvpn:

cat /etc/config/openvpn

config openvpn 'custom_config'
        option config '/etc/openvpn/server.conf'
        option script_security '3'
        option up '/etc/openvpn.up'
        option down '/etc/openvpn.down'
        option enable '1'

         cat /etc/config/openvpn_gargoyle

config server 'server'
        option internal_ip '10.8.0.1'
        option internal_mask '255.255.255.0'
        option cipher 'BF-CBC'
        option keysize '128'
        option duplicate_cn 'true'
        option redirect_gateway 'false'
        option subnet_access 'true'
        option subnet_ip '192.168.100.0'
        option subnet_mask '255.255.255.0'
        option pool '10.8.0.2 10.8.0.254 255.255.255.0'
        option port '8086'
        option client_to_client 'true'
        option enabled 'true'
        option proto 'udp'

config client 'client'
        option enabled 'false'

config allowed_client 'klient1'
        option id 'klient1'
        option name 'Klient1_xx.xx.xx.xx'
        option remote 'xx.xx.xx.xx'
        option enabled 'true'

Could you look into that problem ?

When I restarted mwan3  there is a info displayed:

I have got restriction for MAC address in my firewall:

/etc/init.d/mwan3 restart
iptables -t filter -A egress_restrictions  -p tcp   -m mac
--mac-source   30:17:C8:63:E2:B9  -j REJECT --reject-with tcp-reset
iptables -t filter -A egress_restrictions   -m mac --mac-source
30:17:C8:63:E2:B9  -j REJECT
iptables -t filter -A egress_restrictions  -p tcp   -m mac
--mac-source   30:17:C8:63:E2:B9  -j REJECT --reject-with tcp-reset
iptables -t filter -A egress_restrictions   -m mac --mac-source
30:17:C8:63:E2:B9  -j REJECT
ERROR: No valid dynamic DNS service configurations defined

There is my mwan3 config file:

cat /etc/config/mwan3

config interface 'wan'
        option enabled '1'
        list track_ip '8.8.8.8'
        list track_ip '8.8.8.8'
        list track_ip '212.77.100.101'
        option reliability '1'
        option count '2'
        option timeout '3'
        option interval '15'
        option down '5'
        option up '3'
        option reroute '1'

config interface 'wan2'
        option enabled '1'
        list track_ip '8.8.8.8'
        list track_ip '8.8.8.8'
        list track_ip '212.77.100.101'
        option reliability '1'
        option count '2'
        option timeout '3'
        option interval '15'
        option down '5'
        option up '3'
        option reroute '1'

config interface 'wan3'
        option enabled '1'
        list track_ip '8.8.8.8'
        list track_ip '8.8.8.8'
        list track_ip '212.77.100.101'
        option reliability '1'
        option down '10'
        option up '3'
        option reroute '1'
        option count '3'
        option timeout '5'
        option interval '3'

config member 'wan_m1_w1'
        option metric '1'
        option weight '10'
        option interface 'wan'

config member 'wan_m2_w2'
        option metric '2'
        option weight '10'
        option interface 'wan'

config member 'wan2_m1_w1'
        option interface 'wan2'
        option metric '1'
        option weight '2'

config member 'wan2_m2_w2'
        option interface 'wan2'
        option metric '2'
        option weight '2'

config member 'wan3_backup'
        option interface 'wan3'
        option metric '3'
        option weight '1'

config policy 'wan_only'
        list use_member 'wan_m1_w1'

config policy 'wan2_only'
        list use_member 'wan2_m1_w1'

config policy 'wan_wan2_loadbalanced'
        list use_member 'wan_m1_w1'
        list use_member 'wan2_m1_w1'

config policy 'wan_pri_wan2_sec_wan3_backup'
        list use_member 'wan_m1_w1'
        list use_member 'wan2_m2_w2'
        list use_member 'wan3_backup'

config policy 'wan2_pri_wan_sec'
        list use_member 'wan2_m1_w1'
        list use_member 'wan_m2_w2'

config policy 'wan_pri_wan2_sec'
        list use_member 'wan_m1_w1'
        list use_member 'wan2_m2_w2'

config rule 'balancer'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option use_policy 'wan_wan2_loadbalanced'
        option equalize '1'

And network

cat /etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config alias
        option interface 'loopback'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.255'

config interface 'lan'
        option ifname 'eth0.1'
        option type 'bridge'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.100.100'
        option dns '208.67.222.222 208.67.220.220'

config interface 'wan'
        option auto '1'
        option ifname 'eth0.2'
        option proto 'static'
        option ipaddr '10.0.0.155'
        option netmask '255.255.255.0'
        option gateway '10.0.0.1'
        option metric '10'
        option peerdns '0'
        option dns '208.67.222.222 208.67.220.220'

config interface 'wan2'
        option auto '1'
        option ifname 'eth0.3'
        option macaddr 'C8:3A:35:4E:4F:CD'
        option proto 'dhcp'
        option peerdns '1'
        option dns '208.67.220.220 208.67.222.222 8.8.8.8 8.8.4.4'
        option defaultroute '1'
        option metric '20'

config interface 'wan3'
        option auto '1'
        option proto '3g'
        option device '/dev/ttyUSB0'
        option apn 'darmowy'
        option service 'umts'
        option mobile_isp 'Polska - Aero2'
        option peerdns '0'
        option dns '208.67.220.220 208.67.222.222 8.8.8.8 8.8.4.4'
        option defaultroute '1'
        option metric '30'

config switch
        option enable '1'
        option name 'rtl8366rb'
        option reset '1'
        option enable_vlan '1'
        option blinkrate '2'

config switch_vlan
        option device 'rtl8366rb'
        option vlan '1'
        option ports '2 3 4 5t'

config switch_vlan
        option device 'rtl8366rb'
        option vlan '2'
        option ports '0 5t'

config switch_vlan
        option device 'rtl8366rb'
        option vlan '3'
        option ports '1 5t'

config interface 'vpn'
        option ifname 'tun0'
        option proto 'none'
        option defaultroute '0'
        option peerdns '0'
        option auto '1'

Regards,
Robert

I overcame the two interfaces on the same gateway problem by just adding a cheap refurb'd e900.

3 modems on a consumer router. Thanks so much, this is too good.

(Last edited by jigglywiggly on 6 Nov 2013, 10:05)

Hi,

I tried to set openvpn and mwan3 according to that manual from wiki openwrt

I am using -  mwan3-1.2-20 - file /etc/hotplug.d/iface/15-mwan3 because previous error reported belove.

>>>>
OpenVPN

    Related pages:
        vpn.openvpn

Example 1: Have OpenVPN Server be accessible through multiple WAN interfaces

If load-balancing between multiple WAN interfaces, it is desirable to have OpenVPN clients be able to connect through all active WAN interfaces.

In a multiple WAN interface failover scenario, OpenVPN will not accept client connections on the secondary WAN interface after a failover, as it started listening on the primary WAN interface when it was started.

The following configuration will allow multiple WAN interface to be used with OpenVPN Server.
Step 1: Listen only on the internal LAN interface

    Configure OpenVPN Server to listen on the internal LAN interface only, not on any WAN interface. The internal LAN interface will not go down or change, and so it provides a stable listening interface for OpenVPN.

vi /etc/openvpn/my-vpn.conf

...
# Which local IP address should OpenVPN
# listen on? (optional)
;local a.b.c.d
## Customization: have OpenVPN listen on the internal LAN interface IP only to allow client re-connections after a WAN interface failover
local 192.168.1.1

...

Step 2: Set up port-forward(s)

    Configure a port-forward on the "wan" source zone to OpenVPN Server listening on the internal LAN interface. The port-forward will be active on every WAN interface and work the same way regardless of what WAN interface is active.
    Create a firewall rule like the following:
        Network > Firewall > Port Forwards
            Name: OpenVPN forward to unchanging inside IP
            Protocol: UDP
            Source zone: wan
            Source IP address: any
            External IP address: any
            External port: 1194 (the default OpenVPN UDP port)
            Internal zone: lan
            Internal IP address: (the internal LAN interface IP address)
            Internal port: 1194
            Enable NAT Loopback: enabled (the default)

Step 3: OpenVPN client and DNS configuration

    If load-balancing between multiple active WAN interfaces, the suggested approach is to register multiple DNS A records for the same DNS name. Clients will use just one of the IPs. As per the OpenVPN man page description of the –remote client parameter, "If host is a DNS name which resolves to multiple IP addresses, one will be randomly chosen, providing a sort of basic load-balancing and failover capability."
    If failing over from a primary to a secondary WAN interface, one approach is to use ddns-scripts to update the IP of the DNS name used by OpenVPN clients

>>>

I have 2 zones in firewal config -  for wan and secondary is wan2 - it is required for specyfic reason of routing.

1 )and I set forwarding in firewal fo wan and wan2 separately - it does not work
2) I removed second forward on port of openvpn for wan2 and it gives connection only on wan - when I stop a wan the second wan - wan2 does not taking care of trafic and routing for openvpn - I am not able to connect on wan2 when wan is down.

3) when it is in original state the coonection to openvpn is avaliable only on primary wan, when I stop wan connection through wan 2 is not allowed - request refused.

mwan3 config

Software versions :

OpenWrt - OpenWrt Attitude Adjustment 12.09.1 (r38462) mwan3 - 1.3-0 luci-app-mwan3 - unknown

Firewall default output policy (must be ACCEPT) :

ACCEPT

Output of "ip route show" :

default via 10.0.0.1 dev eth0.2 proto static metric 10 default via 192.168.200.1 dev eth0.3 proto static metric 20 default via 10.64.64.64 dev 3g-wan3 proto static metric 30 10.0.0.0/24 dev eth0.2 proto static scope link metric 10 10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1 10.64.64.64 dev 3g-wan3 proto kernel scope link src 93.154.214.50 192.168.100.0/24 dev br-lan proto kernel scope link src 192.168.100.100 192.168.200.0/24 dev eth0.3 proto static scope link metric 20

Output of "ip rule show" :

0: from all lookup local 1001: from all fwmark 0x100/0xff00 lookup 1001 1002: from all fwmark 0x200/0xff00 lookup 1002 1003: from all fwmark 0x300/0xff00 lookup 1003 1016: from all fwmark 0x1000/0xff00 lookup 1016 1017: from all fwmark 0x1100/0xff00 lookup 1017 1018: from all fwmark 0x1200/0xff00 lookup 1018 1019: from all fwmark 0x1300/0xff00 lookup 1019 1020: from all fwmark 0x1400/0xff00 lookup 1020 1021: from all fwmark 0x1500/0xff00 lookup 1021 32766: from all lookup main 32767: from all lookup default

Output of "ip route list table 1001-1099" (1001-1015 = interface tables, 1016-1099 = policy tables) :

1001 default via 10.0.0.1 dev eth0.2 1002 default via 192.168.200.1 dev eth0.3 1003 default via 10.64.64.64 dev 3g-wan3 1016 default via 10.0.0.1 dev eth0.2 metric 1 1017 default via 192.168.200.1 dev eth0.3 metric 1 1018 default metric 1 nexthop via 10.0.0.1 dev eth0.2 weight 10 nexthop via 192.168.200.1 dev eth0.3 weight 2 1019 default via 10.0.0.1 dev eth0.2 metric 1 default via 192.168.200.1 dev eth0.3 metric 2 default via 10.64.64.64 dev 3g-wan3 metric 3 1020 default via 192.168.200.1 dev eth0.3 metric 1 default via 10.0.0.1 dev eth0.2 metric 2 1021 default via 10.0.0.1 dev eth0.2 metric 1 default via 192.168.200.1 dev eth0.3 metric 2

Output of "iptables -L -t mangle -v -n | awk '/mwan3/' RS=" :

Chain PREROUTING (policy ACCEPT 674 packets, 89146 bytes) pkts bytes target prot opt in out source destination 13980 2163K mwan3_pre all -- * * 0.0.0.0/0 0.0.0.0/0 1011 129K fwmark all -- * * 0.0.0.0/0 0.0.0.0/0 Chain INPUT (policy ACCEPT 399 packets, 48219 bytes) pkts bytes target prot opt in out source destination 2718 266K mwan3_post all -- * * 0.0.0.0/0 0.0.0.0/0 182 26613 openvpn_down_bw udp -- eth0.2 * 0.0.0.0/0 0.0.0.0/0 udp dpt:8086 Chain OUTPUT (policy ACCEPT 413 packets, 89783 bytes) pkts bytes target prot opt in out source destination 3098 416K mwan3_pre all -- * * 0.0.0.0/0 0.0.0.0/0 179 37049 openvpn_up_bw udp -- * eth0.2 0.0.0.0/0 0.0.0.0/0 udp spt:8086 Chain POSTROUTING (policy ACCEPT 653 packets, 129K bytes) pkts bytes target prot opt in out source destination 14281 2298K mwan3_post all -- * * 0.0.0.0/0 0.0.0.0/0 Chain mwan3_default (1 references) pkts bytes target prot opt in out source destination 16 791 MARK all -- * * 0.0.0.0/0 224.0.0.0/3 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 73 5296 MARK all -- * * 0.0.0.0/0 127.0.0.0/8 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 0 0 MARK all -- * * 0.0.0.0/0 10.0.0.0/24 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 0 0 MARK all -- * * 0.0.0.0/0 10.8.0.0/24 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 0 0 MARK all -- * * 0.0.0.0/0 10.64.64.64 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 16 832 MARK all -- * * 0.0.0.0/0 192.168.100.0/24 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 0 0 MARK all -- * * 0.0.0.0/0 192.168.200.0/24 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 Chain mwan3_interfaces (1 references) pkts bytes target prot opt in out source destination Chain mwan3_post (2 references) pkts bytes target prot opt in out source destination 305 48811 MARK all -- * eth0.2 0.0.0.0/0 !10.0.0.0/24 mark match !0x7f00/0xff00 MARK xset 0x100/0xff00 183 15240 MARK all -- * 3g-wan3 0.0.0.0/0 !10.64.64.64 mark match !0x7f00/0xff00 MARK xset 0x300/0xff00 5222 838K MARK all -- * eth0.3 0.0.0.0/0 !192.168.200.0/24 mark match !0x7f00/0xff00 MARK xset 0x200/0xff00 3975 457K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x8000/0x8000 MARK and 0xffff7fff 16999 2564K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save mask 0xff00 Chain mwan3_pre (2 references) pkts bytes target prot opt in out source destination 17078 2579K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore mask 0xff00 336 41997 MARK all -- eth0.2 * !10.0.0.0/24 0.0.0.0/0 mark match !0x7f00/0xff00 MARK xset 0x8100/0xff00 170 14172 MARK all -- 3g-wan3 * !10.64.64.64 0.0.0.0/0 mark match !0x7f00/0xff00 MARK xset 0x8300/0xff00 3506 404K MARK all -- eth0.3 * !192.168.200.0/24 0.0.0.0/0 mark match !0x7f00/0xff00 MARK xset 0x8200/0xff00 717 62552 mwan3_default all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 456 37507 mwan3_interfaces all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 456 37507 mwan3_rules all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 Chain mwan3_rules (1 references) pkts bytes target prot opt in out source destination 0 0 MARK all -- * * 192.168.100.0/24 0.0.0.0/0 mark match 0x0/0xff00 MARK xset 0x7f00/0xff00 0 0 MARK all -- * * 192.168.100.0/24 0.0.0.0/0 mark match 0x0/0xff00 MARK xset 0x1300/0xff00 14 1061 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 statistic mode random probability 0.166000 MARK xset 0x200/0xff00 61 4916 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 statistic mode random probability 1.000000 MARK xset 0x100/0xff00

Output of "ifconfig" :

3g-wan3 Link encap:Point-to-Point Protocol inet addr:93.154.214.50 P-t-P:10.64.64.64 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:741 errors:0 dropped:0 overruns:0 frame:0 TX packets:813 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:3 RX bytes:61428 (59.9 KiB) TX bytes:67758 (66.1 KiB) br-lan Link encap:Ethernet HWaddr F8:D1:11:B7:41:56 inet addr:192.168.100.100 Bcast:192.168.100.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:62892 errors:0 dropped:5 overruns:0 frame:0 TX packets:61775 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:16082956 (15.3 MiB) TX bytes:15785520 (15.0 MiB) eth0 Link encap:Ethernet HWaddr F8:D1:11:B7:41:56 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:238161 errors:0 dropped:8 overruns:59736 frame:0 TX packets:102093 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:28569978 (27.2 MiB) TX bytes:23811663 (22.7 MiB) Interrupt:4 eth0.1 Link encap:Ethernet HWaddr F8:D1:11:B7:41:56 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8851 errors:0 dropped:0 overruns:0 frame:0 TX packets:13024 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1997937 (1.9 MiB) TX bytes:3805886 (3.6 MiB) eth0.2 Link encap:Ethernet HWaddr F8:D1:11:B7:41:56 inet addr:10.0.0.155 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:671 errors:0 dropped:0 overruns:0 frame:0 TX packets:484 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:67691 (66.1 KiB) TX bytes:91354 (89.2 KiB) eth0.3 Link encap:Ethernet HWaddr C8:3A:35:4E:4F:CD inet addr:192.168.200.100 Bcast:192.168.200.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:19758 errors:0 dropped:0 overruns:0 frame:0 TX packets:22233 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2459429 (2.3 MiB) TX bytes:4800175 (4.5 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:12793 errors:0 dropped:0 overruns:0 frame:0 TX packets:12793 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1207866 (1.1 MiB) TX bytes:1207866 (1.1 MiB) tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.8.0.1 P-t-P:10.8.0.1 Mask:255.255.255.0 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:105 errors:0 dropped:0 overruns:0 frame:0 TX packets:107 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:11053 (10.7 KiB) TX bytes:35300 (34.4 KiB) wlan0 Link encap:Ethernet HWaddr F8:D1:11:B7:41:56 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:54013 errors:0 dropped:0 overruns:0 frame:0 TX packets:53718 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:14881867 (14.1 MiB) TX bytes:15356827 (14.6 MiB)

Output of "cat /etc/config/mwan3" :

config rule 'to_default' option src_ip '192.168.100.0/24' option dest_ip '0.0.0.0/0' option proto 'all' option use_policy 'default' config rule 'failover' option src_ip '192.168.100.0/24' option dest_ip '0.0.0.0/0' option proto 'all' option use_policy 'wan_pri_wan2_sec_wan3_backup' config rule 'balancer' option dest_ip '0.0.0.0/0' option proto 'all' option use_policy 'wan_wan2_loadbalanced' option equalize '1' option src_ip '0.0.0.0/0' config interface 'wan' option enabled '1' list track_ip '8.8.8.8' list track_ip '212.77.100.101' option reliability '1' option count '2' option timeout '3' option down '5' option up '3' option reroute '1' option interval '10' config interface 'wan2' option enabled '1' list track_ip '8.8.8.8' list track_ip '212.77.100.101' option reliability '1' option count '2' option timeout '3' option down '5' option up '3' option reroute '1' option interval '10' config interface 'wan3' option enabled '1' list track_ip '8.8.8.8' list track_ip '212.77.100.101' option reliability '1' option down '10' option reroute '1' option count '3' option timeout '5' option up '10' option interval '20' config member 'wan_m1_w1' option metric '1' option weight '10' option interface 'wan' config member 'wan_m2_w2' option metric '2' option weight '10' option interface 'wan' config member 'wan2_m1_w1' option interface 'wan2' option metric '1' option weight '2' config member 'wan2_m2_w2' option interface 'wan2' option metric '2' option weight '2' config member 'wan3_backup' option interface 'wan3' option metric '3' option weight '1' config policy 'wan_only' list use_member 'wan_m1_w1' config policy 'wan2_only' list use_member 'wan2_m1_w1' config policy 'wan_wan2_loadbalanced' list use_member 'wan_m1_w1' list use_member 'wan2_m1_w1' config policy 'wan_pri_wan2_sec_wan3_backup' list use_member 'wan_m1_w1' list use_member 'wan2_m2_w2' list use_member 'wan3_backup' config policy 'wan2_pri_wan_sec' list use_member 'wan2_m1_w1' list use_member 'wan_m2_w2' config policy 'wan_pri_wan2_sec' list use_member 'wan_m1_w1' list use_member 'wan2_m2_w2'

Output of "cat /etc/config/network" :

config interface 'loopback' option ifname 'lo' option proto 'static' option ipaddr '127.0.0.1' option netmask '255.0.0.0' config alias option interface 'loopback' option proto 'static' option ipaddr '192.168.1.1' option netmask '255.255.255.255' config interface 'lan' option ifname 'eth0.1' option type 'bridge' option proto 'static' option netmask '255.255.255.0' option ipaddr '192.168.100.100' option dns '208.67.222.222 208.67.220.220' config interface 'wan' option auto '1' option ifname 'eth0.2' option proto 'static' option ipaddr '10.0.0.155' option netmask '255.255.255.0' option gateway '10.0.0.1' option metric '10' option peerdns '0' option dns '208.67.222.222 208.67.220.220' config interface 'wan2' option auto '1' option ifname 'eth0.3' option macaddr 'C8:3A:35:4E:4F:CD' option proto 'dhcp' option peerdns '1' option dns '208.67.220.220 208.67.222.222 8.8.8.8 8.8.4.4' option defaultroute '1' option metric '20' config interface 'wan3' option auto '1' option proto '3g' option device '/dev/ttyUSB0' option apn 'darmowy' option service 'umts' option mobile_isp 'Polska - Aero2' option peerdns '0' option dns '208.67.220.220 208.67.222.222 8.8.8.8 8.8.4.4' option defaultroute '1' option metric '30' config switch option enable '1' option name 'rtl8366rb' option reset '1' option enable_vlan '1' option blinkrate '4' config switch_vlan option device 'rtl8366rb' option vlan '1' option ports '2 3 4 5t' config switch_vlan option device 'rtl8366rb' option vlan '2' option ports '0 5t' config switch_vlan option device 'rtl8366rb' option vlan '3' option ports '1 5t' config interface 'vpn' option ifname 'tun0' option proto 'none' option defaultroute '0' option peerdns '0' option auto '0'

mwan3 config

config rule 'to_default'
    option src_ip '192.168.100.0/24'
    option dest_ip '0.0.0.0/0'
    option proto 'all'
    option use_policy 'default'

config rule 'failover'
    option src_ip '192.168.100.0/24'
    option dest_ip '0.0.0.0/0'
    option proto 'all'
    option use_policy 'wan_pri_wan2_sec_wan3_backup'

config rule 'balancer'
    option dest_ip '0.0.0.0/0'
    option proto 'all'
    option use_policy 'wan_wan2_loadbalanced'
    option equalize '1'
    option src_ip '0.0.0.0/0'

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '212.77.100.101'
    option reliability '1'
    option count '2'
    option timeout '3'
    option down '5'
    option up '3'
    option reroute '1'
    option interval '10'

config interface 'wan2'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '212.77.100.101'
    option reliability '1'
    option count '2'
    option timeout '3'
    option down '5'
    option up '3'
    option reroute '1'
    option interval '10'

config interface 'wan3'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '212.77.100.101'
    option reliability '1'
    option down '10'
    option reroute '1'
    option count '3'
    option timeout '5'
    option up '10'
    option interval '20'

config member 'wan_m1_w1'
    option metric '1'
    option weight '10'
    option interface 'wan'

config member 'wan_m2_w2'
    option metric '2'
    option weight '10'
    option interface 'wan'

config member 'wan2_m1_w1'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config member 'wan3_backup'
    option interface 'wan3'
    option metric '3'
    option weight '1'

config policy 'wan_only'
    list use_member 'wan_m1_w1'

config policy 'wan2_only'
    list use_member 'wan2_m1_w1'

config policy 'wan_wan2_loadbalanced'
    list use_member 'wan_m1_w1'
    list use_member 'wan2_m1_w1'

config policy 'wan_pri_wan2_sec_wan3_backup'
    list use_member 'wan_m1_w1'
    list use_member 'wan2_m2_w2'
    list use_member 'wan3_backup'

config policy 'wan2_pri_wan_sec'
    list use_member 'wan2_m1_w1'
    list use_member 'wan_m2_w2'

config policy 'wan_pri_wan2_sec'
    list use_member 'wan_m1_w1'
    list use_member 'wan2_m2_w2'

I resolved the problem with multivan and wan1 and wan2 for openvpn to get connection from wan and wan2

wan has 10.0.0.155

wan2 has 192.168.200.100

lan has 192.168.100.0

openvpnserver is running on 10.0.0.155 with forwarding port to local network 192.168.100.100 for 8086 port

I add rule for forwarding from wan2 to wan to 10.0.0.155 port 8086

config redirect
        option target 'DNAT'
        option src_dport '8086'
        option dest_port '8086'
        option name 'Forward8086'
        option proto 'udp'
        option src 'wan2'
        option dest 'wan'
        option dest_ip '10.0.0.155'

and right now I have multiwan with loadbalancer and failover and I am able to connect simultaneously to my openvpn server through wan and wan2

my client config is set to double remote server

resolv-retry 60
remote [ipserver] 8086
remote [ipserver 2] 8086

dev             tun
proto           udp

Technically the setting for always up is there as you just don't configure any test IP addresses but I will add a setting for that which hides the tracking options if you want the interface always up.

I'm adding 5,10,15,30 minutes to the interval.

I need to fix a bug with the bootstrap theme that hides some of the instructions from the user and I'll be updating the package soon.