ps -w 

To stop it truncating to your terminal width.

I can confirm that ps -w solves the problem - the full line in /usr/sbin/mwan3 is now

if [ -n "$(ps -w | grep mwan3track | grep -v grep | sed '/.*\/usr\/sbin\/mwan3track \([^ ]*\) .*$/!d;s//\1/' | awk '$1 == ("'$1'")')" ]; then

What would be the recommended way to implement "WAN persistence" when having more than 2 WANs ? (i.e. what is currently achieved with the "sticky_odd" and "sticky_even" rules)

I'm on mwan3 1.3 at the moment and I haven't had the same issue. 1.3 is buggier in terms of when an interface goes offline it might say it is still up/keep switching. That and if I apply new rules I have to reset the interfaces manually(but only then). In 1.4 if I apply new rules all my connections stay up. Even if I don't change anything eth1 will turn off eventually for v4 traffic in 1.4.

I'll stay on 1.3 for a while longer to see if the same problem eventually happens. If it doesn't, I'll try 1.4 again and see if it's 100% consistent.

(Last edited by jigglywiggly on 22 Jun 2014, 23:55)

I will add it to the new version asap.. Thnx for finding this!

Here it is. I just replaced real ips with <wan1subnet> and <wan2subnet>:

config 'interface' 'wan1'
    option 'enabled' '1'
    option 'reroute' '1'
config 'interface' 'wan2'
    option 'enabled' '1'
    option 'reroute' '1'
config 'interface' 'wan_openvpn'
    option 'enabled' '1'
    option 'reroute' '1'

config 'member' 'wan1_m1'
    option 'interface' 'wan1'
    option 'metric' '1'
    option 'weight' '1'
config 'member' 'wan2_m2'
    option 'interface' 'wan2'
    option 'metric' '2'
    option 'weight' '1'
config 'member' 'openvpn_m5'
    option 'interface' 'wan_openvpn'
    option 'metric' '5'
    option 'weight' '1'

config 'policy' 'wan1_only'
    list 'use_member' 'wan1_m1'
config 'policy' 'wan2_only'
    list 'use_member' 'wan2_m2'
config 'policy' 'openvpn_only'
    list 'use_member' 'openvpn_m5'

config 'rule' 'rule10'
    option 'dest_ip' '<wan1subnet>'
    option 'use_policy' 'wan1_only'
config 'rule' 'rule20'
    option 'dest_ip' '<wan2subnet>'
    option 'use_policy' 'wan2_only'
config 'rule' 'rule50'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'openvpn_only'

Hi md55,

As you dont use the track_ip option, mwan3 assumes that the interface is always UP, even though it isn't. That is why the router still tries to forward the traffic, even if the tun interface is unaivailable. To fix this add track_ip options to the vpn interface in your mwan3 config.

On a side note, the reroute option is obsolete since version 1.4 and does not do anything. You can remove it.

Grtz Adze.

Hi Adze
I need to always forward all traffic to vpn, even when it's down, but mwan routes traffic via wan1 when vpn is down.
It's a problem for me. I guess when vpn is down mwan uses default routing table for some reason.
How can I deny all traffic except to wan1subnet being routed via wan1?
Btw I still use mwan3_1.3

How to make that a specific site was opened only through one interface ?
Please help.

You have two options:
1. Follow the guide in the wiki - http://wiki.openwrt.org/doc/howto/mwan3
Use the IP address(es) of the website as the destination in MWAN3 rules.

2. Create static routes in OpenWrt directing the IP address(es) of the website out the desired interface.

(Last edited by arfett on 30 Jun 2014, 22:51)

Hi Adze,

i have a little problem, might be as well just understanding issue, with the mwan3 package in use with strongswan 5.1.3 net-to-net.
mwan is configured with 2 wan interfaces, load balancing is default.
After starting strongswan, routes and firewall rules are set up to make the ipsec tunnel accessible.

strongswan does initialize the ipsec tunnel via wan without problems, it is possible to ping the remote subnet from openwrt but packets from local subnet get lost somewhere...

It seems, packets are sent out from local subnet to remote subnet correctly (i can see the ping arriving on remote) but the reply is not forwarded.

After stopping mwan3 everything works as expected, so my suspicion is some rule that can not match the returning packet and dropping it.

Any ideas are very much appreciated

[edit]
version is latest from github, 1.4-20
[/edit]

Details on Configuration:
/etc/config/mwan3 (pretty much default ...)


config rule 'sticky_even'
    option src_ip '0.0.0.0/0.0.0.1'
    option dest_port '443'
    option proto 'tcp'
    option use_policy 'wan_umts'

config rule 'sticky_odd'
    option src_ip '0.0.0.1/0.0.0.1'
    option dest_port '443'
    option proto 'tcp'
    option use_policy 'umts_wan'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

config interface 'wan'
    option enabled '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    list track_ip '8.8.8.8'
    option reliability '1'

config interface 'umts'
    option enabled '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option reliability '1'
    list track_ip '8.8.8.8'
    

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'umts_m1_w2'
    option interface 'umts'
    option metric '1'
    option weight '2'

config member 'umts_m2_w2'
    option interface 'umts'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'

config policy 'umts_only'
    list use_member 'umts_m1_w2'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'umts_m1_w2'

config policy 'wan_umts'
    list use_member 'wan_m1_w3'
    list use_member 'umts_m2_w2'

config policy 'umts_wan'
    list use_member 'wan_m2_w3'
    list use_member 'umts_m1_w2'

Output of ip rule show (with mwan3 running)

root@box:~# ip rule show
0:    from all lookup local 
220:    from all lookup 220 
1002:    from all iif 3g-umts lookup main 
2002:    from all fwmark 0x200/0xff00 lookup 2 
2254:    from all fwmark 0xfe00/0xff00 unreachable
32766:    from all lookup main 
32767:    from all lookup default 

ip route show table  220

root@box:~# ip route show table 220
{remote subnet} dev ipsec0  proto static  src {ip of box}

(Last edited by gh0st on 1 Jul 2014, 09:57)

Hi Ghost,

Please try and add this rule to your mwan3 config. Place it on top of all other rules:

config rule 'ipsec'
    option dest_ip '{remote subnet}'
    option use_policy 'default'

On a sdie note: I see that you use custom route table 220. This can be a problem as mwan3 might wipe this table when it's stopped. If you use a number higher then 255 you should be fine...

(Last edited by Adze on 1 Jul 2014, 10:10)

does not work, please give an example

Let's turn it around. Please show your config and explain in detail what you want to achieve and what does not work.

want to make a particular site was opened only via wan2
and do so

Hi Vahe91,

You're on the right track! That is indeed how you create a rule!

but this rule not working

Is the order of rules right?

You have to give me more info then "it is not working", before i can help you. I'm not here to take you by the hand and solve your problems. So please post your complete config or ask questions as detailed as possible.

unfortunately I do not know English so well in order to explain in detail
for example, I want to open the site www.2ip.ru only through wan2
here is my configuration
-------------------
config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan2'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option enabled '1'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'
    list use_member 'wan3_m1_w4'
    list use_member 'wan4_m1_w5'

config policy 'wan_wan2'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_wan'
    list use_member 'wan_m2_w3'
    list use_member 'wan2_m1_w2'

config rule 'sticky_even'
    option src_ip '0.0.0.0/0.0.0.1'
    option dest_port '443'
    option proto 'tcp'
    option use_policy 'wan_wan2'

config rule 'sticky_odd'
    option src_ip '0.0.0.1/0.0.0.1'
    option dest_port '443'
    option proto 'tcp'
    option use_policy 'wan2_wan'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

config interface 'wan3'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '8.8.4.4'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '3'

config interface 'wan4'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '8.8.4.4'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '3'

config member 'wan3_m1_w4'
    option interface 'wan3'
    option metric '1'
    option weight '4'

config member 'wan4_m1_w5'
    option interface 'wan4'
    option metric '1'
    option weight '5'

config policy 'wan3_only'
    list use_member 'wan3_m1_w4'

config policy 'wan4_only'
    list use_member 'wan4_m1_w5'

config rule 'rule_1'
    option dest_ip '188.40.74.0/26'
    option proto 'all'
    option use_policy 'wan2_only'
--------------------------

what else to do, because with such a configuration 2ip.ru opens as always, and not only through wan2 ?

Please add the following rule on top of all other rules:

config rule 'www.2lp.ru'
    option dest_ip '62.109.26.77'
    option use_policy 'wan2_only'

Remember that order of rules important. In your posted config the last rule will never be hit.

in my example 2ip.ru, not 2lp.ru
Thanks, everything turned out just needed to move the rule to the first line

Should I be concerned by the following timeout error "Timeout waiting for older hotplug processes to finish. ifup interface wan (eth0.2) aborted" ?

It occured while stress-testing my xDSL line (link saturation causing ping timeout and mwan3 marking both WANs as being down).

Mon Jun 30 01:03:34 2014 user.info mwan3track: Lost 1 ping(s) on interface wan2 (eth0.3)
Mon Jun 30 01:03:38 2014 user.info mwan3track: Lost 2 ping(s) on interface wan (eth0.2)
Mon Jun 30 01:03:54 2014 user.notice mwan3track: Interface wan2 (eth0.3) is online
Mon Jun 30 01:03:56 2014 user.notice mwan3: ifup interface wan2 (eth0.3)
Mon Jun 30 01:03:57 2014 user.notice mwan3: ifup interface wan2 (eth0.3)
Mon Jun 30 01:04:14 2014 user.notice mwan3track: Interface wan (eth0.2) is online
Mon Jun 30 01:04:15 2014 user.notice mwan3: ifup interface wan (eth0.2)
Mon Jun 30 01:05:17 2014 user.warn mwan3: Timeout waiting for older hotplug processes to finish. ifup interface wan (eth0.2) aborted
Mon Jun 30 01:25:14 2014 user.info mwan3track: Lost 2 ping(s) on interface wan (eth0.2)
root@OpenWrt:~#

PS: OpenWRT could use a "qosmon" util, like the one offered by Gargoyle, that will tune up/down the line bandwidth based on latency, for those of us who don't enjoy a "guaranteed" line speed of our xDSL line

I just wanted to add that while in your particular case it was rather easy (since www.2ip.ru resolves to only 2 IPs), there are other cases where policy-routing a Website via a certain WAN might be quite difficult, since some popular Websites can resolve to 10s or 100s of different IPs (or CDNs).

(Last edited by kpv on 1 Jul 2014, 15:28)

It's hard to tell afterwards why this happened. Only thing i can say is that there was another hotplug process running, that started before the aborted one, and took more than the time-out of 1 minute to finish (or did not finish at all). I'm curius to what this might caused it..

Can you reproduce this on command? If so, could you list the processes running just before it times out. I know this would not be easy, but it could shed some light on the cause...

I don't know if I can reproduce this error, but I'll be keeping an eye on the logs.

I assume that the other hotplug event would be related to the up/down of the other WAN link. Note: since both WANs in my test setup go out via the same Internet link, they both go down together and come back up again together.

PS: As I've written you, I've added -w to all invocations of iptables by mwan3, I'm not sure if that might be related ...