Cool. Look forward to trying it out. Will report back

Snowyowlster.

yep. i'm trying 1.09 now and load balancing does seem to be better using multi-threaded download managers.

By allowing this method of balancing, it leaves it up to the client on how many new connections they want to make.

Definitely install 1.0.12 though, as I've made it so you can still use the original balancing method and the new method simultaneously.

The old method offers some benefits, particularly if there is an incompatibility in the load balancing with regards to a particular application or site.

Anyhow, another big change is the probability matrix which I struggled with for a while, but it is now corrected.

Right you are there. I had a problem accessing our cpanel with 1.09. My work around was I routed all traffic going there through one interface.
I'll give 1.0.12 soon and give you feedback as soon as I can.

Thanks!

(Last edited by andyballon on 15 May 2010, 15:58)

I am using a fonera 2.0g and i can reach 3 gateway with  only one interface in different subnets. Can i create virtual devices (eth0.1, eth0.2, eth0.3) using them as wan?

Yes, I recently tested this type of configuration, it seems to work, at least in basic testing w/o QoS.

(Last edited by SouthPawn on 17 May 2010, 21:16)

Hi SouthPawn,

I'm using your scripts and I am loving it, however, I do have a small issue with it. It seems that specifying default route does nothing in my case. 

etc/config/multiwan file:
config 'multiwan' 'config'
        option 'default_route' 'wan'

config 'interface' 'wan'
        option 'icmp_hosts' 'dns'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'dns' 'auto'
        option 'health_interval' '5'
        option 'failover_to' 'fastbalancer'
        option 'weight' '5'

config 'interface' 'wan2'
        option 'icmp_hosts' 'dns'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'dns' 'auto'
        option 'health_interval' '5'
        option 'failover_to' 'fastbalancer'
        option 'weight' '5'

config 'mwanfw'
        option 'wanrule' 'wan'

etc/config/network file:
config 'switch' 'eth0'
        option 'vlan101' '0'
        option 'vlan102' '0'
        option 'vlan103' '0'

config 'interface' 'loopback'
        option 'ifname' 'lo'
        option 'proto' 'static'
        option 'ipaddr' '127.0.0.1'
        option 'netmask' '255.0.0.0'

config 'interface' 'wan'
        option 'ifname' 'eth0.101'
        option 'proto' 'dhcp'
        option 'defaultroute' '0'
        option 'peerdns' '0'

config 'interface' 'wan2'
        option 'proto' 'dhcp'
        option 'ifname' 'eth0.102'
        option 'defaultroute' '0'
        option 'peerdns' '0'

config 'interface' 'lan1'
        option 'ifname' 'eth0.103'
        option 'proto' 'static'
        option 'ipaddr' '192.168.1.1'
        option 'netmask' '255.255.255.0'

Kernel IP routing table
Destination          Gateway         Genmask         Flags      Metric      Ref    Use     Iface
192.168.100.0         *             255.255.255.0      U            0           0        0    eth0.102
192.168.101.0         *             255.255.255.0      U            0           0        0    eth0.101
192.168.1.0            *             255.255.255.0      U            0           0        0    eth0.103
default          192.168.100.254      0.0.0.0          UG           0           0        0    eth0.102
default          192.168.101.252      0.0.0.0          UG           0           0        0    eth0.101
default          192.168.101.252      0.0.0.0          UG           0           0        0    eth0.101

I want wan to be my primary connection and wan2 serve as a backup. However, I get connected to wan2 whenever OpenWrt reboots regardless of what I choose as default route. The only time when I'm routed through wan is when I bring down wan2 manually by running "ifdown wan2" and fail-over kicks in. Please advice if I am doing something wrong and how this issue can be fixed.

Can you give me the output of 'iptables -L MultiWanRules -t mangle -v', 'ip route show table 10', 'iptables -L FW1MARK -t mangle -v', and 'ip rule' ?

Thanks

Thanks for the quick reply, here are the outputs:

root@OpenWrt:/# iptables -L MultiWanRules -t mangle -v
Chain MultiWanRules (2 references)
pkts     bytes     target       prot     opt      in      out     source               destination
2984     206K    FW1MARK     all       --      any    any   anywhere              anywhere            mark match 0x0
    0         0     FastBalancer  all       --      any    any   anywhere              anywhere            mark match 0x0

"ip route show table 10" returns nothing so I ran "ip route" instead, hopefully this is what you were looking for:
root@OpenWrt:/# ip route
192.168.100.0/24 dev eth0.102  src 192.168.100.102
192.168.101.0/24 dev eth0.101  src 192.168.101.164
192.168.1.0/24    dev eth0.103  src 192.168.1.1
default via 192.168.100.254 dev eth0.102
default via 192.168.101.252 dev eth0.101
default via 192.168.101.252 dev eth0.101  src 192.168.101.164

root@OpenWrt:/# iptables -L FW1MARK -t mangle -v
Chain FW1MARK (6 references)
pkts   bytes     target     prot    opt     in     out     source      destination
9343   897K      MARK      all       --    any    any   anywhere    anywhere            MARK set 0x10
9343   897K  CONNMARK   all      --    any    any   anywhere    anywhere            CONNMARK save

root@OpenWrt:/# ip rule
ip: RTNETLINK answers: Operation not supported
ip: dump terminated

There's the problem, it sounds like this would most likely be caused by CONFIG_IP_MULTIPLE_TABLES not being enabled in your kernel.

The reason ifdown wan2 seemingly makes it work is not because it enables failover, but because it disables the last configured interface, leaving the first wan to be the only default route and therefore all traffic goes through it. In the current setup you would see the same behavior with or without the Multi-WAN script.

Hope this helps,
-Craig

I checked my kernel-build configuration and it seems that the feature is enabled and compiled into the kernel image.

Symbol: IP_MULTIPLE_TABLES [=y]   
  ? Prompt: IP: policy routing     
  ?   Defined at net/ipv4/Kconfig:101       
  ?   Depends on: NET [=y] && INET [=y] && IP_ADVANCED_ROUTER [=y]
  ?   Location:
  ?     -> Networking support (NET [=y]) 
  ?       -> Networking options
  ?         -> TCP/IP networking (INET [=y])
  ?   Selects: FIB_RULES [=y]

Any further advices would be greatly appreciated, thanks.

An update on my problem, I deselected and selected the option and recompiled it.  Now it's working nicely, thanks

SouthPawn,

I totally appreciate your efforts with this script. I struggled a little with the installation a while and ultimately followed andyballon's instructions and was able to get it into a working configuration. My goal is to use it to fail-over between 2 moderately failure prone connections.

The hardware is a (classic) WRT54GS, wan is on eth0.1 (option 'vlan1' '0 5') and wan2 is on eth0.2 (option 'vlan2' '1 5'). Both are Ethernet based (when they connect to the router). My goal is to use wan2 when its available and fail back to wan. As such I have my default_route, in multiwan, set to wan2. Everything works great, until wan2 fails (ping timeout). Traffic to sites with already cached DNS entries appears to work fine, but the strangest thing happens.

Calls to the DNS servers on the WAN interface appear to be coming from WAN2's IP address. Obviously this doesn't work properly, as these are 2 different providers with different IP spaces. I've tried all kinds of config changes but nothing I do seems to work around this issue. Failing the connections manually (ifdown or ifup on wan or wan2) doesn't appear to exhibit this behavior.

Thoughts? Thanks!

Hi Craig,

Been using your script for quite some time now and it really works great. But maybe you could help me with a specific set-up? What i want to achieve is the following:

I have three ISP links. Two of them are flat-fee/high bandwidth connections. These two are balanced and works great. The third connection is a 3G low bandwidth/high-cost link. I would like to let the 3G kick-in when both other isp have failed. My firewall config is already setup so that only business critical traffic can traverse over the 3G link.

How do i setup your multiwan script so that the first 2 links are balanced and the third kicks in only after both have failed?

Thank you for your time !

Greetz Adze

Wow this sounds great
Has somebody already experience with this in combination with a Netgear WNDR3700 or TP-Link WR1043ND (I have them both with OpenWRT installed) working ?

And does this method mean that you may use both(or more) connections together to burst download and upload speeds by example ?
My home situation is right now the following:
- 1 ADSL(2) connection of 8 Mbit
- 1 ADSL(2) connection of 8/10 Mbit is comming soon
- 1 ADSL(2) connection of 8/10 Mbit is comming maybe

I do use right now a Linksys VPN-Router(type I don't know at the moment) which is connected with my Alcatel SpeedTouch 546v6 ADSL2 modem, that lot is connected to my Cisco 2924 as my main switch to distribute trough my building(I do live in a company building). On the main floor I have 1 Netgear WNDR3700 with OpenWRT running to provide WiFi access and two TP-Link 1043ND also to provide WiFi access, all three of these WiFi devices to have their own small 'network' at this moment(from the Linksys VPN-Router trough my Cisco everything is connected to the WAN ports on the WiFI devices), but this is not the ideal situation I think.

I think I would like the following devices to use for the new situation:

- Cisco 2924 as my main switch to all the patchpanels trough the building
- Alcatel SpeedTouch 546v6 ADSL2 modem
- "Some" ADSL(2) modem
- "Some" ADSL(2) modem(for maybe the third line)
- TP-Link 1043ND with OpenWRT
- TP-Link 1043ND with OpenWRT
- Nethear WNDR3700 with OpenWRT

I think it is best to move the old Linksys VPN-Router, which is used right now, move away and not to use it anymore.
The idea is I think to use one of the TP-Link's OR the Netgear as my main router, with Multi-WAN Load Balancing thingie on it, and use the other WiFI devices to provide WiFi access trough the building just as plain AccessPoints without routing or NAT-ting them self.

So the question is Which is best solution ? How do I build this properly ?

Thanks in Advance.

Greetz, Priyantha

Hmm, sounds odd... if you could, provide an output of 'iptables -t mangle -L MultiWanDNS -v' and paste what's in the /tmp/resolv.conf.auto as well as 'iptables -t mangle -L FW1MARK' and 'iptables -t mangle -L FW2MARK' while in failover.

option 'weight' 'disable' within /etc/config/multiwan for the third wan, or from LuCI select None for the balance ratio for that interface, this will keep it from being load balanced.

As for failing over to the third only after both interfaces have failed, that can be difficult as really you can only specify one alternate route for each failed wan. (currently)
The 'failover to' option only applies to the custom configured outbound rules and default route, since the routes will be removed from the load balancer tables upon failure.

Unfortunetly I have no experience with any of the hardware mentioned, that question might be best answered in a seperate thread.

Thanks all,
-Craig

(Last edited by SouthPawn on 18 Jun 2010, 15:21)

FYI, I have DNS on eth0.1/wan set to 4.2.2.1 and 4.2.2.3 and eth0.2/wan2 set to 4.2.2.4 and 4.2.2.6 (in /etc/config/network). Mostly for easier reading and troubleshooting. Both interfaces are DHCP, eth0.1/wan is a valid public IP, eth0.2/wan2 is behind NAT and always gets assigned 192.168.100.100. There is just no valid reason for 192.168.100.100 to try and transmit over eth0.1/wan.

Indeed. It would appear it does not have to be in a failover state for wan2's IP to try and talk on wan's interface. Here are some tcptumps from eth0.1 as I first booted everything up tonight and started my IM client (not in a failover state):

root@OpenWrt:~# tcpdump -i eth0.1 -n net 4.2.2 and not icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.1, link-type EN10MB (Ethernet), capture size 96 bytes
19:44:51.245808 IP 192.168.100.100.49859 > 4.2.2.1.53: 6724+ A? omega.contacts.msn.com. (40)
19:44:51.246516 IP 192.168.100.100.49859 > 4.2.2.3.53: 6724+ A? omega.contacts.msn.com. (40)

(Obviously nobody responded to the above requests.) Vs. wan2 which actually can communicate with that IP:

root@OpenWrt:~# tcpdump -i eth0.2 -n net 4.2.2 and not icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.2, link-type EN10MB (Ethernet), capture size 96 bytes
19:44:46.241766 IP 192.168.100.100.49859 > 4.2.2.4.53: 6724+ A? omega.contacts.msn.com. (40)
19:44:51.245103 IP 192.168.100.100.49859 > 4.2.2.6.53: 6724+ A? omega.contacts.msn.com. (40)
19:44:51.247077 IP 192.168.100.100.49859 > 4.2.2.4.53: 6724+ A? omega.contacts.msn.com. (40)
19:44:51.576133 IP 4.2.2.4.53 > 192.168.100.100.49859: 6724 2/0/0 CNAME[|domain]
19:44:51.734450 IP 4.2.2.6.53 > 192.168.100.100.49859: 6724 2/0/0 CNAME[|domain]

root@OpenWrt:~# iptables -t mangle -L MultiWanDNS -v
Chain MultiWanDNS (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   92  7656 FW1MARK    all  --  any    any     anywhere             vnsc-pri.sys.gtei.net 
   92  7656 FW1MARK    all  --  any    any     anywhere             vnsc-lc.sys.gtei.net 
  307 24983 FW2MARK    all  --  any    any     anywhere             vnsc-pri-dsl.genuity.net 
  105  9136 FW2MARK    all  --  any    any     anywhere             vnsc-lc-dsl.genuity.net

Because I like numbers better:

root@OpenWrt:~# iptables -t mangle -L MultiWanDNS -v -n
Chain MultiWanDNS (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  109  9084 FW1MARK    all  --  *      *       0.0.0.0/0            4.2.2.1             
  109  9084 FW1MARK    all  --  *      *       0.0.0.0/0            4.2.2.3             
  467 38351 FW2MARK    all  --  *      *       0.0.0.0/0            4.2.2.4             
  123 10648 FW2MARK    all  --  *      *       0.0.0.0/0            4.2.2.6

root@OpenWrt:~# cat /tmp/resolv.conf.auto 
nameserver 4.2.2.1
nameserver 4.2.2.3
nameserver 4.2.2.4
nameserver 4.2.2.6

When a failure occurs it deletes the values of the nameserver associated with the interface that drops, eg when wan2 goes down, the .4 and .6 nameservers disappear from the list and dnsmasq appears to notice them missing after reading the file. I'm sure you're aware of all that but here is what logread shows me:

Jun 14 19:58:30 OpenWrt user.notice root: [Multi-WAN Notice]: wan2 has failed and is currently offline.
Jun 14 19:58:32 OpenWrt daemon.info dnsmasq[849]: reading /tmp/resolv.conf.auto
Jun 14 19:58:32 OpenWrt daemon.info dnsmasq[849]: using nameserver 4.2.2.3#53
Jun 14 19:58:32 OpenWrt daemon.info dnsmasq[849]: using nameserver 4.2.2.1#53
Jun 14 19:58:32 OpenWrt daemon.info dnsmasq[849]: using local addresses only for domain lan
Jun 14 19:59:18 OpenWrt user.notice root: [Multi-WAN Notice]: wan2 has recovered and is back online!
Jun 14 19:59:20 OpenWrt daemon.info dnsmasq[849]: reading /tmp/resolv.conf.auto
Jun 14 19:59:20 OpenWrt daemon.info dnsmasq[849]: using nameserver 4.2.2.6#53
Jun 14 19:59:20 OpenWrt daemon.info dnsmasq[849]: using nameserver 4.2.2.4#53
Jun 14 19:59:20 OpenWrt daemon.info dnsmasq[849]: using nameserver 4.2.2.3#53
Jun 14 19:59:20 OpenWrt daemon.info dnsmasq[849]: using nameserver 4.2.2.1#53
Jun 14 19:59:20 OpenWrt daemon.info dnsmasq[849]: using local addresses only for domain lan

root@OpenWrt:~# iptables -t mangle -L FW1MARK
Chain FW1MARK (6 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere            MARK set 0x10 
CONNMARK   all  --  anywhere             anywhere            CONNMARK save

root@OpenWrt:~# iptables -t mangle -L FW2MARK
Chain FW2MARK (3 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere            MARK set 0x20 
FW1MARK    all  --  anywhere             anywhere            
CONNMARK   all  --  anywhere             anywhere            CONNMARK save

Like you asked, above was while wan2 is failed. FW2MARK looks a lot more like FW1MARK while wan2 is online:

root@OpenWrt:~# iptables -t mangle -L FW2MARK
Chain FW2MARK (5 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere            MARK set 0x20 
CONNMARK   all  --  anywhere             anywhere            CONNMARK save

Thanks so much for your assistance. Argh, totally monopolizing the thread but I feel really close, I'm just no good with iptables. Hopefully it's a quick, easy, obvious fix. Thanks!

Everything seems normal as far the resolv file, the failover rules, etc..

We must take a look at 'ip rule show', 'ip route show table 10', 'ip route show table 20' and 'ip route show table 123'

Of course you're welcome to mask your public IP if you don't want it displayed.

Thanks,

I'm too exhausted to break them all out, so here you go. (Seems better this way as it imbeds a scroll bar, keeping the post shorter. Replaced my public IP with 1.2.3.4 and gateway with 1.2.3.1. Thanks again!

root@OpenWrt:~# ip rule show
0:    from all lookup local 
10:    from 1.2.3.4 lookup MWAN1 
11:    from all fwmark 0x10 lookup MWAN1 
20:    from 192.168.100.100 lookup MWAN2 
21:    from all fwmark 0x20 lookup MWAN2 
123:    from all fwmark 0x123 lookup LoadBalancer 
32766:    from all lookup main 
32767:    from all lookup default 

root@OpenWrt:~# ip route show table 10
192.168.100.0/24 dev eth0.2  proto kernel  scope link  src 192.168.100.100 
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1 
70.41.160.0/22 dev eth0.1  proto kernel  scope link  src 1.2.3.4 
default via 1.2.3.1 dev eth0.1  proto static  src 1.2.3.4 

root@OpenWrt:~# ip route show table 20
192.168.100.0/24 dev eth0.2  proto kernel  scope link  src 192.168.100.100 
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1 
70.41.160.0/22 dev eth0.1  proto kernel  scope link  src 1.2.3.4 
default via 192.168.100.1 dev eth0.2  proto static  src 192.168.100.100 

root@OpenWrt:~# ip route show table 123
192.168.100.0/24 dev eth0.2  proto kernel  scope link  src 192.168.100.100 
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1 
70.41.160.0/22 dev eth0.1  proto kernel  scope link  src 1.2.3.4 
default via 192.168.100.1 dev eth0.2  proto static

Not sure its normal or not. I don't know if I mentioned I'm not using balancing, which I believe is what 123 is for. I'm questioning if the eth0.2 should be in table 10 and eth0.1 should be in table 20 but I'm sure you'll tell me. Nice to see the troubleshooting methodology. Did anyone end up setting up a wiki? If not, where should it be setup. I'm happy to share what I've learned for the community.

It's a bug... will squish shortly.

Glad to know I'm not going crazy, or at least this doesn't prove without a doubt that I'm going crazy. Thanks for troubleshooting with me and taking care of the problem. If it's going to take a while to fix, I'm not opposed to implementing a workaround or helping to work out the problem. Please let me know if I can help.

Another thing I'll mention is that when I installed this package (or as a result of one of the pre-requisites) my Switch Config menu option disappeared from LuCI. Is this normal/expected? I feel like a couple of other things may have changed (disappeared as well). I just remember playing around with the switch config through the gui before installing the package and when I went back to troubleshoot afterwards, it was gone.

I don't mind managing the config through the /etc/config folder but...

Any idea if this is eventually going to make it into the repository? I think this is valuable enough to be available automatically for install. Anyway, thanks again!

Updated, 1.0.13 should take care of it, let me know if it does not.

Thanks Daniel,
-Craig

SouthPawn, I am unable to download a copy of your script as your FTP server is set to require authentication.

Can you please look into this / send me a copy via other means?

Thanks for your help, and I look forward to trying this out.

~Tyson

Updated.. try again, sorry, it had something in there that shouldn't of been... it's now 1.0.13b.

Ok tried 13b, appears to be doing exactly the opposite. wan's IP on wan2's interface. There might be other problems but I haven't determined if it's something on my end or not.

What are the new table numbers?

*update: I also saw an example of wan2's IP on wan's interface again. Almost seems worse now somehow?

Also I found /etc/iproute2/rt_tables, which answers my questions about the new table names (I noticed a reference to starting at 300 in your first post for the new version, which makes more sense now). So what I'm seeing is 300 for the balancer and 300+eth0.x*10, so eth0.1 = 310, eth0.2 = 320 and so on. Sadly, I don't have any routes in those tables.

root@OpenWrt:/usr/bin# cat /etc/iproute2/rt_tables 
#
# reserved values
#
255    local
254    main
253    default
0    unspec
#
# local
#
#1    inr.ruhep
310     MWAN1
320     MWAN2
300     LoadBalancer
root@OpenWrt:/usr/bin# ip route show table 310
root@OpenWrt:/usr/bin# ip route show table 320
root@OpenWrt:/usr/bin# ip route show table 300

Also, I don't know if I mentioned I'm running Kamakaze, should that cause any problems? I don't have a problem upgrading, it just usually takes me a while to remember the whole process and get everything setup the way I like, so I only do it once a year or so.

Thanks again for working on it!

(Last edited by daniel_bergamini on 19 Jun 2010, 13:01)