Hi!

I have 3 PC-s at home, and i would like to share the bandwith. I want low latencies even if other computers are downloading. Unfortunately there is little or confusing documentation about this topic. Some of the documentation claims it is not possible to shape ingress (incoming) traffic, but (i think) that is not true.

I had to install the dd-wrt firmware and learned the qos concepts it uses.

I assign static IP addresses to the computers with dhcp (dnsmasq, /etc/ethers).

This picture helped me a lot: http://upload.wikimedia.org/wikipedia/c … ecture.png
http://openwrt.org/OpenWrtDocs/Configuration

I have a WRT54GS. Here is what i did:
Install Whiterussion RC2
ipkg update
ipkg install kmod-sched
ipkg install kmod-iptables-extra
ipkg install tc
ipkg install iptables-extra

My script is attached below.

TODO:
It would be nice to have IMQ - with that it is possible to shape incoming traffic for both WLAN and LAN. Shaping is a per device thing, so this script only shapes the wired ethernet.
Outgoing traffic should be shaped too (possibly on the ppp0 interface)

Please share your thoughts!

---------------------
#!/bin/sh

# to check the status of the qos stuff:
#  iptables -t mangle -L
#  tc -s qdisc show dev eth1
#  tc -s class show dev eth1

. /etc/functions.sh

#It is ok for WRT54-G/GS, it may differ for other
#http://openwrt.org/OpenWrtDocs/Configuration
DOWN_IFACE=vlan0

# uplink bandwidth
# specified in kbits (about 90% of actual max uplink rate)
UP_RATE=100
DOWN_RATE=800

#Inserting various kernel modules
insmod ipt_TOS
insmod ipt_tos
insmod ipt_length
insmod sch_prio
insmod sch_red
insmod sch_htb
insmod sch_sfq
insmod sch_ingress
insmod cls_tcindex
insmod cls_fw
insmod cls_route
insmod cls_u32

echo alma0

# Clear all traffic control things to start from a clean state
tc qdisc del dev $DOWN_IFACE root

echo alma1

tc qdisc add dev $DOWN_IFACE root handle 1: htb default 1

#This is for the intra-LAN traffic
tc class add dev $DOWN_IFACE parent 1: classid 1:1 htb rate 10000kbit burst 6k cburst 2624b
tc class add dev $DOWN_IFACE parent 1:1 classid 1:2 htb rate ${DOWN_RATE}kbit ceil ${DOWN_RATE}kbit

#Class for the first computer minimum 250kbit maximum all
tc class add dev $DOWN_IFACE parent 1:2 classid 1:10 htb rate 250kbit ceil ${DOWN_RATE}kbit burst 6k cburst 2624b
#Class for the second computer
tc class add dev $DOWN_IFACE parent 1:2 classid 1:20 htb rate 250kbit ceil ${DOWN_RATE}kbit burst 6k cburst 2624b
#Class for the third computer
tc class add dev $DOWN_IFACE parent 1:2 classid 1:30 htb rate 250kbit ceil ${DOWN_RATE}kbit burst 6k cburst 2624b
#Class for any other computer
tc class add dev $DOWN_IFACE parent 1:2 classid 1:40 htb rate 50kbit ceil ${DOWN_RATE}kbit burst 6k cburst 2624b

echo alma2

#See this webpage for what these numbers mean:
#http://lartc.org/howto/lartc.adv-qdisc.red.html
#Maybe i should play with them later
tc qdisc add dev $DOWN_IFACE parent 1:10 handle 10: red limit 400000b min 10000b max 50000b avpkt 1000 burst 10 ecn
tc qdisc add dev $DOWN_IFACE parent 1:20 handle 20: red limit 400000b min 10000b max 50000b avpkt 1000 burst 10 ecn
tc qdisc add dev $DOWN_IFACE parent 1:30 handle 30: red limit 400000b min 10000b max 50000b avpkt 1000 burst 10 ecn
tc qdisc add dev $DOWN_IFACE parent 1:40 handle 40: red limit 400000b min 10000b max 50000b avpkt 1000 burst 10 ecn

# I could use SFQ, but RED seems to be better

#tc qdisc add dev $DOWN_IFACE parent 1:10 handle 10: sfq perturb 10
#tc qdisc add dev $DOWN_IFACE parent 1:20 handle 20: sfq perturb 10
#tc qdisc add dev $DOWN_IFACE parent 1:30 handle 30: sfq perturb 10
#tc qdisc add dev $DOWN_IFACE parent 1:40 handle 40: sfq perturb 10

echo alma3

#Flush the mangle table
iptables -t mangle -F

#Not really sure exactly what traffic comes to the POSTROUTING chain,
#but it works

#Mark all incoming, outgoing traffic (should separate later!)

#Default
iptables -t mangle -A POSTROUTING -s 192.168.1.0/24 -d ! 192.168.1.0/24 -j MARK --set-mark 0x40
iptables -t mangle -A POSTROUTING -d 192.168.1.0/24 -s ! 192.168.1.0/24 -j MARK --set-mark 0x40
#This is my first computer
iptables -t mangle -A POSTROUTING -d 192.168.1.50 -s ! 192.168.1.0/24 -j MARK --set-mark 0x10
iptables -t mangle -A POSTROUTING -s 192.168.1.50 -d ! 192.168.1.0/24 -j MARK --set-mark 0x10

#Second computer
iptables -t mangle -A POSTROUTING -d 192.168.1.51 -s ! 192.168.1.0/24 -j MARK --set-mark 0x20
iptables -t mangle -A POSTROUTING -s 192.168.1.51 -d ! 192.168.1.0/24 -j MARK --set-mark 0x20

#Third computer
iptables -t mangle -A POSTROUTING -d 192.168.1.52 -s ! 192.168.1.0/24 -j MARK --set-mark 0x30
iptables -t mangle -A POSTROUTING -s 192.168.1.52 -d ! 192.168.1.0/24 -j MARK --set-mark 0x30

echo alma4

#We assign the traffic to classes using the marks
tc filter add dev $DOWN_IFACE protocol ip parent 1: handle 0x10 fw classid 1:10
tc filter add dev $DOWN_IFACE protocol ip parent 1: handle 0x20 fw classid 1:20
tc filter add dev $DOWN_IFACE protocol ip parent 1: handle 0x30 fw classid 1:30
tc filter add dev $DOWN_IFACE protocol ip parent 1: handle 0x40 fw classid 1:40

I'm really new to OpenWRT, but I can give you this advice:  Check out the 'wondershaper' script.  It shapes the WAN traffic (incoming AND outgoing) and all you really have to configure is up and downstream bandwidth.

It's also already been packaged w/ ipkg.  Check the package tracker for the download info.

You obviously know more about traffic shaping than I do... one thing I would like to figure out is how to modify the wondershaper script to set max bandwidth for certain protocols/ports/comps/whatever.  I'll figure it out one of these days!

Wondershaper is ok if you have a single computer, but it does not shape nor shares the incoming traffic.

My script shares the bandwidth equally between the 3 computers.

(Last edited by nug on 26 Aug 2005, 08:28)

*bows to your superior knowledge, and stumbles off to remove his foot from his mouth*

*goes off to research some more stuff*

hmmm, are you sure your script shares the download bandwidth?
I don't see anything that shapes the ingress traffic...

Also you have set htb rates that are higher than the ceil rates. I'm not sure if it works...

the theory of traffic shapeing inbound traffic.

if your ISP will also run QOS on their end of the link you can readily shape incomeing traffic.

if they won't (or won't give you control of it) then you are very limited in what you can do with inbound traffic.

since you don't directly control when the packets are sent you can't prevent the inbound traffic from completely flooding you link (even if the traffic is all going to be denied by your firewall settings, it has to get to the firewall before you can decide you don't care about it)

however there is a trick that can be pulled.

since TCP adapts it's sending rate to the bandwidth of the link, if you don't acknowledge the incomeing packets as quickly as you can fool the remote side into thinking that it's on a slower link, this doesn't do any good for UDP traffic, and your gateway device needs to take a lot of things into account when decideing how long to delay the acknowledgement (and the machine on the far side will periodicly try to increase the speed so you will end up useing a little more bandwidth then you expect), and you need to account for how many connections are in use to the one internal destination (or your limiting gets bypassed by just opening up more connections in parallel)

most traffic shapeing options don't go to this sort of effort, they just limit the outbound traffic and don't try to play these more complicated games with the inbound stuff. This is why in most cases you need control of the settings on both ends of the link so that things don't get sent on the slow link unless you want them there.

I had to go lookup what the sch_ingress module does.

it controlls the incomeing traffic rate by dropping packets when they exceed the threashold that you set.

for TCP this forces the sender to send the packet again (useually after a 30 second delay) and so it indirectly slows things down (but if you have short connections it can cause lengthy delays for those short connections)

for UDP traffic (most games, many streaming audio protocols) dropping the packets just means that they don't get to the machine on the inside, they still eat up your bandwidth, and the sender doesn't get any indication that they should slow down, so the usefullness of this is limited.

I'm really a newbie and still waiting for the good news of a new firmware that support WRT54G v4. Currently trying to absorb as much information as I can.
The reason I'd like to try the third party firmware is because of this:
- My house is networked so that me and all the roomates get internet access both wired and wirelessly.
- Me and my brother do a lot of gaming and during certain times, connection becomes really laggy and my suspicion is because the roomate is downloading something. Always when he is not at home the connection is fine.
- I can't download anything big when my brother is playing because it will lag really bad.

I was thinking that using the 3rd party firmware, I can prioritize the game connection and allocate only certain bandwidth to the roommate so that we won't feel the lag when he is downloading. Can this be achieved?

Yes, it really works for TCP connections. I have not seen any ,,30 sec delay'', just well shaped downloads.

"RED statistically drops packets from flows before it reaches its hard limit. This causes a congested backbone link to slow more gracefully, and prevents retransmit synchronization. This also helps TCP find its 'fair' speed faster by allowing some packets to get dropped sooner keeping queue sizes low and latency under control. The probability of a packet being dropped from a particular connection is proportional to its bandwidth usage rather than the number of packets it transmits."

However:

"RED isn't a cure-all for this, applications which inappropriately fail to implement exponential backoff still get an unfair share of the bandwidth, however, with RED they do not cause as much harm to the throughput and latency of other connections."

My main problem was, that the P2P and other downloads overloaded our incoming line, so gaming, web browsing, ssh was slow (big latency). This solution solved my problems. Yes, it can not shape incoming UDP connections, but that is not a problem (games do not need big bandwidth).

The high rated (default) HTB class is for LAN-LAN traffic. Class 1:2 is for the incoming traffic from the net.

I have not seen any sample scripts like this (only in the DD-WRT firmware), but it seems to work.

Could you please test it, so i can get the script better? Lets make the world better.

(Last edited by nug on 28 Aug 2005, 06:56)

If you are a newbie, you should give DD-WRT or SVEASOFT Talisman (20$) a try. They are more user friendly than OpenWRT and maybe can done the job for you.

I tried sch_ingress, but i could not find any good documentation. I figured out that i can not use it as a qdisc under a HTB class. So it can shape traffic, but can not do any borrowing between the classes.
RED is a similar technic, but can be used as a qdisc.

(maybe i'm totally wrong)

just limiting your upload will lead to huge results. it prevents your modem fifo to fill and raise the latency up to seconds.

Seeing your script, I still cannot understand why it works, because you are shaping things at the download rate, not at the upload rate.

The wondershaper does all these things: it shapes both down (with sch_ingress) and up traffic. You are wrong when you say it is only for one workstation as it uses SFQ (FQ means fair queing between connections)

RED is a very low cost algorithm for very high speed routers. SFQ should be much better.

You should read all the lartc.org...

david lang already explanied it

Wondershaper does not share the bandwith between multiple computer.
Yes, I will try SFQ, too.

(Last edited by nug on 29 Aug 2005, 18:30)

Why do you say that?  The wondershaper is nothing more than htb and sfq. it is almost the same as your script but use sfq instead of red.

From my experience, it does help greatly with multiple computers.  Now, my experience is only from using it for about a week, but I can make vonage phone calls w/ no problems while downloading using edonkey, with no restrictions set in edonkey.  previously I had to limit the upload speed to 50k using the settings in edonkey, which slowed down my downloads to less than max.  Now I can let it go, and wondershaper is handling that beautifully.

I can also surf the web with any slowdown being unoticeable, as well as IRC.

My understanding is that SFQ gives all connections a fair portion of downloads...  And the more bandwidth a connection is using, the more packets are dropped from it to bring the total downloads into line with the max if necessary, so while there is no individual queue for each computer, it does seem to allocate bandwidth fairly, while allowing one computer to consume more if none of the others are using it.

Other than the fact that occassionally my ssh connections have a bit of lag, I'm pretty happy with it, overall.

I have a little problem with wondershaper, cohabitation between voip and mldonkey is still difficult.
voip is still a little choppy with p2p running.
I would like to make a

here is my status:

     qdisc htb 1: r2q 1 default 20 direct_packets_stat 1
qdisc htb 1:      :      15478 (123.826400 kbit/s) sent=368226601  :
      Sent 368226601 bytes 1741347 pkts (dropped 0, overlimits 220919 requeues 0)
      backlog 83p
     qdisc sfq 10: parent 1:10 limit 128p quantum 1514b perturb 10sec
qdisc sfq 10:     :        940 (  7.520800 kbit/s) sent=48197246  :
      Sent 48197246 bytes 843978 pkts (dropped 0, overlimits 0 requeues 0)
     qdisc sfq 20: parent 1:20 limit 128p quantum 1514b perturb 10sec
qdisc sfq 20:     :         50 (  0.404800 kbit/s) sent=2946942  :
      Sent 2946942 bytes 32917 pkts (dropped 0, overlimits 0 requeues 0)
     qdisc sfq 30: parent 1:30 limit 128p quantum 1514b perturb 10sec
qdisc sfq 30:     :      14487 (115.900800 kbit/s) sent=317082359  :
      Sent 317082359 bytes 864451 pkts (dropped 0, overlimits 0 requeues 0)
      backlog 83p
     qdisc ingress ffff: ----------------
qdisc ingress ffff: :      14162 (113.302400 kbit/s) sent=809220903  :
      Sent 809220903 bytes 1428396 pkts (dropped 0, overlimits 0 requeues 0)
     class htb 1:1 root rate 115Kbit ceil 115Kbit burst 1613b cburst 1613b
class htb 1:1     :      14514 (116.115600 kbit/s) sent=368185791  :
      Sent 368185791 bytes 1741263 pkts (dropped 0, overlimits 0 requeues 0)
      rate 14497bit 46pps
      lended: 677720 borrowed: 0 giants: 0
      tokens: -143918 ctokens: -143918

     class htb 1:10 parent 1:1 leaf 10: prio 0 rate 80Kbit ceil 115Kbit burst 6Kb cburst 1613b
class htb 1:10    :        940 (  7.520800 kbit/s) sent=48197246  :
      Sent 48197246 bytes 843978 pkts (dropped 0, overlimits 0 requeues 0)
      rate 1040bit 18pps
      lended: 843978 borrowed: 0 giants: 0
      tokens: 624230 ctokens: 111553

     class htb 1:20 parent 1:1 leaf 20: prio 1 rate 23Kbit ceil 115Kbit burst 1601b cburst 1613b
class htb 1:20    :         50 (  0.404800 kbit/s) sent=2946942  :
      Sent 2946942 bytes 32917 pkts (dropped 0, overlimits 0 requeues 0)
      rate 178bit 1pps
      lended: 32774 borrowed: 143 giants: 0
      tokens: 553495 ctokens: 111553

     class htb 1:30 parent 1:1 leaf 30: prio 2 rate 11Kbit ceil 115Kbit burst 1600b cburst 1613b
class htb 1:30    :      14487 (115.900800 kbit/s) sent=317082359  :
      Sent 317082359 bytes 864451 pkts (dropped 0, overlimits 0 requeues 0)
      rate 13306bit 26pps backlog 83p
      lended: 186791 borrowed: 677577 giants: 0
      tokens: -1933831 ctokens: -29201

There is a wiki article: OpenWrt: QoS (Qualitiy of Service)