Hi all, Hi all,
I've been using linux and OpenWRT for a good few years, and any problems I've had have always been solved by someone previously, often on this great forum. However, this one has me stumped, so I thought I'd better ask the experts.

I've got a small LAN at home, consisting of my media pc (Fedora 6), a Dreambox, an Asus WL500g running OpenWRT Kamikaze(7.09, acting as a file server, with the attached usb storage), an old laptop running Slackware (wireless), and a Linksys WRT54GS (also OpenWRT Kamikaze 7.07). This setup worked great (or so I thought) for about 6 months, until I tried to stream video between the devices (specifically between the PC and Dreambox). I originally had the asus acting as a wireless bridge to the main Linksys router, so I put the 2MB/s(ish) transfer rate down to wireless networking problems.

I recently bought a couple of Powerline adapters to try and improve the streaming. They didn't . I was still getting 2MB/s max transfer. So I thought I'd investigate. I've now found that ALL the devices can only communicate with each other at a maximum of about 20mbps. I've got this maximum using both 'iperf' and by netcatting /dev/zero between devices, as well as ftp'ing, scp'ing and NFS'ing between devices.

Even single hops (e,g, PC to router) will only get 2MB/s. I had a go at changing things like TCP window sizes through /proc/sys/..., in line with what I'd found from google, but I get no difference. I think something else is wrong, and I've no idea what it could be. Might cable quality be a factor? I've connected everything to everything, using about 5 different cables, and everything still gives the same 2MB/s max throughput.

I'm sure there must be something stupid I've missed, but for the life of me I can't think what it is.

I've run ethtool on the PC, Asus and Linksys, and it comes up with Connection speed 100mbps for all the devices.
For instance, here's the output from 'ethtool' on the Linksys WRT54GS:

Settings for eth0:
        Supported ports: [ MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 30
        Transceiver: external
        Auto-negotiation: on
        Current message level: 0x000000ff (255)
        Link detected: yes

.
Is there anything wrong with the above output?

Also, ifconfig doesn't show anything strange (as far as I can tell). Here's the output from the Asus's ifconfig:

br-lan    Link encap:Ethernet  HWaddr 00:11:2F:22:48:59
          inet addr:192.168.1.137  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:185665 errors:0 dropped:0 overruns:0 frame:0
          TX packets:115445 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:16629441 (15.8 MiB)  TX bytes:113111465 (107.8 MiB)

eth0      Link encap:Ethernet  HWaddr 00:11:2F:22:48:59
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:185684 errors:0 dropped:0 overruns:0 frame:0
          TX packets:115439 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:19972741 (19.0 MiB)  TX bytes:113685819 (108.4 MiB)
          Interrupt:3

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:46 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3220 (3.1 KiB)  TX bytes:3220 (3.1 KiB)

, and here it is from the PC:

eth0      Link encap:Ethernet  HWaddr 00:1B:FC:59:8D:A7
          inet addr:192.168.1.175  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::21b:fcff:fe59:8da7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7076937 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12208448 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3715284439 (3.4 GiB)  TX bytes:10671251888 (9.9 GiB)
          Interrupt:23 Base address:0xe000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:7681 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7681 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3031928 (2.8 MiB)  TX bytes:3031928 (2.8 MiB)

vmnet1    Link encap:Ethernet  HWaddr 00:50:56:C0:00:01
          inet addr:192.168.114.1  Bcast:192.168.114.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10112 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

.

I've just noticed the vmware virtual network on the PC. Could that make a difference? I don't see why it might affect every other device on the network.

Thanks a lot for reading this overly long post. I hope someone has some ideas.

Cheers

(Last edited by beady on 5 Mar 2008, 17:55)

I can see similar speeds running iperf -s and "netcating" /dev/zero between the router and any PC. It never goes beyond 20.1 Mbits/sec. This happens over wired and wireless link. Is the speed limited somehow? Just coincidence?

I'm using a WRT54GL 1.1 with OpenWRT KAMIKAZE (7.09)

Regards,
Michel

Wireless 802.1g actual throughput is about 22 megabits/sec = about 2.2 megabytes/sec
Wireless 802.1b actual throughput is about 6.5 megabits/sec = about 650 kilobytes/sec

100base-T is supposed to go to about 12.5 megabytes/sec .

My network attached disk clocks 5 megabytes/sec reading from and 3.25 megabytes/sec writing to. This is typical of stand alone NAS boxes. I don't know of one that reaches the full bandwidth of 100base-T ?  My ReplayTV boxes stream video at less than half 10base-T speeds on normal quality and would still not saturate at high quality. Are your videos bigger than two GB/hour? I suspect not. Now, I can't really play .avi files off of my NAS as the player programs seem to want to read through the whole file before starting, maybe that is what you are seeing? Anybody know what players do and don't do this?

That's the whole point, it doesn't even reach the 12.5 MB/sec on LAN (100base-T). I get roughly 22 Mb/s with TCP and 1 Mb/s with UDP. It's that normal?

root@OpenWrt:~# iperf -s -u
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 32.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 5001 connected with 192.168.1.3 port 32798
[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec  0.295 ms    0/  893 (0%)

root@OpenWrt:~# iperf -s   
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 42.7 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.3 port 54475
[  4]  0.0-10.0 sec  33.2 MBytes  27.8 Mbits/sec

(Last edited by mpeterson on 14 Mar 2008, 21:14)

Running iperf on the router is affected by the CPU performance of the router. Two PCs communicating over the built-in switch do not involve the routers CPU. Communicating between LAN and WLAN *does* involve the routers CPU, and that's the bottleneck :-(
LAN to WLAN throughput on my WL-500gd is about 2.6MByte/sec.
If you cannot reach acceptable performance on LAN to LAN, i'd the check cables of the attached devices.
Do you have an additional switch to test?

I get similar results with LAN to LAN and NOT running iperf on the router. I don't even reach an acceptable performance. I've tested the cables too, they are not the problem Still need to test with another switch.

My vague understanding is that Linux has no device drivers to make use on onboard DMA engines, so all data copying has to be done by the CPU.  Most units have DMA engines because the CPU is not meant for these high bandwidth applications, theres just been no attempt to use them.

I dont know anything at all about the onboard switch and whether it should be able to give you fast copy speeds / bypass the CPU.

Sorry I've not replied. Just got back from holiday. Anyway, glad to see it's not just me that's having issues. I had the chance to run a direct file transfer between two fast PC's across the network (just the fedora media pc and a windows laptop, so no real performance testing, just sending large files across), and it got up to around 6-7 Mbytes/s, so I think that part of the problem must be trying to use a router as one end of the test (i.e. I max out the CPU). It's strange that I get exactly the same values for both routers though, since the WRT is over twice as fast (at least according to the bogomips) as the Asus.

Anyway, the reason it's a problem is that I am streaming files larger than 2gb/hour, or more specifically, raw DVB TS streams are transfered from the dreambox to a network share on the PC. I'm going to get hold of a gigabit switch to directly connect the two to check if it makes a difference.

Out of interest, the dreambox is using an NFS share for it's root filesystem (which is located on the Asus). The target recording directory is then mounted on top of that as either a samba or nfs share (neither really work well). I naively thought that the rootfs being on NFS shouldn't affect performance, since read/write operations on the samba share wouldn't need communication with the root nfs. Am I wrong about this? When the kernel writes to a samba share, does it have to communicate constantly with the root fs? If so, maybe this is a problem. I'll get tcpdump on the case to see what's happening.

Thanks for your time

Beady

@mpeterson: The reason your udp is so low is that you've got to increase the udp bandwidth (I fell for it too). Try something like:

iperf -b 100M

You'll see lots of dropped packets (most of them actually). Keep changing the -b value until you find you don't loose packets.