Testing methodology.
The objective was to measure the speed of filesystems on the router with OpenWrt excluding the impact of other factors as much as possible. My TP-Link WR1043ND runs OpenWrt Backfire 10.03.1-RC4. Kernel version: 2.6.32.25-1.
As an external USB-drive I used Samsung HD204UI connected to the router via SATA-USB controller based on Prolific Technology PL3507 chip. About 15 partitions with different FS and 1GB in size were created with the default settings. For comparison I also made a read/write test with unformatted partition - this rate can be considered as the limit of the hardware, because not any FS-driver is involved in this case.

As benchmark tool full version of dd was used. Thus the resources of the router were not spent on data transmission over the network and the corresponding server. To minimize source file read/write expenses on the router side, /dev/zero and /dev/null weer used, respectively. Multiple tests with a file of 800 MB on Ext2 gave 3% spread only so this file size was used in all tests subsequently. All filesystems were tested with 2 values of the read/write block size: 4 KB and 2 MB.
For the purity of the experiment unnecessary services and servers were unloaded for benchmark time; no router accesses except for SSH took place. In this state, the average CPU Idle was 99.7%.

Results.
The results are shown in the graphs. In some cases, the mount options mentioned.

In all tests the CPU Idle was 0% (measurements were performed on another iteration, as it might influence the results).

Reiser4, UFS, JFS, ZFS are not supported by OpenWrt, so not tested.

Hope this test will be useful for someone.

(Last edited by XChesser on 3 Apr 2015, 15:18)

Well, comments are appreciated.

thanks for you effort and it confirms again that ntfs should be avoided at any costs .

Thanks for the effort! About time we take performance more seriously! Can you please post the mount options?

Next thing would be network transfer over ftp, nfs, samba and.... scp.

Orca,
Ext3, ordered, barrier: -t ext3 -o data=ordered,barrier=1
Ext4, ordered, barrier: -t ext4 -o data=ordered,barrier=1
Ext4, writeback, barrier: -t ext4 -o data=writeback,barrier=1
Ext4, ordered, no barrier: -t ext4 -o data=ordered,barrier=0
Ext4, journal, barrier: -t ext4 -o data=journal,barrier=1
In other cases default options were used (just 'mount device mountpoint').

Yeah, I really was going to test network file servers and even made some tests for performance with NFS-kernel, unfs3 and Samba3 with different options but I faced the problem that I get much worse speed than some other TP-Link 1043 owners. I have described it with them, tried to fix the problem with replacing a patch cord, testing with the other client hardware and with different OS (Linux/Windows), tried to tweak Ethernet card with ethtool, tested network performance with iperf and so on - no solution yet. Via NFS I see real speed at ~7MB/s that is too low - this router is capable to give twice more...

These systems are quite strange.  My buffalo g300nh dd tests (ext2,ext3,ext4 same mount settings) are around 7-8 MB sec write and 25 MB read with dd, yet in samba3 7 MB/sec write and 11 MB/sec read despite any changes to settings.  Same benchmarks with ftp too! 

My load according to htop during the write tests was 35% with ext2, up to 40% with ext4.  With samba or ftp, it pegs the cpu.

In the past, I've tried an Asus RT-N16 with more ram and faster CPU, using tomato and stock firmware and got slower!  So, something else is causing these limits!

XChesser,

Thanks a lot for your tests!

My understanding is that ReiserFS is a journaling file system, so it should be benchmarked against ext4, right?

I am surprised though about relatively low performance of XFS file system.

Maximum performance for reading which I could achieve with USB flash drive connected to USB port on my WR1043ND is 12.5MB/sec.
It was with vsftpd as FTP server, and lukemftp client (OpenSUSE 11.4 Factory), and gFTP

Connection to router was 1Gigabit Ethernet cable.

CPU utilization was close to 100% during those tests.

vplessky, yes, Reiser is a journaling file system but you must distinguish all data journaling and metadata only journaling. Both Ext3 and Ext4 support any of these kinds of journaling (see 'data' mount option).
Reiser also supports both types of journaling so you should compare its results properly.

XFS is known for its high CPU utilization so results are predictable for the router I think.

HFS+ seems to be the best.But does it run stably in Openwrt?

It should be as stable on OpenWrt as with other Linux systems.

I have never used it - recent Fedora and OpenSUSE default to ext4, you can also use xfs (in Fedora) and ReiserFS in OpenSUSE.
Give it a try, would be interesting to know results for HFS+ with FTP, NFS, Samba servers.

I found out why I was running a bit slow even in DD write tests.
cat /sys/block/sda/device/max_sectors
240

echo 1024 > /sys/block/sda/device/max_sectors  OR add /sys/block/sda/device/max_sectors=1024 to /etc/sysctl.conf file and then run sysctl -p or reboot.
Then running the DD tests I get 19 MB a second, similar to your results!

Why samba slow is because of this
https://forum.openwrt.org/viewtopic.php?id=28578

I doubt it, when CPU is idle, it is still slow.  With htop installed I can change smbd to nice 0 or even negative values and it is still slow.  Same goes for ftp transfers.  Straight DD gets 19 MB/sec writes, but ftp or samba, no matter what nice/prio seem to be limited to max 10 MB/sec on this box.  I even changed many settings on smb.conf!

I haven't done such optimizations. Probably it will increase the performance in my case too, I'll make a try. Thanks!

What block size do you use for write tests? By default dd sets bs=512 that is not optimal for writes. If it's your case you must have a luck with larger blocks, for example bs=4k.

I get ~14MB/s read and 10.7MB/s write speed with FTP. It is average speed not peak. I tested 16 FTP-clients in Linux with vsftpd on my router. Speed is much dependent on client! For me FOFF appeared to be the fastest client. Filezilla, ncftp make good work too. I also tested OS influence: 4 cross-platform clients were benchmarked both in Linux and Windows 7 on the same computer. Windows gave much worse performance. Such strange results were also confirmed by two other users (on non-English forum). Active/passive FTP-mode doesn't make any difference.
With Samba I get opposite results: Windows client is faster.

It would be interesting to lock the transfer rate to a common level and see what the system loads are for each of the different systems.  Is it possible to limit the transfer rate??

This whole thread gets more and more tumultuous (UNübersichtlich). We should continue discussing here, but relocate the documentation where it belongs:
http://wiki.openwrt.org/doc/howto/performance

Writing a file to /dev/null is hardly a good filesystem benchmark, more like a sequential write hard disk benchmark. As the results show there is very little difference between filesystems in this type of test.

Here's an example how you benchmark filesystems:

http://www.phoronix.com/scan.php?page=a … &num=1

Yes, I tested just sequential read/write. I think it's the most common behavior for a HDD connected to router. Network storage is usual model of HDD use with router and in this case it's just seq operations that we see.
Another reason is lack of benchmark tools for OpenWrt. There are no packages for iozone, bonnie, dbench in repo. So dd is the only tool for OpenWrt that I see.

As I already stated in the wiki, the remote filesystems could be benchmarked on the client pc. If you have the time ;-)

What about remote filesystems I mount them and benchmark with iozone. I tried many other tools but my choice is iozone. 'bonnie++' is inconvenient as it has no autoremount option that must be used not to measure cache performance. Manual scripting is a solution of course but iozone does it for me.

Hi guys,

I'm having very bad results in terms of disk speed with my TP-WDR3600 with OpenWrt Attitude Adjustment 12.09-rc1:

This gives ~2.6Mb/s which is really bad! I tested the same disk on a desktop computer and I got 32.2Mb/s. It is a 300Gb sata disk which I divided in two partitions, both on ext4. The first partition (512Mb) I use with extroot to increase the system disk space. The second partition I use for downloads.

Kind regards,
Américo

I was using fstab to mount the disk. I discovered that if I mount manually (using the mount command) I was able to reach 30Mb/s.

Thanks anyway.

Regards,
Américo

The picture on imageshack with the performance comparison is unavailable. Does someone have it?

I'm thinking about reformatting my HDD as ext4, because the performance with NTFS is very bad, and I would like to know how much faster ext4 would be. With NTFS-3G the write speed is about 3-4 MB/s.

Regards,

Joel

Anyone got the image? The link is dead