Hi,
I have made a wiki page and a benchmark script to compare performance of hardware of different devices running OpenWRT.
It's here: http://wiki.openwrt.org/HardwarePerformance

I have an ASUS WL-500g, could others that have a different device (Linksys, NSLU) also run the bench and update the page? That would be nice!

Regards.
-jec

I guess it is not that easy:

root@openwrt:~# ./openwrt_cpu_bench
./openwrt_cpu_bench: can't load library 'libstdc++.so.6'

Ps. Dose it really make sense to measure performance of that kind of devices ? They all  use very similar parts....

What are you trying to prove? Why a synthetic benchmark? Pick something that you would actually do and devise a way to test how well it does that.

There are already standard benchmarking tools, why write custom software?

A cpu only benchmark? The right compiler settings can unroll the program, essentially precaclulating the results and making the executable into a glorified print statement.

...

What it really boils down to is the fact that many of the boards are 200Mhz broadcom chips based off the same reference design; there won't be a statistically signifigant difference between them.

As an example, look at /proc/cpuinfo. See that 199 bogomips? That's a very simple benchmark of how fast your system can do absolutely nothing. Unsurprsingly, it's directly related to the Mhz the system is clocked at, in this case 200Mhz. In other words any 200Mhz mips system will report 199 bogomips. Is it a synthetic benchmark? Absolutely.

(edit - oops, jecuendet's wl-500g is only 125Mhz)

(Last edited by mbm on 30 Apr 2006, 21:51)

There's nothing wrong with benchmarking CPU per se, if a CPU benchmark is what you want; but in isolation it's not very meaningful, especially for an almost entirely I/O-bound application like routing - particularly the low-end static routing that OpenWRT is aimed at. Large route tables and protocols like BGP, which wouldn't be practical on the sort of hardware OpenWRT runs on. Encryption, compression and decompression duties aside, CPU is the least of our worries.

It would probably be more useful to stress-test the network interfaces singly and together to see what their bus bandwidth is like, particularly given that these interfaces are likely to contain the biggest deviations from, or additions to, the reference designs, and hence the biggest differences between different manufacturers' designs.

...and to be honest, such a stress-test would not tell you much more than you could find out by scripting ping -f with a few different parameters.

Thanks for your comments guys.
The point is that I don't ask you to give reason why I shouldn't have done that CPU test. I asked to test your device and report your results!
The point is that there *are* differences between devices, especially between WL-500g and Linksys NSLU. That's what intersts me most. And I don't know for WRT54g

For samba file sharing, I have a low IO rate because (I think) the CPU is not that powerful... And the same for SFTP since encryption is heavy on resources.
That's the whole point of the test!

And I agree that compiler can change CPU bench a lot. But the point is not to glorify your results but to compare them. Just to be able to buy what's best for CPU "intensive" applications (read SSH and samba)

Another point is that if you want to do Donkey p2p (amule, mldonkey) you need a *fast* CPU. Also RAM is needed.

That said, I agree that having made a C++ application (instead of plain C) isnot a good idea... I'll change that and re-upload.

Thanks for reading.
-jec

just a small note to your wiki-page: either you have a wl-500g and a 125mhz cpu or a wl-500g deluxe and a 200mhz cpu - i think you need to correct that!

also, why don't you make a package out of your cpu-test or upload the binary somewhere? i wouldn't want to fetch the whole sdk just to compile the cpu-test!

greetings,
andy

Following the above comments, I corrected:
- Made a plain C bench instead of c++
- Corrected my device (Deluxe ASUS WL-500G)
- Uploaded a precompiled binary

There is a new version of the bench there: http://wiki.openwrt.org/HardwarePerformance

>And the same for SFTP since encryption is heavy on resources. That's the whole point of the test!

So why not test ssh, sftp, pptp etc (compiled with diffrent flags)? Synthetic floating point benchmarks are meaningless on routers.

jruhe

(Last edited by jruhe on 1 May 2006, 16:06)

There is a new version posted. It adds pure floating point calculation (as is done by SSH for encryption for example)
This is v0.4
Retest it please.

Another new version: v0.5
I found an error in the PI calculation bench. The previous calculation os completely wrong. Please re-test with that new version
Thanks and sorry for the inconvenience.

> It adds pure floating point calculation (as is done by SSH for encryption for example)

Encryption usually does not involve any floating point calculations. Take a look into dropbear sources (e.g dropbear-0.48.1\libtomcrypt\src\ciphers\aes\aes.c)

jruhe

(Last edited by jruhe on 1 May 2006, 18:03)

I changed the Links on the wiki to internal Links (not as long as the prev. http.... links )
To save some space, you could also remove the column completly and use the device section for it:

 [:OpenWrtDocs/Hardware/Linksys/WRT54GS:WRT54GS]

-> only WRT54GS is shown

What I have been looking for is throughput benchmarks for each OpenWrt supported hardware. Since OpenWrt is mainly used serving clients or acting as a client on a network, network throughput is important.

I would like to see a discussion about how network benchmarking would be done and what tests to include (vanilla install, openvpn, wifi, different encryptions etc).

I totaly agree with you.
A routing bench would be very interesting. I was recently desapointed by the WRT54G (v1.1) for that task.

The maestro spoke:

(Last edited by olli on 9 May 2006, 16:15)

I find cpu benchmarks rather amusing. Personally, I like to use Povray for such purposes, however the lack of an fpu on my WRT54GL means that it'd take weeks to finish just the benchmark scene. My other favorite is nbench.

Here are the results of my router underclocked to 183 MHz:

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          35.805  :       0.92  :       0.30
STRING SORT         :           2.907  :       1.30  :       0.20
BITFIELD            :      9.1928e+06  :       1.58  :       0.33
FP EMULATION        :          5.4075  :       2.59  :       0.60
FOURIER             :           5.393  :       0.01  :       0.00
ASSIGNMENT          :         0.39841  :       1.52  :       0.39
IDEA                :          133.08  :       2.04  :       0.60
HUFFMAN             :            11.4  :       0.32  :       0.10
NEURAL NET          :       0.0049759  :       0.01  :       0.00
LU DECOMPOSITION    :         0.15314  :       0.01  :       0.01
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 1.250
FLOATING-POINT INDEX: 0.007
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
**System used for compilation:
**Linux red 2.4.22-1-k6 #5 Sat Oct 4 14:38:05 EST 2003 i686 GNU/Linux
**C compiler: mipsel-uclibc-gcc (unknown version)
**CFLAGS : -O3 -fomit-frame-pointer -mcpu=r4000 -mips2
**libc: unknown version
**Date of compilation: Tue Dec 14 21:58:19 CET 2004
MEMORY INDEX        : 0.296
INTEGER INDEX       : 0.324
FLOATING-POINT INDEX: 0.004
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38

For comparison, here's my Sempron 64 1.6 GHz:

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          1522.2  :      39.04  :      12.82
STRING SORT         :           146.6  :      65.50  :      10.14
BITFIELD            :      3.5012e+08  :      60.06  :      12.54
FP EMULATION        :          119.34  :      57.27  :      13.21
FOURIER             :           14750  :      16.77  :       9.42
ASSIGNMENT          :          17.152  :      65.27  :      16.93
IDEA                :          2966.5  :      45.37  :      13.47
HUFFMAN             :          1233.6  :      34.21  :      10.92
NEURAL NET          :          23.166  :      37.21  :      15.65
LU DECOMPOSITION    :          840.12  :      43.52  :      31.43
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 50.947
FLOATING-POINT INDEX: 30.062
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : AuthenticAMD AMD Hammer Family processor - Model Unknown 1600MHz
L2 Cache            : 128 KB
OS                  : Linux 2.6.14
C compiler          : gcc version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13)
libc                : 
MEMORY INDEX        : 12.913
INTEGER INDEX       : 12.566
FLOATING-POINT INDEX: 16.673
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38