Topic: Openwrt build: low bandwidth on 11ac radio module;

The content of this topic has been archived between 14 Apr 2018 and 3 May 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

Page 2 of 2

Post #26

sjuliao

29 Sep 2016, 10:16

bzh35 wrote:

Hi sjuliao,
Regarding the performance issue on 11ac,
Did you check the kernel network scheduler and queuing ?
This part of the kernel has changed from 3.18 version.

Yes, I've checked my image configurations.

The OEM, when running tc qdisc command gives me:

root@OpenWrt:/# tc qdisc
qdisc hyfi_pfifo_fast 0: dev eth0 root refcnt 2 [Unknown qdisc, optlen=24] 
qdisc hyfi_pfifo_fast 0: dev wlan0 root refcnt 2 [Unknown qdisc, optlen=24] 
!!!Deficit -4, rta_len=48
qdisc pfifo_fast 0: dev wifi0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
root@OpenWrt:/#

But when I looked at my custom images, neither of them had the kmod-sched, kmod-sched-core or tc command selected (by default). Also, the default queuing/scheduling policy set on the kernel_menuconfig for all my custom images is "Fair Queue Controlled Delay AQM (FQ_Codel)".
So it seemed to me that they share the same configuration on this, reason why I didn't thought it could be that...and since at this moment I only want to validate the typical bandwidths, I didn't loose time setting and optimizing these policies (I would prefer to do that later, when my system is in its real environment and near the production stage).

But that could perfectly explain why even the loopback interface decreases its performance so drastically between K3.x and K4.x.
Is there an huge difference between K3.x and K4.x on the this?

Do you think that the default configuration is the source of my poor performance problem?
If yes, can you advise me on the best way to set and test these?

bzh35 wrote:

Did you check hardware low level settings on the different bus involved on the data path used to manage skb down to the 11ac chip ?
Up to now, I never check the 11ac performance only 11ad on my target. What is your platform ? Can I purchase one ? Is it a product under development or other thing ?

I am sorry but I don't know what you are referring here (are you refering to the device tree file? can you explain better?).
Also, after doing the loopback test and realizing the decrease of performance, which is more or less consistent with the behavior of all other interfaces, I start to wondering if this is only an 11ac issue...that is the reason why I am trying to despite the processor out of the equation by now (if it is not the processor - which I don't think it is - only lefts something related with the kernel versions).

Thank you for all your efforts and attention.

(Last edited by sjuliao on 29 Sep 2016, 11:52)

Post #27

bzh35

29 Sep 2016, 12:38

Hi sjuliao,
Yes, it is working with skb > 255 bytes. The modification is at least on host pcie and/or Adm source code. I will post a patch.

Can you check UDP throughput on your target ?
Over Ethernet and 11ac interfaces ?

Post #28

bzh35

29 Sep 2016, 13:51

Hi sjuliao,
I'm refering on the hardware system on chip : It means interconnect, AXI arbiter, bus memory, clocks...Any hardware peripherals around the CPU. Anything impacting pure software processing.

Did you check hardware low level settings on the different bus involved on the data path used to manage skb down to the 11ac chip ?

Post #29

sjuliao

4 Oct 2016, 20:20

bzh35 wrote:

Hi sjuliao,
Can you check UDP throughput on your target ?
Over Ethernet and 11ac interfaces ?

Hi bzh35,

Sorry for the delay in my answer.

Here are the UDP throughput tests.

Ethernet 1Gb (between Router and LP2):

First, running iperf on router as server and running command "iperf -c 192.168.2.1 -u -b 1000M" on LP2. Changing window size and/or other iperf parameters it could be possible to achieve better results but to me, what matters with this test is to evaluate the performance across all linux kernel versions (assuming same command and identical conditions).

OEM Image (k3.4.103)

root@OpenWrt:/# iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 40499
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   858 MBytes   720 Mbits/sec   0.016 ms 9058/621380 (1.5%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 57868
[  3]  0.0-10.0 sec   735 MBytes   616 Mbits/sec   0.015 ms 134363/658305 (20%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 36452
[  3]  0.0-10.0 sec   905 MBytes   759 Mbits/sec   0.025 ms   61/645628 (0.0094%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 35988
[  3]  0.0-10.0 sec   817 MBytes   685 Mbits/sec   0.019 ms   38/582734 (0.0065%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 50874
[  3]  0.0-10.0 sec   794 MBytes   665 Mbits/sec   0.060 ms  112/566139 (0.02%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order

K3.18

root@OpenWrt:/# iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 55156
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   731 MBytes   613 Mbits/sec   0.104 ms 69718/591231 (12%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 34494
[  3]  0.0-10.0 sec   839 MBytes   704 Mbits/sec   0.022 ms  117/598796 (0.02%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 59781
[  3]  0.0-10.3 sec   697 MBytes   570 Mbits/sec  15.591 ms 76667/573539 (13%)
[  3]  0.0-10.3 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 58598
[  3]  0.0-10.0 sec   802 MBytes   673 Mbits/sec   0.115 ms 3070/575425 (0.53%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 56056
[  3]  0.0-10.0 sec   744 MBytes   624 Mbits/sec   0.028 ms 40711/571563 (7.1%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order

K4.1

root@OpenWrt:/# iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 57055
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.3 sec   232 MBytes   190 Mbits/sec  13.452 ms 434330/599967 (72%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 47679
[  3]  0.0-10.3 sec   324 MBytes   265 Mbits/sec  14.892 ms 372998/604188 (62%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 48697
[  3]  0.0-10.2 sec   340 MBytes   280 Mbits/sec  15.490 ms 379735/622273 (61%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 47411
[  3]  0.0-10.3 sec   308 MBytes   252 Mbits/sec  15.128 ms 335453/554966 (60%)
[  3]  0.0-10.3 sec  1 datagrams received out-of-order
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 54945
[  3]  0.0-10.3 sec   213 MBytes   174 Mbits/sec  13.287 ms 451722/603714 (75%)

The results are pretty similar to the "bad performance" behavior observed with the TCP traffic: with the linux kernel version increase, the throughput seems to decrease while the jitter and % of errors increase. Also, as can be seen, repeating the tests for the same image, sometimes the % of errors increases dramatically. Not entirely sure why that is happening and if it is normal.

Still concerning Ethernet 1Gb but running iperf as server on LP2:

OEM Image (k3.4.103)

sjuliao@sjuliao:~/Projects/11ac/$ iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 57483
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   573 MBytes   481 Mbits/sec   0.036 ms 12531/421440 (3%)
[  3]  0.0-10.0 sec  28 datagrams received out-of-order
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 33209
[  4]  0.0-10.0 sec   555 MBytes   466 Mbits/sec   0.038 ms 6423/402150 (1.6%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 57891
[  3]  0.0-10.0 sec   574 MBytes   482 Mbits/sec   0.047 ms 8177/417430 (2%)
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 37517
[  4]  0.0-10.0 sec   590 MBytes   495 Mbits/sec   0.059 ms 9512/430385 (2.2%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 53183
[  3]  0.0-10.0 sec   586 MBytes   492 Mbits/sec   0.055 ms 8495/426331 (2%)

K3.18

sjuliao@sjuliao:~/Projects/11ac/$ iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 50559
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   538 MBytes   451 Mbits/sec   0.228 ms 6289/389984 (1.6%)
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 33228
[  4]  0.0-10.0 sec   547 MBytes   459 Mbits/sec   0.229 ms 11363/401563 (2.8%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 43542
[  3]  0.0-10.0 sec   552 MBytes   463 Mbits/sec   0.238 ms 9448/402868 (2.3%)
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 47757
[  4]  0.0-10.0 sec   546 MBytes   458 Mbits/sec   0.217 ms 11475/400729 (2.9%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 36080
[  3]  0.0-10.0 sec   544 MBytes   457 Mbits/sec   0.194 ms 14297/402558 (3.6%)

K4.1

sjuliao@sjuliao:~/Projects/11ac/$ iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 55306
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   259 MBytes   217 Mbits/sec   0.302 ms 1649/186224 (0.89%)
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 39660
[  4]  0.0-10.0 sec   266 MBytes   223 Mbits/sec   0.244 ms  293/190113 (0.15%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 35444
[  3]  0.0-10.0 sec   266 MBytes   223 Mbits/sec   0.296 ms  157/189597 (0.083%)
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 53613
[  4]  0.0-10.0 sec   265 MBytes   223 Mbits/sec   0.269 ms  823/190161 (0.43%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 56700
[  3]  0.0-10.0 sec   265 MBytes   222 Mbits/sec   0.271 ms  458/189429 (0.24%)

For this case, the faulty behavior is the same. However for K4.1 we can see that the maximum throughput is basically 50% less, with an jitter and % of errors that are not good (for this bandwidth, they should be much better than this...).

I've already done the 11ac tests but I noticed now a mistake in my test with the OEM image, so I will need to repeat that test next Friday, when I return to my office. I will post the results for 11ac all together.

I intend to do also an direct LP2 <--> LP1 UDP test via Ethernet so I can have an idea of the maximum throughput that I can achieve and verify if the huge "% of errors" variance between tests persists.

Kind regards

(Last edited by sjuliao on 4 Oct 2016, 20:34)

Post #30

sjuliao

7 Oct 2016, 16:26

Some good news:

I found that the problem for the cpufreq stats and remaining options not being exported on the "/sys/devices/system/..." path was due to a bad regulator description on the DT for the K4.1 and K4.4.7.
After correcting this on K4.1 and testing again, I finally have cpufreq exporting successful all the information as it should be. Also, now I am entirely sure that my processor is running at the right frequency, according to the governor policy chosen.

Solved this problem and after doing some more Eth1Gb and 11ac network tests on this image, I found that the throughput has also improved which means that the difference seen before between K3.18 and K4.1 was related with the regulator...

So, the current status is

Ethernet 1Gb:
(iperf -s on router; average on 60s):
OEM Image --- 903 Mbits/sec
CustomK3.18 --- 930 Mbits/sec
CustomK4.1 --- 930 Mbits/sec

(iperf -c on router; average on 60s):
OEM Image --- 711 Mbits/sec
CustomK3.18 --- 889 Mbits/sec
CustomK4.1 --- 900 Mbits/sec

So, on the Ethernet, things seem to be more consistent now (regarding TCP test only!). The OEM image seems to have less throughput on both test types (worse when running iperf -c on router) but I won't spent time looking for the problem (my priority is to solve problems in my images). There is ~30MBits/s difference between test types but I think this is normal and I would accept this for now.
On the UDP tests, I think I have a problem but I would explain that on another post.

On the 11ac, things have improved but I still have some bandwidth limitation.

Wireless 11ac:
(iperf -s on router; average for 5 tests, each during 60s):
OEM Image --- 337 Mbits/sec
CustomK3.18 --- 325.4 Mbits/sec
CustomK4.1 --- 287.4 Mbits/sec (improved but not similar to K3.18)

(iperf -c on router; average for 5 tests, each during 60s):
OEM Image --- 342 Mbits/sec
CustomK3.18 --- 252 Mbits/sec
CustomK4.1 --- 254.2 Mbits/sec (improved and similar to K3.18)

Between my custom images (K3.18 and K4.1) and the OEM image, things are still not perfect.: when running iperf -c on router, there is an huge bandwidth difference between them...and even when running iperf -s, there is an limitation with the K4.1 when compared with K3.18.

Any suggestions?

Regards.

(Last edited by sjuliao on 7 Oct 2016, 17:14)

Post #31

stangri

7 Oct 2016, 21:55

sjuliao wrote:

I found that the problem for the cpufreq stats and remaining options not being exported on the "/sys/devices/system/..." path was due to a bad regulator description on the DT for the K4.1 and K4.4.7.
After correcting this on K4.1 and testing again, I finally have cpufreq exporting successful all the information as it should be. Also, now I am entirely sure that my processor is running at the right frequency, according to the governor policy chosen.

Have you (or are you going to) submitted the fix to trunk? Which hardware was it for?
Thanks!

Post #32

sjuliao

10 Oct 2016, 09:19

stangri wrote:

sjuliao wrote:
I found that the problem for the cpufreq stats and remaining options not being exported on the "/sys/devices/system/..." path was due to a bad regulator description on the DT for the K4.1 and K4.4.7.
After correcting this on K4.1 and testing again, I finally have cpufreq exporting successful all the information as it should be. Also, now I am entirely sure that my processor is running at the right frequency, according to the governor policy chosen.
Have you (or are you going to) submitted the fix to trunk? Which hardware was it for?
Thanks!

Hi stangri,

Yes, I don't mind to upload an patch for that problem but I wounder if it will be useful to anyone else.
A few weeks ago and after being working on the openwrt image for more than two months, I did a new clone of the main git trunk to start a clean image from scratch. At that time and after spending some time to realize why things were different in my local openwrt directory (for instance, linux kernel 4.1 was not present anymore and instead of it, there was support for k4.4.14, among many other changes), I discover that there is an huge difference on using "git clone git://git.openwrt.org/openwrt.git" (which leads to my current image) and "git clone https://github.com/openwrt/openwrt.git" (which leads to what seems to be the latest branch state). I don't know if this behavior is desired/normal or if it is happening due to an mistake in the git pointers.
What I noticed is that the image based on k4.4.14 seems to have the cpufreq working again, so there should be an patch already doing the job (although for example the USB is not working for my hardware). In my case, since by now, I need to put things working with this particular version (k4.1) due to other software components, I have to try to solve all the problems manually (cpufreq was one of them). Later when this linux kernel version is full working, I will spend time trying to update the linux kernel to k4.4.x.

My hardware, as already described in initial posts of this thread, is based on the AP148 board and uses an IPQ8064 processor from Qualcomm. I think it is not available to general customer market, although it could be possible to acquire it from Qualcomm directly in special conditions (e.g. NDA agreement and for developing products purposes).
So, depending on which command people use to clone the openwrt project and their target hardware, the cpufreq could be already working without any problems.
But if necessary and useful, I can send an patch without any problems, just let me know.

Kind regards

Post #33

stangri

10 Oct 2016, 11:10

Oh, sorry, my bad, for some reason I thought you were testing (and fixing) things on a common router model.

If you decide to submit support for your unique model, this is the place where development happens nowdays: https://www.lede-project.org/development.html

Post #34

sjuliao

10 Oct 2016, 16:37

Hi,

Regarding the UDP tests, I have some doubts that I would like to clarify or at least comment with someone else.

Consider the following setup for this subject:
- laptop connected to Router through an 2 meters ethernet cable;
- iperf v2.0.5 running on both router & pc (changing only the mode);
- OEM image has qdisc set to hyfi_pfifo_fast

root@OpenWrt:/# tc qdisc
qdisc hyfi_pfifo_fast 0: dev eth0 root refcnt 2 [Unknown qdisc, optlen=24] 
qdisc hyfi_pfifo_fast 0: dev wlan0 root refcnt 2 [Unknown qdisc, optlen=24] 
!!!Deficit -4, rta_len=48
qdisc pfifo_fast 0: dev wifi0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
root@OpenWrt:/#

- both custom images have default qdisc set to fq_codel (default)

root@OpenWrt:QC-IPDock:/# tc qdisc
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn

Commands used:
- iperf in server mode: iperf -u -s
- iperf in client mode: iperf -u -c 192.168.2.x -b 1000M
Note: not changing packet size or any other parameter deliberately.

On OEM image:
(running iperf as server on router)

root@OpenWrt:/# iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49200
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec  1.06 GBytes   911 Mbits/sec   0.016 ms 3813/777747 (0.49%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49201
[  3]  0.0- 9.9 sec  1.08 GBytes   935 Mbits/sec   0.022 ms 10950/796270 (1.4%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49202
[  3]  0.0-10.0 sec  1.06 GBytes   907 Mbits/sec   0.012 ms 2425/774281 (0.31%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49203
[  3]  0.0-10.2 sec   524 MBytes   429 Mbits/sec  15.062 ms 418154/791653 (53%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49204
[  3]  0.0- 9.1 sec   476 MBytes   440 Mbits/sec   0.028 ms 433265/772786 (56%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49205
[  3]  0.0- 9.1 sec   472 MBytes   435 Mbits/sec   0.024 ms 444554/781277 (57%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49206
[  3]  0.0-10.0 sec   542 MBytes   455 Mbits/sec   0.022 ms 375189/761780 (49%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49207
[  3]  0.0-10.0 sec   489 MBytes   410 Mbits/sec   0.036 ms 440635/789700 (56%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49208
[  3]  0.0- 9.9 sec  1.02 GBytes   883 Mbits/sec   0.023 ms 44287/791117 (5.6%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49209
[  3]  0.0-10.0 sec  1.09 GBytes   933 Mbits/sec   0.021 ms 1522/795344 (0.19%)
[  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 49210
[  3]  0.0-10.0 sec  1.06 GBytes   914 Mbits/sec   0.019 ms 1015/777971 (0.13%)

(running iperf as client on router)

C:\iperf>iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 37917
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   590 MBytes   495 Mbits/sec   0.039 ms  904/421734 (0.21%)
[  3] 0.00-9.99 sec  15 datagrams received out-of-order
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 49299
[  4]  0.0-10.0 sec   601 MBytes   504 Mbits/sec   0.028 ms 1990/430620 (0.46%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 49679
[  3]  0.0-10.0 sec   601 MBytes   505 Mbits/sec   0.023 ms 2105/430985 (0.49%)
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 48248
[  4]  0.0-10.0 sec   602 MBytes   505 Mbits/sec   0.038 ms  960/430634 (0.22%)
[  3] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 57819
[  3]  0.0-10.0 sec   562 MBytes   472 Mbits/sec   0.027 ms 1717/422803 (0.41%)
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 50314
[  4]  0.0-10.0 sec   577 MBytes   484 Mbits/sec   0.042 ms  847/433097 (0.2%)

In my custom images the behavior is basically the same, so I will not pollute more this post with their outputs. This also happens on the 11ac interface.
I know that I am setting the bandwidth a little above of the real maximum ethernet capability but that is intentional to stress the connection. Also, I am aware that UDP does not implement congestion or traffic flow mechanisms (and others), reason why I would expect to have some error.
But is it normal to have that huge increase/decrease in error (aka lost datagrams) between consecutive tests?
Has anyone else done similar tests with UDP, that can comment on this behavior and on these metrics, so I can have an idea if they are acceptable?

I suspect that I can improve this throughput changing the queue/network scheduling policy but when I tried to change it to PRIO on my custom K4.1 image, I didn't see any improvements or anything that shows that the policy is working. Also, trying to apply some commands as described on the example https://wiki.openwrt.org/doc/howto/pack … r.example1, I get some strange messages:

root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# iperf -u -s -D
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
Running Iperf Server as a daemon
The Iperf daemon process ID : 745
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# [  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 60831
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   771 MBytes   646 Mbits/sec   0.017 ms 222700/772431 (29%)
[  0]  0.0-10.0 sec   230 MBytes   193 Mbits/sec   0.005 ms 620530/784431 (79%)
[  0]  0.0-10.0 sec   245 MBytes   207 Mbits/sec   0.013 ms 628874/803679 (78%)
[  0]  0.0-10.0 sec  1.02 GBytes   879 Mbits/sec   0.006 ms 33244/780663 (4.3%)
[  0]  0.0-10.0 sec  1.03 GBytes   886 Mbits/sec   0.006 ms 32538/785940 (4.1%)
[  0]  0.0-10.2 sec   241 MBytes   197 Mbits/sec  15.081 ms 624693/796343 (78%)

root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# insmod /lib/modules/4.1.23/net_sched/sch_prio.ko 
module is already loaded - sch_prio
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# tc -s qdisc show dev eth0
qdisc fq_codel 0: root refcnt 2 limit 1024p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn 
 Sent 10460 bytes 20 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# tc qdisc add dev eth0 root handle 1: prio default 30
What is "default"?
Usage: ... prio bands NUMBER priomap P1 P2...[multiqueue]
root@OpenWrt:QC-IPDock:/# tc qdisc add dev eth0 root handle 1: prio
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# tc -s qdisc show dev eth0
qdisc prio 1: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
root@OpenWrt:QC-IPDock:/# tc class add dev eth0 parent 1: classid 1:1 prio rate 
1000kbit?
Error: Qdisc "prio" is classless.
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# tc -s qdisc show dev eth0
qdisc prio 1: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# iperf -u -s -D
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
Running Iperf Server as a daemon
The Iperf daemon process ID : 756
root@OpenWrt:QC-IPDock:/# 
root@OpenWrt:QC-IPDock:/# [  3] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 62121
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.0 sec   730 MBytes   612 Mbits/sec   0.018 ms 266525/787019 (34%)
[  0]  0.0-10.2 sec   694 MBytes   568 Mbits/sec  15.250 ms 307865/802608 (38%)
[  0]  0.0-10.0 sec   714 MBytes   599 Mbits/sec   0.014 ms 282951/792288 (36%)
[  0]  0.0-10.2 sec   709 MBytes   581 Mbits/sec  15.279 ms 283310/789091 (36%)
[  0]  0.0-10.0 sec   906 MBytes   760 Mbits/sec   0.009 ms 133781/779729 (17%)
[  0]  0.0-10.0 sec   730 MBytes   612 Mbits/sec   0.024 ms 279199/799639 (35%)
[  0]  0.0-10.0 sec   723 MBytes   607 Mbits/sec   0.011 ms 269335/785359 (34%)
[  0]  0.0-10.0 sec   899 MBytes   754 Mbits/sec   0.006 ms 143421/784914 (18%)
[  0]  0.0-10.2 sec   751 MBytes   615 Mbits/sec  14.379 ms 258632/794064 (33%)

root@OpenWrt:QC-IPDock:/#

So, according to the information available on http://man.cx/tc and even on "man tc" command, PRIO is CLASSFUL QDISC but on the output it complains that it is CLASSLESS...
Also I was not able to apply the same command as stated on openwrt link.
And the list of network schedulers available in my openwrt kernel_menuconfig is limited to only

drwxrwxr-x 2 sjuliao sjuliao  4096 Oct 10 14:06 .
drwxrwxr-x 6 sjuliao sjuliao  4096 Oct 10 14:06 ..
-rw-r--r-- 1 sjuliao sjuliao 20264 Oct 10 14:02 sch_cbq.ko
-rw-r--r-- 1 sjuliao sjuliao  8700 Oct 10 14:02 sch_choke.ko
-rw-r--r-- 1 sjuliao sjuliao  8012 Oct 10 14:02 sch_codel.ko
-rw-r--r-- 1 sjuliao sjuliao 10200 Oct 10 14:02 sch_drr.ko
-rw-r--r-- 1 sjuliao sjuliao  8536 Oct 10 14:02 sch_dsmark.ko
-rw-r--r-- 1 sjuliao sjuliao 10452 Oct 10 14:02 sch_esfq.ko
-rw-r--r-- 1 sjuliao sjuliao 10924 Oct 10 14:02 sch_fq.ko
-rw-r--r-- 1 sjuliao sjuliao 10784 Oct 10 14:02 sch_gred.ko
-rw-r--r-- 1 sjuliao sjuliao 18972 Oct 10 14:02 sch_hfsc.ko
-rw-r--r-- 1 sjuliao sjuliao  8092 Oct 10 14:02 sch_hhf.ko
-rw-r--r-- 1 sjuliao sjuliao 19984 Oct 10 14:02 sch_htb.ko
-rw-r--r-- 1 sjuliao sjuliao  4168 Oct 10 12:42 sch_ingress.ko
-rw-r--r-- 1 sjuliao sjuliao  7148 Oct 10 14:02 sch_mqprio.ko
-rw-r--r-- 1 sjuliao sjuliao  7796 Oct 10 14:02 sch_multiq.ko
-rw-r--r-- 1 sjuliao sjuliao 12120 Oct 10 14:02 sch_netem.ko
-rw-r--r-- 1 sjuliao sjuliao  8380 Oct 10 14:02 sch_pie.ko
-rw-r--r-- 1 sjuliao sjuliao  4136 Oct 10 14:02 sch_plug.ko
-rw-r--r-- 1 sjuliao sjuliao  7640 Oct 10 14:02 sch_prio.ko
-rw-r--r-- 1 sjuliao sjuliao 16988 Oct 10 14:02 sch_qfq.ko
-rw-r--r-- 1 sjuliao sjuliao  9496 Oct 10 14:02 sch_red.ko
-rw-r--r-- 1 sjuliao sjuliao  9228 Oct 10 14:02 sch_sfb.ko
-rw-r--r-- 1 sjuliao sjuliao 13048 Oct 10 13:15 sch_sfq.ko
-rw-r--r-- 1 sjuliao sjuliao 10272 Oct 10 14:02 sch_tbf.ko
-rw-r--r-- 1 sjuliao sjuliao  9400 Oct 10 14:02 sch_teql.ko

So I have no sch_pfifo_fast (for instance) as stated on https://wiki.openwrt.org/doc/howto/pack … ssfulqdisc. Is this normal?
Does it depend on some selection?
Is there some tutorial or easy way to setup these network schedulers without much effort to check which ones perform better or do I have to test them individually? In this last case, anyone can suggest a good recipe to test them individually without depending too much on their specific parameters (my intention is to have an quick and overall idea of their general performance and only then, spend time tuning the most promising ones)?

Kind regards

(Last edited by sjuliao on 10 Oct 2016, 16:41)

Post #35

sjuliao

10 Oct 2016, 17:19

bzh35 wrote:

Hi sjuliao,
Yes, it is working with skb > 255 bytes. The modification is at least on host pcie and/or Adm source code. I will post a patch.

Hi bzh35,

Have you uploaded or posted the patch for the 11ad?
If you can share it or the details of what you have changed, I would like to apply it to my image and start also doing some bandwidth measurements with the 11ad (without that 255KByte annoying limitation).

Kind regards

The discussion might have continued from here.

Page 2 of 2