I ran NAT Tests, and the results were interesting. Two tests. Test 1 without NAT and Test 2 with NAT.
Test 1 (No NAT) File Transfer
45.9GB File
Start Time 5pm
Time to finish the transfer 9:09 minutes
Test 2 (NAT) File Transfer
45.9GB File
Start Time 5:25pm
Time to finish the transfer 10:34minutes
Throughput Utilization

CPU Utilization %

Load

Seems around 100mbps was lost by using "NAT" as opposed to just layer 2 switching, in this test. I realize people are saying NAT is cpu bound on OpenWrt, but I didn't find it to be CPU bound as the TOP graph above shows. For those who know networking, Layer 3 (Network) is always going to be slower than Layer 2, and inherently so. Was the slowness due to NAT translations or the IP layer (maybe a combination of the two)?
Load was increased during the download whilst NAT was involved, but I don't believe the load was cpu bound, but IRQ bound.
The below is a sample of highest load processes during NAT download. Notice ksofirqd/0 is eating some processor.
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
3 2 root SW 0 0% 31% [ksoftirqd/0]
414 2 root RW 0 0% 6% [kworker/0:2]
A little bit of research about ksofirqd shows the following....
"In some situations IRQs come very very fast one after the other and the operating system cannot finish servicing one before another one arrives. This can happen when a high speed network card receives a very large number of packets in a short time frame.
Because the operating system cannot handle IRQs as they arrive (because they arrive too fast one after the other), the operating system queues them for later processing by a special internal process named ksoftirqd.
If ksoftirqd is taking more than a tiny percentage of CPU time, this indicates the machine is under heavy interrupt load."
So this bears the question.... Has anyone tried to tweak affinity >> smp_affinity so as to reduce IRQ load?
(Last edited by davidc502 on 30 Jun 2015, 00:46)