It is already on my nerves   Has been since release 3 or 4 when I moved my machine over to btrfs and figured out how to do an in-place upgrade.

I actually just ran through the process again this evening.  Decided it was time to upgrade my live version now that the QoS stuff and a number of other items were taken care of with your help.  Managed to figure out why my one wifi card was having troubles: PCI riser card jumpers were wrong and won't allow some card combinations.  Took 4 hours of fiddling to get wifi online but only 20 minutes to upgrade .  I will be situating my router over the next few days and a bi-weekly automated build going.  Once those two things are complete I'm planning on starting in on the in-place upgrade work.  You'll want to keep an eye on the thread for updated builds though.  There may be some fixes if I find any broken items (uhttpd + ssl seems broken but I'm not 100% sure).

If you're interested below are links to some pics I took of my setup.  The only difference is I ended up having to swap the NIC in the picture for a Soekris lan1602 card (the one in the pic won't work with the wifi adapter.

https://www.dropbox.com/s/9bz3q9m1ttefd … 10355..jpg

https://www.dropbox.com/s/y0s6lcellvw5d … 90894..jpg

https://www.dropbox.com/s/dvy469p0rgdl3 … 00915..jpg

https://www.dropbox.com/s/b2kjmps6xmt92 … 79659..jpg

Very interesting.

I was originally going to have two WiFi cards in my box too but as I would need to replace the Buffalo router with a 4 or 8 port switch, it made more sense to just keep using the Buffallo as a switch and WiFi AP.  Plus it means I can always revert to using it as a router if I have any problems with the Atom box.

I have done some more testing with the latest build and not having a lot of luck with it.

Firstly, PPPoE isn't working correctly.

The Buffalo calls:
/usr/sbin/pppd plugin rp-pppoe.so mtu 1492 mru 1492 persist usepeerdns defaultroute replacedefaultroute user USERNAME password PASSWORD ipparam wanifname pppoe-wan nodetach nic-eth1

Your build calls:
/usr/sbin/pppd nodetach ipparam wanifname pppoe-wan nodefaultroute persist maxfail 1 user USERNAME password PASSWORD ip-up-script /lib/netifd/ppp-up ipv6-up-script /lib/netifd/ppp-up ip-down-script /lib/netifd/ppp-down ipv6-down-script /lib/netifd/ppp-down mtu 1492 mru 1492 plugin rp-pppoe.so nic-eth1

Notice the "nodefaultroute", no wonder I couldn't get routing working.  However its puzzling as that option is NOT ticked in LuCI or set in /etc/config/network.  I'm also puzzled why its calling ipv6 up/down scripts when "Enable IPv6 negotiation on the PPP link" is unticked.

However this build also seems to have real issues with the on board LAN adapter too. 

I noticed even after passing that power management disabling line I mentioned before, pings still fluctuated between 0.3 and 5ms.  I'm not sure what to do about that to be honest as that parameter always fixed all latency issues before.  I'm starting to wonder if I should consider the board defective and contact Intel about getting it replaced.

Bummer, I upgraded to the latest build this weekend and I've been having good success once I figured out the issue with my riser card.

Odd, my PPPoE stuff is working properly.  I know it took a minute to come up the first time but it has been stable since then.  Can you try re-using the buffalo config?  When I did my upgrade I copied everything in /etc/config over from the previous install.

I have no idea what would cause this.  If the on-board adapter works fine with other distro's I'd say it's something in the kernel, not the board.

I do have my auto-build stuff mostly complete.  If it helps a new build will be uploaded within the next few days (I'll be reporting back when it is online).  I plan on having the build bot run on the 1st and 15th of each month.  If it helps you out, I can run more frequent builds and hopefully things will get stable for you.  I am building the trunk so some things may not be 100%.

New build is online: https://nusku.net/openwrt/r195de1616405 … e6ecdeb70/

I have a build bot running now that will build on the 1st and 15th of each month and auto-publish.  I am working on an update in place program, will post back when that is complete as well.

Strange, seems its not likely to be that PPPoE command line as I finally got a PPPoE test server up and it connects to that instantly including setting up the routing.

The bigger problem is neither Intel card will keep the link up when connected to my modem, not just the on-board.

I'm glad to hear PPPoE looks to be working.  Bummer on the Intel cards.  I just checked the kernel config and it doesn't look like there is anything special set for the Intel cards beyond "enable driver".  Could it be the linux 3.3 kernel being problematic for the cards?

That is definitely an aspect to it, as it worked without issue on Mandriva 2011 which had a 2.6 kernel, I think the problem appeared in a later 2.6 version though.  But there has been plenty of commits since and I even tried compiling the latest version from Intels source which didn't fix it, very puzzling.

I just installed Windows 7 on it to see how that reacts and its fine, at least with the stock driver that comes with SP1, not tried seeing if it breaks if I use a newer driver.  Pinging it from my Linux machine its getting 0.2 to 0.6ms response, none of the 4ms spikes I was getting when it was running OpenWRT.  All the hardware assists seemed to be on too, which stomps on the theory that its one of those causing problems.

Its frustrating as IF anyone fixes this, they don't seem to have mentioned it online.  Plenty of people who had this and other problems with this LAN chipset on their servers, no fixes that I can find.

Is the 4ms response time ultimately an issue?  I would think you could get away with a 4ms response time while the driver situation gets figured out.

If not, does your box have any pciE or pci expansion?  You could go with a multi-port pciE or pci card in the interim while the on-board stuff is broken.

If it was JUST 4ms all the time, perhaps, but 4ms jitter could really mess up online gaming.  Although obviously the bigger issue is that I can't even get the link to stay up when plugged into the modem.

Gotcha, I don't do any online gaming so I wouldn't be affected by a constant 4ms delay.  I've heard 4ms can really mess up any kind of gaming.  I would imagine if there is jitter that would make it infintely worse.

If it helps uname -a returns "Linux beriberi 3.3.8 #14 SMP Thu Sep 6 12:10:55 EDT 2012 i686 GNU/Linux" on my install of r59e9c735c2ef86a73584c421eddae047c39f76a7.

Upgraded to latest build, grrrrrr:

64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=0.590 ms
64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=0.461 ms
64 bytes from 10.0.0.2: icmp_req=3 ttl=64 time=0.432 ms
64 bytes from 10.0.0.2: icmp_req=4 ttl=64 time=0.451 ms
64 bytes from 10.0.0.2: icmp_req=5 ttl=64 time=4.37 ms
64 bytes from 10.0.0.2: icmp_req=6 ttl=64 time=3.37 ms
64 bytes from 10.0.0.2: icmp_req=7 ttl=64 time=2.30 ms
64 bytes from 10.0.0.2: icmp_req=8 ttl=64 time=0.812 ms
64 bytes from 10.0.0.2: icmp_req=9 ttl=64 time=2.70 ms
64 bytes from 10.0.0.2: icmp_req=10 ttl=64 time=0.476 ms
64 bytes from 10.0.0.2: icmp_req=11 ttl=64 time=2.15 ms
64 bytes from 10.0.0.2: icmp_req=12 ttl=64 time=1.15 ms
64 bytes from 10.0.0.2: icmp_req=13 ttl=64 time=3.83 ms
64 bytes from 10.0.0.2: icmp_req=14 ttl=64 time=1.26 ms
64 bytes from 10.0.0.2: icmp_req=15 ttl=64 time=0.514 ms
64 bytes from 10.0.0.2: icmp_req=16 ttl=64 time=4.10 ms
64 bytes from 10.0.0.2: icmp_req=17 ttl=64 time=3.18 ms
64 bytes from 10.0.0.2: icmp_req=18 ttl=64 time=1.77 ms
64 bytes from 10.0.0.2: icmp_req=19 ttl=64 time=0.382 ms

Interseting.  I get a consistent .1-.2ms response time from my server to router on the wired lan.  Having said that my 5ghz wifi connection yoyo's between 10ms and 65ms.  Are you doing this over wifi or wired?

Wired.  Latency over wireless is FAR more stable - but they are only a couple of feet apart right now:

64 bytes from 192.168.1.111: seq=0 ttl=128 time=1.458 ms
64 bytes from 192.168.1.111: seq=1 ttl=128 time=1.195 ms
64 bytes from 192.168.1.111: seq=2 ttl=128 time=1.273 ms
64 bytes from 192.168.1.111: seq=3 ttl=128 time=1.100 ms
64 bytes from 192.168.1.111: seq=4 ttl=128 time=1.551 ms
64 bytes from 192.168.1.111: seq=5 ttl=128 time=1.248 ms
64 bytes from 192.168.1.111: seq=6 ttl=128 time=1.237 ms
64 bytes from 192.168.1.111: seq=7 ttl=128 time=1.273 ms
64 bytes from 192.168.1.111: seq=8 ttl=128 time=1.185 ms
64 bytes from 192.168.1.111: seq=9 ttl=128 time=1.247 ms
64 bytes from 192.168.1.111: seq=10 ttl=128 time=1.257 ms
64 bytes from 192.168.1.111: seq=11 ttl=128 time=1.249 ms
64 bytes from 192.168.1.111: seq=12 ttl=128 time=1.207 ms
64 bytes from 192.168.1.111: seq=13 ttl=128 time=1.421 ms
64 bytes from 192.168.1.111: seq=14 ttl=128 time=1.105 ms
64 bytes from 192.168.1.111: seq=15 ttl=128 time=1.239 ms

Then again the WiFi card in the router is IDENTICAL to the one in my laptop so should work perfectly together.

Seems like you and I have inverted problems.  Though I have a feeling my WiFi issues are due to being able to see >32 access points from my couch at all times...

What is your plan going forward if I may ask?  If there is something I can do with the build I'm willing to run an updated one with the necessary tweaks and work with you to get it nailed down 100% for your intel card(s).

Been looking at the latest drivers on the Intel site.

First step is probably to try using those as-is.  Failing that, the following may be relevant to my situation:

TROUBLESHOOTING: Some systems have trouble supporting MSI and/or MSI-X 
interrupts.  If you believe your system needs to disable this style of 
interrupt, the driver can be built and installed with the command:

     # make CFLAGS_EXTRA=-DDISABLE_PCI_MSI install

It would probably be useful to include ethtool in the build too so I can try forcing 100-BASE-T and see if that forces the link to stay up.

Incidentally, is there some way I can clone your OpenWRT build environment so I can try fiddling with the configuration myself?

I'll poke through the kernel config, maybe there is an option for turning off PCI_MSI support, if so I'll run an updated build.

I'll build a package file and add it to the list of packages to include.  [s]I likely won't include it in the default install but it will be available as a package going forward.[/s]

Edit: I just added ethtool as part of the base build. I'll post a link once the package is built.  Once I have ethtool taken care of I'll be looking at the kernel's PCI_MSI support.

Yep, see the below process for re-creating my environment.

# Check out the wiki and install the build deps and setup a fresh checkout of the sources
#     NOTE: I switched to git for managing my OpenWRT source tree and I have written some helper scripts that make building MUCH easier.  You want to do a checkout using git (see the bottom of the trac page that has repo links).
# Go to https://bitbucket.org/mcrosson/nnopenwrt and clone the project into a directory that is not the OpenWRT sources
cp /path/to/nnopenwrt/feeds.conf /path/to/openwrt/src/  # Setup the feed (package manager) config to include my package files
cd /path/to/openwrt/src
editor feeds.conf
    # Update the location of the nusku feed to be /path/to/nnopenwrt/feed
./scripts/feeds update -a  # Ensure the packages are up to date
for package in `cat /path/to/nnopenwrt/openwrt_packages`; do ./script/feeds install -p nusku $package; done
    # Install various packages and ensure you get what you need.  You'll see errors fly by when running this.  Ignore them.
cd /path/to/nnopenwrt
./bin/build.sh /path/to/openwrt/src/ /path/to/output/dir
    # This is a wrapper script I wrote that will run through a full build including any pre-patching that is necessary.  I use this with Jenkins to run my builds.

(Last edited by mcrosson on 12 Sep 2012, 15:11)

I forgot to mention in my previous post: If you run "make menuconfig" and update the main OpenWRT config at all, you'll need to manually update the config file to disable the serial console.  The config file is .config, if you do a search for "ttyS0", you'll find the two lines in question.

Also, the bitbucket project page has the issue tracker setup for tracking items that we have discussed and other things I've noticed while working on my builds.  Feel free to add if you find anything.

(Last edited by mcrosson on 12 Sep 2012, 15:08)

Ethtool packaging is done.  Link: http://dl.dropbox.com/u/23765633/openwr … -1_x86.ipk

You can pull the ipk right onto the router and install with: opkg install /path/to/ipk

Future builds will include ethtool as part of the base install.

@Alex: I looked closer at your findings with MSI and MSI-X and both are enabled in my kernel.  It looks like I should be able to build the kernel with that option disabled.

I am running an updated build and I'll publish for your testing as soon as it's complete.  I made a couple small kernel tweaks including the disable of MSI and MSI-X.

The bitbucket project site includes my changes from today, including the kernel changes.  I normally commit every change I make and push as I go along so others can see what is going on more easily.

(Last edited by mcrosson on 12 Sep 2012, 16:06)

@Alex: New build is online, this time without MSI interrupts.  Given your driver instructions showed to compile without the MSI interrupts I think this should be equivalent.  Let me know what the results are.

https://nuskunetworks.box.com/s/qc7e6sx07xdkaj43gn1v

It will be interesting to see if it causes any performance drop as I presume with classic interrupts it will use more CPU power. 

I think its plausible the new interrupts weren't working right as I noticed routing speed capping at 70Mbit when testing to my local PPPoE server yet it wasn't maxing the CPU.  Although the PPPoE server is also an Atom so it could have been maxing out that end, was going to test using my i5 as the server, will try the new build first.

I am not sure what the real difference is between classic interrupts and the new style.  I am also curious if it has any impact on my inability to use a gigE adapter in my PCI riser with the WiFi adapter.

I also have a full build uploading right now.  If the test build works for you, pull the latest from https://nusku.net/openwrt It has the same kernel changes as the experimental build, it's just a release based off my release tree.  I plan on migrating to a build without the MSI stuff once the upgrade in place tool is done.  I've started looking into the bits I'll need to write the program and hopefully I'll have it done within the next few weeks.

The test built certainly seems to have fixed the latency problem, although that power management hack I used seems to still make a very slightly improvement even on top of the interrupt fix.

Does this mean you have a stable line on the Intel cards?