A port of the DIFFUSE (http://caia.swin.edu.au/urp/diffuse/) classification system for Attitude Adjustment. Provides automated QoS for your router (based on a flow’s statistical properties, not port numbers or payloads). A firmware image (for the TPLink WR-1043ND), packages, source and documentaion are available at: http://caia.swin.edu.au/urp/diffuse/openwrt/

This is prototype work, not rigorously debugged nor ready for prime time. But we hope it is nevertheless of some interest!

About DIFFUSE
-------------

DIFFUSE provides dynamic/automated traffic classification and prioritisation of IP flows using a Machine Learning classification model. DIFFUSE will calculate flow statistics (such as mean packet length, mean packet inter-arrival time) for bi-directional IP flows that have an entry in the flow table (e.g. flows between a LAN host and an Internet host).

By enabling DIFFUSE on a router, latency-sensitive applications (online games, VoIP) are automatically identified and prioritised. You do not need to pre-configure port numbers or LAN hosts (i.e. classification will occur independently of the port number used or the LAN host generating the traffic).

DIFFUSE is currently integrated with the IPFW/Dummynet Linux port, using IPFW for flow statistics calculation and Dummynet for priority queuing and bandwidth shaping.

There is an included sample model that will identify and prioritise First Person Shooter traffic heading upstream (from LAN to WAN). There is an automated process for building your own classification models to discriminate different types of IP traffic.

Summary
-------

OpenWRT Version:
- Attitude Adjustment (trunk) r29537

Available downloads:
- TPLink WR-1043ND firmware with DIFFUSE included
- TPLink WR-1043ND packages for Attitude Adjustment
- All source files
- Sample classification model and shell script to automate setup
- VirtualBox Ubuntu VM with all sources needed to build packages/firmware (includes OpenWRT trunk/r29537 and DIFFUSE source code)
- Step-by-step instructions on getting it all running

(Last edited by nw on 18 Apr 2012, 08:27)

Hi.
looks very very interesting.
Please submit it as new package and it make everybody happy.
https://dev.openwrt.org/timeline or https://lists.openwrt.org/mailman/listi … nwrt-devel

Looks very interesting, but configuring it to work even with the current versions of the ar71xx target seems a bit hard, as there are hard-coded paths in several places in the makefiles.

The package for Openwrt would create much wider interest, if the it would define dependencies more clearly and adapt easily to different architectures regarding the compilation itself. Right now it is just a collection of four separate packages with makefiles needing editing.

Yes, the Makefiles are not ideal (unfortunately I didn't have enough time to clean up the Makefiles much). I will try to address some of the issues over the next little while whenever I can, but agreed for now it may not be the most streamlined of installs (though hopefully not too difficult). But it's out there if anyone wants to tinker.

Anyway thanks for the feedback, I'll be sure to fix up the paths as soon as I get a chance.

I managed to easily compile one of of the four packages for my ar71xx/wndr3700, but others started to complain about include paths, so it needs more work.

I thought that easiest might be to create a meta-package on top of those four packages, and mark it dependent on those.

Is the intention, that all four packages are always installed? If so, then the Makefiles might actually be combined to one makefile with all the definitions. That would also ease in finding all the places with references to the paths.

I think that ideally the packages should be placed in the feeds/packages/(net) section, not in the 'package' repository, as 'package' is mainly for the core components. (I don't see this as a core component yet, but it might be permanent enough to have a place in the feeds/packages).

From the version management perspective, the wrapping should also include the correct download path, so that selecting the packages at menuconfig the then download the current version's source code from your server at the compilation time. (That is the easy part on the package config, but I mention it as a part of the usual practice. Rather seldomly the full source code of the package is included in Openwrt SVN repository.)

You probably don't need to worry about installing all four packages unless you are running flow classifications on another box and exporting them to the router. Try the 'ipfw' package and the 'dummynet2' package ('dummynet2' is basically the kernel module for ipfw, and the 'ipfw' package is the userspace portion of ipfw).

I think package/ipfw should compile okay (if there are any absolute paths, like KERNELPATH ?= or USRDIR ?= in the Makefile just comment them out). The dummynet2 package does have some paths that need changing, these could potentially be an issue:

in dummynet2/Makefile change:
LINUX_DIR=<userpath>/openwrt/trunk/build_dir/linux-ar71xx_generic/linux-2.6.39.4

also in dummynet2/src/Makefile
KERN_SRC=<userpath>/openwrt/trunk/build_dir/linux-ar71xx_generic/linux-2.6.39.4

you might also want to add the cross compiler to your PATH
<userpath>/trunk/staging_dir/toolchain-mips_r2_gcc-4.5-linaro_uClibc-0.9.32/bin/  (your toolchain might be the same for the wndr3700)

I don't know if you tried the VM but if you don't mind a ~300MB download it is basically what I used to build the firmware and packages, so there shouldn't be much messing around with the Makefiles (you will need to run the config and compile the source though...).

I tried a quick build of the firmware/packages with your target using the VM... see http://caia.swin.edu.au/urp/diffuse/ope … iffuse.tgz I don't have the hardware to test the packages/firmware but you could try them if you are game.

I think the idea of combining the Makefile is a good one, and it would be nice to have it in the OpenWRT svn for an easier install, but time constraints mean I won't be able to do this right away.

I actually tried the VM and it started ok as an environment. (I am using Virtualbox and Ubuntu 11.10 as my main Openwrt environment, so this is familiar ground.)

However, I didn't complete the compilation there, as my goal is to integrate this to my already existing build, which has kernel already at 3.2.15 as of yesterday (and 3.3.2 since later today thanks to recent checkins). See details here: https://forum.openwrt.org/viewtopic.php?id=28392

Having the kernel path, Linux path etc. hardcoded in the makefiles makes it a bit hard.

I will try to convert those to use the normal (STAGING_DIR) (BUILD_DIR) etc. conventions normally used in Openwrt makefiles.

Your clarification that only two modules are need, is great.

(Last edited by hnyman on 20 Apr 2012, 07:55)

Hi hnyman, did you ever get a release with this working and, if so, could you make it available please

Thanks

No, I dropped my trials as the new Codel logic as the main QoS works well enough. That was introduced roughly at the same time. The Diffuse code was too tied to the old kernel and environment. They had not tried to create a clean and flexible package, but had just hard-coded their own code too integrated to their specific development environment, probably more like proof-of-concept way than a package for long-term maintenance.

(Last edited by hnyman on 21 Nov 2012, 10:18)