The OpenWrt build environment

One of the biggest challenges to getting started with embedded devices is that you just can't install a copy of Linux and expect to be able to compile a firmware. Even if you did remember to install a compiler and every development tool offered, you still wouldn't have the basic set of tools needed to produce a firmware image. The embedded device represents an entirely new hardware platform, which is incompatible with the hardware on your development machine, so in a process called cross compiling you need to produce a new compiler capable of generating code for your embedded platform, and then use it to compile a basic Linux distribution to run on your device.

The process of creating a cross compiler can be tricky, it's not something that's regularly attempted and so the there's a certain amount of mystery and black magic associated with it. In many cases when you're dealing with embedded devices you'll be provided with a binary copy of a compiler and basic libraries rather than instructions for creating your own -- it's a time saving step but at the same time often means you'll be using a rather dated set. Likewise, it's also common to be provided with a patched copy of the Linux kernel from the board or chip vendor, but this is also dated and it can be difficult to spot exactly what has been changed to make the kernel run on the embedded platform.

(Article continues below)

OpenWrt takes a different approach to building a firmware, downloading, patching and compiling everything from scratch, including the cross compiler. Or to put it in simpler terms, OpenWrt doesn't contain any executables or even sources, it's an automated system for downloading the sources, patching them to work with the given platform and compiling them correctly for the platform. What this means is that just by changing the template, you can change any step in the process.

As an example, if a new kernel is released, a simple change to one of the Makefiles will download the latest kernel, patch it to run on the embedded platform and produce a new firmware image -- there's no work to be done trying to track down an unmodified copy of the existing kernel to see what changes had been made, the patches are already provided and the process ends up almost completely transparent. This doesn't just apply to the kernel, but to anything included with OpenWrt -- It's this one simple understated concept which is what allows OpenWrt to stay on the bleeding edge with the latest compilers, latest kernels and latest applications.

So let's take a look at OpenWrt and see how this all works

download openwrt

This article refers to the "Kamikaze" branch of OpenWrt, which can be downloaded via subversion using the following command:

svn co https://svn.openwrt.org/openwrt/trunk

Additionally, there's a trac interface on http://dev.openwrt.org/ which can be used to monitor svn commits and browse the sources.

The directory structure

There are three key directories in the base:
- toolchain
- target
- package

Toolchain refers to the compiler, the c library, and common tools which will be used to build the firmware image. The result of this is two new directories, toolchain_build_<arch> which is a temporary directory used for building the toolchain for a specific architecture, and staging_dir_<arch> where the resulting toolchain is installed. You won't need to do anything with the toolchain directory unless you intend to add a new version of one of the components above.

Target refers to the embedded platform, this contains items which are specific to a specific embedded platform. Of particular interest here is the "target/linux" directory which is broken down by platform and contains the kernel config and patches to the kernel for a particular platform. There's also the "target/image" directory which describes how to package a firmware for a specific platform.

Package is for exactly that -- packages. In an OpenWrt firmware, almost everything is an ipk, a software package which can be added to the firmware to provide new features or removed to save space.

Both the target and package steps will use the directory "build_<arch>" as a temporary directory for compiling. Additionally, anything downloaded by the toolchain, target or package steps will be placed in the "dl" directory.

Building openwrt

While the OpenWrt build environment was intended mostly for developers, it also has to be simple enough that an inexperienced end user can easily build his or her own customized firmware.

Running the command "make menuconfig" will bring up OpenWrt's configuration menu screen, through this menu you can select which platform you're targeting, which versions of the toolchain you want to use to build and what packages you want to install into the firmware image. Similar to the linux kernel config,
almost every option has three choices, y/m/n which are represented as follows:

  <*> (pressing y) This will be included in the firmware image
  <m> (pressing m) This will be compiled but not included (for later install)
  <n> (pressing n) This will not be compiled

After you've finished with the menu configuration, exit and when prompted, save your configuration changes. To begin compiling the firmware, type "make". By default OpenWrt will only display a high level overview of the compile process and not each individual command.

  make[2] toolchain/install
  make[3] -C toolchain install
  make[2] target/compile
  make[3] -C target compile
  make[4] -C target/utils prepare
  ...

This makes it easier to monitor which step it's actually compiling and reduces the amount of noise caused by the compile output. To see the full output, run the command "make V=99".

During the build process, buildroot will download all sources to the "dl" directory and will start patching and compiling them in the "build_<arch>" directory. When finished, the resulting firmware will be in the "bin" directory and packages will be in the "bin/packages" directory.

Creating your own packages

One of the things that we've attempted to do with OpenWrt's template system is make it incredibly easy to port software to OpenWrt. If you look at a typical package directory in OpenWrt you'll find two things:

- package/<name>/Makefile
- package/<name>/patches

The patches directory is optional and typically contains bug fixes or optimizations to reduce the size of the executable. The package makefile is the important item, provides the steps actually needed to download and compile the package.

Looking at one of the package makefiles, you'd hardly recognize it as a makefile. Through what can only be described as blatant disregard and abuse of the traditional make format, the makefile has been transformed into an object oriented template which simplifies the entire ordeal.

Here for example, is package/bridge/Makefile:

include $(TOPDIR)/rules.mk

PKG_NAME:=bridge
PKG_VERSION:=1.0.6
PKG_RELEASE:=1

PKG_BUILD_DIR:=$(BUILD_DIR)/bridge-utils-$(PKG_VERSION)
PKG_SOURCE:=bridge-utils-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=@SF/bridge
PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd
PKG_CAT:=zcat

include $(INCLUDE_DIR)/package.mk

define Package/bridge
  SECTION:=base
  CATEGORY:=Network
  DEFAULT:=y
  TITLE:=Ethernet bridging configuration utility
  DESCRIPTION:=Ethernet bridging configuration utility\\\
    Manage ethernet bridging; a way to connect networks together to\\\
    form a larger network.
  URL:=http://bridge.sourceforge.net/
endef

define Build/Configure
  $(call Build/Configure/Default,--with-linux-headers=$(LINUX_DIR))
endef

define Package/bridge/install
        install -m0755 -d $(1)/usr/sbin
        install -m0755 $(PKG_BUILD_DIR)/brctl/brctl $(1)/usr/sbin/
endef

$(eval $(call BuildPackage,bridge))

As you can see, there's not much work to be done; everything is hidden in other makefiles and abstracted to the point where you only need to specify a few variables.

  PKG_NAME       - The name of the package, as seen via menuconfig and ipkg
  PKG_VERSION    - The upstream version number that we're downloading
  PKG_RELEASE    - The version of this package Makefile
  PKG_BUILD_DIR  - Where to compile the package
  PKG_SOURCE     - The filename of the original sources
  PKG_SOURCE_URL - Where to download the sources from
  PKG_MD5SUM     - A checksum to validate the download
  PKG_CAT        - How to decompress the sources (zcat, bzcat, unzip)

The PKG_* variables define where to download the package from; @SF is a special keyword for downloading packages from sourceforge. The md5sum is used to verify the package was downloaded correctly and PKG_BUILD_DIR defines where to find the package after the sources are uncompressed into $(BUILD_DIR).

At the bottom of the file is where the real magic happens, "BuildPackage" is a macro setup by the earlier include statements. BuildPackage only takes one argument directly -- the name of the package to be built, in this case "bridge". All other information is taken from the define blocks. This is a way of providing a level of verbosity, it's inherently clear what the DESCRIPTION variable in Package/bridge is, which wouldn't be the case if we passed this information directly as the Nth argument to BuildPackage.

BuildPackage uses the following defines:

Package/<name>
   <name> matches the argument passed to buildroot, this describes the package
   the menuconfig and ipkg entries. Within Package/<name> you can define the
   following variables:

   SECTION     - The type of package (currently unused)
   CATEGORY    - Which menu it appears in menuconfig
   TITLE       - A short description of the package
   DESCRIPTION - A long description of the package
   URL         - Where to find the original software
   MAINTAINER  - (optional) Who to contact concerning the package
   DEPENDS     - (optional) Which packages must be built/installed before this package

Package/<name>/conffiles (optional)
   A list of config files installed by this package, one file per line.
 
Build/Prepare (optional)
   A set of commands to unpack and patch the sources. You may safely leave this
   undefined.

Build/Configure (optional)
   You can leave this undefined if the source doesn't use configure or has a
   normal config script, otherwise you can put your own commands here or use
   "$(call Build/Configure/Default,<args>)" as above to pass in additional
   arguments for a standard configure script.

Build/Compile (optional)
   How to compile the source; in most cases you should leave this undefined.

Package/<name>/install
   A set of commands to copy files out of the compiled source and into the ipkg
   which is represented by the $(1) directory.
   
The reason that some of the defines are prefixed by "Package/<name>" and others are simply "Build" is because of the possibility of generating multiple packages from a single source. OpenWrt works under the assumption of one source per package makefile, but you can split that source into as many packages as
desired. Since you only need to compile the sources once, there's one global set of "Build" defines, but you can add as many "Package/<name>" defines as you want by adding extra calls to BuildPackage -- see the dropbear package for an example.

After you've created your package/<name>/Makefile, the new package will automatically show in the menu the next time you run "make menuconfig" and if selected will be built automatically the next time "make" is run.

Troubleshooting

If you find your package doesn't show up in menuconfig, try the following command to see if you get the correct description:

  TOPDIR=$PWD make -C package/<name> DUMP=1 V=99

If you're just having trouble getting your package to compile, there's a few shortcuts you can take. Instead of waiting for make to get to your package, you can run one of the following:

  make package/<name>-clean V=99
  make package/<name>-install V=99

Another nice trick is that if the source directory under build_<arch> is newer than the package directory, it won't clobber it by unpacking the sources again. If you were working on a patch you could simply edit the sources under build_<arch>/<source> and run the install command above, when satisfied, copy the patched sources elsewhere and diff them with the unpatched sources. A warning though - if you go modify anything under package/<name> it will remove the old sources and unpack a fresh copy.

Final notes

I'm always interested to hear about people's experience with OpenWrt or answer questions about it so please don't hesitate to contact me.

Thank you very much for this documentation!

I think it is very useful and should also go on the Wiki as soon as possible. So if someone has the chance to do that, it would be a very useful thing to do.

I tried to use this documentation to solve my problems with the CLASSIFY target for iptables (see http://forum.openwrt.org/viewtopic.php?id=6776). No full succes so far.

There was no problem putting the necessary patch into target/linux/generic-2.4/patches as 619-netfilter_classify.patch and it definitely gets applied -- the kernel config hangs at the CLASSIFY question, and when answered "m" the kernel builds without errors -- and build_mipsel/linux/net/ipv4/netfilter contains the ipt_CLASSIFY.o along with the other modules.

There is also no need to patch iptables, as iptables 1.3.3 and 1.3.5 (which were the versions in question during my experiments) both come with CLASSIFY support in their extension section, and seem to build the CLASSIFY target by default. In fact: libipt_CLASSIFY.so exists in the iptables build directory after the build.

So theoretically I guess I could now grab the binaries directly and copy them into the right places and that ought to work, but I want to solve this issue correctly and get proper packages for OpenWRT.

So let me continue, because this is where my problem starts: Even though I tried to edit the configuration files as best I could to make a kmod-ipt-classify and iptables-mod-classify, only the iptables-mod-classify_1.3.5-1_mipsel.ipk is afterwards found in bin/packages -- and it is empty, so useless.

Also the kmod-ipt-classify does not exist, at all.

So this is where the documentation could still be improved I think:
* Which are the relevant files to add a kernel patch and afterwards have the module be built and packaged into an additional ipkg?
* Which are the relevant files to add to iptables (or some other pre-existing package) to actually package files that are included in the default (or a patched) compile, but are not packaged for OpenWRT by default?

Regards,
Georg

a quick note, after checking out /trunk, you should probably checkout /packages from svn too.

after you've check them both out to the same folder (so you have trunk and packages as directories) you can then make symlinks from trunk/package that point to the folders in packages/*

e.g.

hydra@hydra02:~/openwrt/svn$ cd trunk/package
hydra@hydra02:~/openwrt/svn/trunk/package$ ln -s ../../packages/utils/vim vim
hydra@hydra02:~/openwrt/svn/trunk/package$ ls -al v*
lrwxrwxrwx 1 hydra hydra 24 2006-10-30 22:19 vim -> ../../packages/utils/vim

after you've created symlinks you can then run "make .pkginfo" in trunk and then grep ".pkginfo" (also in trunk) to check the packages were found correctly.

after that run make menuconfig and the packages will show up in the menus

Hope this helps

(Last edited by Hydra on 30 Oct 2006, 23:33)

Hi MBM!

I want to access my DVD-Drive via USB and made a kernel module and package for isofs (UDF should follow) with OpenWRT's great buildroot. I was astonished, it is really cool!

But my module doesn't work till now and I don't know/understand why.

Nevertheless I have some notes and suggestions for buildroot and its documentation:

* As far as I understand the makefiles download and build many files after subversion
checkout. Some of those files are symbolic links, e.g. build_mipsel/linux
and seem to be generated, too. But those symbolic links arent't relativ, therefore they don't work anymore,
if the main directory is copied or moved to another. Wouldn't it be possible
to create them relative?

* There seems one error in the buildroot documentation:
And copy it, so your changes are not getting lost, when doing a 'make
dirclean'. Here we assume that you are compiling for Broadcom chipset based
devices:
$ cp .config ../../../target/linux/linux-2.4/config/brcm

Seems as it should be:
$ cp .config ../../target/linux/linux-2.4/config/brcm
(Only two levels higher not three?)

* It seems that the buildroot-documentation is missing some important information about some
parameters.

In buildroot documentation you can find:

Define the binary files for the kernel module
Define the binary files for the kernel module by modifying/inserting into
target/linux/linux-2.4/Makefile, e.g.
$(eval $(call KMOD_template,USB_KEYBOARD,usb-kbd,\
        $(MODULES_DIR)/kernel/drivers/input/input.o \
        $(MODULES_DIR)/kernel/drivers/input/keybdev.o \
        $(MODULES_DIR)/kernel/drivers/usb/usbkbd.o \
,CONFIG_USB_KEYB,kmod-usb-core,60,input keybdev usbkbd))

Where CONFIG_USB_KEYB is the kernel option, USB_KEYBOARD is the last part
of BR2_PACKAGE_KMOD_USB_KEYBOARD and usb-kbd is part of the filename of the
created ipkg.

But what is kmod-usb-core,60,input keybdev usbkbd? And which .o-files must be selected for other modules?

I assume, that the parameter "kmod-usb-core" is a dependency.
O.k., I already found that the parameter with value "60" determines the order of execution in the boot process in /etc/modules.d directory.
"input keybdev usbkbd" seems to be the filenames without extension that have been selected.
A hint which files must be included would be helpful. For the module isofs it seems that only 1 of 7 is relevant (isofs.o).

* As far as I can see, allthough my module/package was build correctly, it is still not the same as other modules/packages.
Allthough I possibly did not understand everything, I currently assume, I did everything right.
I looked at the parameters for vfat and ext2 and so on and did it the same:
$(eval $(call KMOD_template,ISOFS,isofs,\
        $(MODULES_DIR)/kernel/fs/isofs/isofs.o \
,CONFIG_ISO9660_FS,,30,isofs))

After installation I executed ipkg status (dumps status for all packages) and found:
Package: kmod-isofs
Version: 2.4.30-brcm-4
Depends: kernel (2.4.30-brcm-4)
Status: install user installed
Architecture: mipsel
Installed-Time: 946685499

But the other modules show something like:
Package: kmod-vfat
Version: 2.4.30-brcm-3
Depends: kernel (2.4.30-brcm-3)
Status: install user installed
Section: sys
Architecture: mipsel
maintainer: OpenWrt Developers Team <openwrt-devel@openwrt.org>
MD5Sum: 1ca7867fa6bc38780c38a3d99a28cf91
Size: 32021
Filename: kmod-vfat_2.4.30-brcm-3_mipsel.ipk
Source: http://svn.openwrt.org/openwrt/branches … /linux-2.4
Description: Kernel modules for VFAT filesystem support

Apparent differences:
sys section (my modules seems not to be in sys section allthough configured like written in buildroot documentation!)
version (my module has brcm-4, the standard package for vfat and others have brcm-3!)

* At least I don't know whether the differences between standard packages and my package that I detected above are the cause for my problems or if I did something different wrong. If I can solve my problems, I can donate the module/package(s), if appreciated.

Well, these are some hints, where the buildroot and its documentation can be updated and perhaps you can even help me solve my CD/DVD-access.
I already followed USB-Howto. My two devices are already detected correctly, e.g. dmesg currently says:
hub.c: 4 ports detected
hub.c: new USB device 01:03.2-1, assigned address 2
scsi0 : SCSI emulation for USB Mass Storage devices
  Vendor: HP        Model: CD-Writer+ 8100   Rev: 1.0g
  Type:   CD-ROM                             ANSI SCSI revision: 02
WARNING: USB Mass Storage data integrity not assured
USB Mass Storage device found at 2

best regards

Thorsten

While attempting to make changes to kernel setting that are not handled by the OpenWRT config, I'm struggling with getting the kernel to recompile.  Is there a simple way to do this?

Edit the target/linux/<platform>/config file and run make. It'll recompile the kernel...

That did it. 

Thanks!

HI MBM and NBD and other guys,

Would that be possible to set up an regular training class in a convenient location , maybe, for one week, teaching us how to use all these tools?

These would cover many howtoes, including set up development environment on linux pc, difference of tools (SDk, imagebuilder, Buildroot). live example of compiling image and package, (e.g. Asterisk). how to help port updated/new softwares to openwrt, etc.?

Hope it's also a good opportunity for us to get together and no expenshive training like other vendor offers.

tks
peng

I had a problem with a Makefile, and some minutes ago I fixed it.
When I typed "make menuconfig", buildroot blocked itself on script/metadata.pl. That because it was an error on the dependencies, there were some recursive dependencies. SDK doesn't care about that, but Buildroot needs to know the dependencies to build its menu.
I updated http://wiki.openwrt.org/BuildRoot page, to avoid someone else to waste time as I did

Dear MBM,

I think Buildroot of OpenWRT is very similar with OpenEmbedded(). But Buildroot is easy to install and use, OE is hard. So, what do you think Buildroot compared with OE? Any comments? Thanks.

Regards,
heha

Is there a really simple doc that describes the procedure of writing your own code and then compiling it as binary and running it on the embedded linux platform. For example, I would like to write a simple xxx.cpp file then compile it and then run it on my wrt54g box?

how do I go by doing this? Using simple printf... etc c++ source?

Thanks

It's efficient to develop program for embedded devices under Openwrt environment.

hello,

How to add two ip addresses to a router with whiterussian?
Is there a configuration file? if so, where is it? How can I change it?

thanks.

Thank you very much for this documentation!

please feel free to review my to websites:

http://www.taraji-online.com

http://www.ghar-elmelh.com

(Last edited by fahmouch on 29 Jul 2009, 19:02)

I'm just coming up to speed with compiling, adding packages, etc and this documentation is great!

Adding packages is no longer cumbersome and works as described!   Thank you!

Now for my question:

One thing that I'm working on now requires recompiling drivers built into the kernel. 

Is there a way to only compile the kernel and have buildroot-ng skip everything else, like the packages (at least) and kernel modules (perhaps) but still build the final image? 

I fear that the make process is spending much more time than I would like spanning the depths of all of the modules and packages when I know the only thing that has changed is one file for a built-in kernel driver.

Thanks!

-Rich

hi,
i have installed kamikaze_8.09 on my machine.
now i want to install the firmware on qemu(emulator).
can you suggest which file is the image after building kamikaze and where it is and how i can make it boot-able so that it can run on qemu safely?
thanks

Thanks for this thread guys.

I am new with using openwrt and also new in this forum.

This thread is very informative and very helpful.

Thanks a lot.

Emo Quiz

MBM   thank you so much for this introduction. I'm compiling my 1st Open-WRT  ever, hope to keep learning and doing more.

If you build over NFS, you might have to do the following:

In ~/.autom4te.cfg.. add

begin-language: "Autoconf-without-aclocal-m4"
args: --no-cache
end-language: "Autoconf-without-aclocal-m4"

Otherwise autoconf fails to configure.  It looks like it hangs forever without disabling the cache.

hello MBM !
Thanking you for this documentation.
I am Michel and I'm new for openwrt and also new in this forum.
This documentation may very useful and helpful for many of us.
You did such a great job.
Thanks once again.

Hey!

In make menuconfig, during configuration which option i need to check?

- build OpenWRT Image
- build OpenWRT SDK
- build OpenWRT Toolchain

I want to make my own image, so i wait finally an ipg file. But i am not sure, cause all the three do something similar.

Cheers, János

(Last edited by kukodajanos on 24 Feb 2011, 08:05)

A simple "make" should suffice. This will compile the toolchain, and the toolchain will then crosscompile the sources into binaries. Then packages will be built. And then the firmware will be built out of the packages. So, as long as you do not already have a current toolchain, let him compile one.

http://wiki.openwrt.org/doc/howto/firmware.compile

Orca - Nice work with this wiki page!

Well, nice work for the entire wiki.

Cheers!