I do not have a spare router on which to try Kamikaze, so I'll just post here some random things I hope are fixed compared to WhiteRussian:
1 - the main one: make upgrading safer.  Everytime a new WhiteRussian release has come up, I learned about it because some of my router's functionality got trashed by "ipkg upgrade".  E.g. today when I run "ipkg upgrade" on my RC6 router, it started to install presumably 0.9 packages, overwriting my /etc/passwd and various other files. Ipkg has no business removing my users and passwords.
BTW, I see nowhere any info indicating whether "ipkg upgrade" did successfully
upgrade my router to 0.9 or if it only did part of it.  I'll probably re-flash 0.9 anyway just to save flash space (by moving the new modules from /jffs to /rom), but I'd be interested to learn whether there's something more to it.
2 - make upgrading easier.  Upgrading from one release to the next is painful: after re-flashing I have to figure out all over again all the files I had changed.
3 - /tmp and /var are different.  dhcp.leases should be in /var, not in /tmp (so that the info is not lost across reboots).  BTW, I have split them here and most things work fine after the split.  I did notice that webif looks for /tmp/dhcp.leases rather than /var/dhcp.leases.

For reference here is the output of "ipkg upgrade" when I ran it today.  All in all, I don't think it has erased too much stuff, but throwing away my root password got me seriously scared since I couldn't log back in via SSH (luckily I still had left the webif reachable indirectly and this worked to reset the password).

~# ipkg upgrade
Upgrading base-files on root from 8 to 9...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
ipkg: /rom/note: Read-only file system
ipkg: Cannot create symlink from ./var to '/tmp': File exists
Upgrading busybox on root from 1.00-4 to 1.00-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading dnsmasq on root from 2.33-1 to 2.35-1...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
    Configuration file '/etc/dnsmasq.conf'
    ==> File on system created by you or by a script.
    ==> File also in package provided by package maintainer.
       What would you like to do about it ?  Your options are:
        Y or I  : install the package maintainer's version
        N or O  : keep your currently-installed version
     The default action is to keep your current version.
    *** dnsmasq.conf (Y/I/N/O) [default=N] ?
Upgrading kmod-brcm-wl on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-crypto on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-ext3 on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-gre on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-ipt-conntrack on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-ipt-nat-extra on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-ipt-nat-pptp on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-mppe on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-ppp on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-pppoe on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-usb-core on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-usb-ohci on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-usb-printer on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-usb-storage on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading kmod-usb-uhci on root from 2.4.30-brcm-4 to 2.4.30-brcm-5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading mtd on root from 4 to 5...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Upgrading uclibc on root from 0.9.27-8 to 0.9.27-9...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Installing libgcc (3.4.4-9) to root...
Downloading http://downloads.openwrt.org/whiterussi … mipsel.ipk
Configuring base-files
Configuring busybox
Configuring dnsmasq
Configuring kmod-brcm-wl
Configuring kmod-crypto
Configuring kmod-ext3
Configuring kmod-gre
Configuring kmod-ipt-conntrack
Configuring kmod-ipt-nat-extra
Configuring kmod-ipt-nat-pptp
Configuring kmod-mppe
Configuring kmod-ppp
Configuring kmod-pppoe
Configuring kmod-usb-core
Configuring kmod-usb-ohci
Configuring kmod-usb-printer
Configuring kmod-usb-storage
Configuring kmod-usb-uhci
Configuring libgcc
Configuring mtd
Configuring uclibc
Successfully terminated.

Sorry - ipkg upgrade will never work.

The real problem is the design of the system; there's just not a heck of a lot available in terms of resources so you really have to work hard to cram everything into as little space as possible. For those of you that aren't familiar with the full technical details of the device let's start out with the basics. Your typical router:

* 4M flash
* 16M ram

The first 256k of flash is eatten by the bootloader, this is basically the bios of the device, required to boot into the OS (ie the openwrt firmware). At the end of the flash is nvram, taking the last (64k) block of flash.  This leaves you with just over 3.6M to fit your entire OS - kernel and filesystem.

[ Bootloader ] [ Firmware ................... ] [ NVRAM ]

The bootloader really has no concept of a filesystem; it just assumes that the firmware starts with executable code, meaning that our kernel is contained outside the filesystem. Technically we could put some sort of code at the start of the firmware to locate the filesystem and pull the kernel out of it, but that would require space for the bootstrap code and more space used by the kernel due to filesystem overhead. The best thing we've come up with so far is to highly compress the kernel using lzma, and boot into an lzma decompressor, making it look something like this:

[ Bootloader ] [ LZMA + Kernel ] [ Rest of firmware .... ] [ NVRAM ]

Now comes the filesystem, we want to make the filesystem as big as possible, that means that we want to start the filesystem immediately after the end of the kernel. Well what happens if the kernel changes? It's not like we left any extra room for a larger kernel - this means that every time the kernel changes it can potentially wipe out a good chunk of the start of the filesystem. Ouch, obviously we're going to have to recreate the filesystem every time the kernel changes.

But let's talk about the filesystem itself; you want a filesystem that's readable and writable, but you also want a filesystem that compresses well. Unfortunately the tradeoff with having a writable filesystem is that you can't compress the files nearly as tightly as you could if the filesystem was readonly. Squashfs gets better compression than JFFS2 but the Squashfs filesystem is readonly. Ouch this is starting to get annoying. Ok let's set up the flash as follows:

[ Bootloader ] [ LZMA + Kernel ] [ Squashfs ] [ Jffs2 ........ ] [ NVRAM ]

The idea is that we'll try to cram as much common functionality as possible into Squashfs to get the best compression, but we'll try to leave out task specific items that not everyone will use -- no point wasting flash space for features the end user isn't interested in, and if they are interested they can just load the package onto JFFS2.

We can hide the split between JFFS2 and Squahsfs by using either symlinks or mini_fo. In the past we've used symlinks from JFFS2 to files on squashfs, but more recently we've switched over to using mini_fo. The mini_fo module provides a copy on write filesystem, meaning that if you try to change anything on the squashfs partition, it will just copy the file over to the jffs2 filesystem and make the changes there.

So what happens with the ipkg upgrade?

This is tricky, there's no way to change the kernel without nuking the filesystem, and if we try to change any of the files that came with the firmware, all we'll really be doing is downloading packages to the JFFS2 partition - there's no way to rewrite the squashfs partition without reflashing. So eventually we'll fill up with JFFS2 partition and our Squashfs partition will go unused because all the applications are now on JFFS2.

This was intentional.

/tmp is a ramdisk

OpenWrt's policy is to never write to flash unless explicitly instructed to do so (with the exception of the first bootup). The basic idea is that frequent writes to the flash will eventually wear out the flash; OpenWrt is designed such that it can run forever without risk of crashing due to flash failure.

I understand that "ipkg upgrade" can't upgrade the kernel, of course.
But there are two issues:
1 - make sure that "ipkg upgrade" doesn't screw up existing files (such as /etc/passwd, /etc/profile, /etc/group, ...).  Clearly this can be done (as my sample session showed: it was careful with /etc/dnsmasq.conf).  Also maybe it should try to avoid (or at least warn the user when) trying to upgrade a package that is in the squashfs part.
2 - make it easier to preserve local changes when upgrading from one version of the firmware to another.  One way I can think of would be to write a tool that takes the old firmware and generates a "diff" by comparing it to the current state of the system, and then another tool that can apply this diff to the system after upgrading to the new firmware.

OK, that makes sense.  But I believe that OpenWRT should still work correctly if the user chooses to split the two.  I.e. all tools should be careful to use /tmp or /var appropriately, in case /var is not a symlink to /tmp.
I.e. it's still a bug for webif to refer to /tmp/dhcp.leases rather than /var/dhcp.leases.

Sorry for hijacking the thread, but anyways, kudos to mbm for a great write-up! I'm not that literate when it comes to linux kernel and OS internals, and it was a great read to help understand insides of OpenWRT a little better.

@mbm:  Very good review off embendded openwrt system, how about put this on wiki .

Please put it to wiki if not yet. Just incidently found this great inside in forum internals.

Yes, it's great inside. I couldn't find this kind of stuff on wiki and it would be nice to have it there.