Hi there,
Sometimes you run out of flash memory when installing too many packages. There are quite a few howtos out there (quite different approches too), which allow you to extend the space onto an usb-stick.
In my opinion the "cleanest" solution is to replace the jffs filesystem (which contains all filesystem-modifications after flashing the firmware) with a ext3 filesystem on the usb-stick AT BOOT-TIME. (Unfortunately it is also the most nasty one to implement.)
The advantages besides much more space and no router flash-wearout are:
* If you screw up the config and cause the router to no longer allow you to ssh in, just put the stick into another linux box and undo what you did wrong. Flip it back in, reboot and you can log in again.
* You no longer have to reconfigure everything after flashing, because your configuration, your packages and your user data is not erased. Just use opkg to update stuff. This allows you to easily do a kernel upgrade.
* No multiple configuration files as with the other "solutions".
* You can easily backup and revert configurations.
However which device-node contains the ext2 mini_fo partition that needs to mounted, if you have 3 usb storage devices connected? (might be /dev/sda1 /dev/sdb1 or /dev/sdc1)
If you mount the wrong one, your router will think that it is starting for the first time and wait for you to telnet it at 192.168.1.1... not so good.
(some people just plug the others devices in when the router has finished booting - isn't a clean solution either)
So? Why not mount the ext2 mini_fo partition by ID?
I'll show you what you need to do on a WL500GP V1 -> Be warned that it is not for the faint-hearted.
Also be warned that this is just "my soup" I think the soup tastes ok, however many others probably won't like it.
Also note two things:
1) Things might become easier soon, because devtmpfs is being included in the kernel mostly voiding the need for udev.
2) The stuff below works for me, but probably still has some bugs with error-handling. It was done with kamikaze r18759 which is Kernel 2.6.30.10.
##########################
How JFFS works in Kamikaze
##########################
First an overview on how the whole JFFS thing works when you have flashed Kamikaze:
The flash-memory of the router consists of 2 parts:
* The rom flash memory part. The read-only firmware, which you flashed (openwrt-brcm47xx-squashfs.trx). -> Everything you see in /rom
* The jffs flash memory part. The writable flash-filesystem which is empty after flashing. -> Everything you see in /jffs
These two flash-memory-parts (filesystems) are then combined to what you see as /.
E.g. create a file with "echo bla > /etc/config/test.txt" and it will be stored as /jffs/etc/config/test.txt which is in the jffs flash-memory-part.
So, if you want to start over without reflashing the router, just do "rm -rf /jffs/*" and reboot afterwards. All your changes will be deleted.
Note: Normally you should not directly manipulate the /jffs directory.
Look up mini_fo on the net for more infos...
--
When the router starts FOR THE FIRST time, the jffs flash-memory-part is still blank or full of useless data:
* The router loads the firmware from the rom flash-memory part
* It loads the kernel modules which it finds in the rom flash-memory
* It finds out that the jffs-memory-part is blank.
* It therefore formats the jffs-memory-part,
* mounts the jffs memory part and
* does some initialisation stuff, like creating ssh keys.
When the router is booting subsequently:
* It loads the firmware from the rom flash memory part again.
* It loads the kernel modules which it finds in the rom flash-memory
* However now it finds out that the jffs flash memory part is already formated.
* So instead of doing initialisation stuff, it just mounts the jffs flash memory part.
Note: First or subsequent boot has nothing to do with telnetting to the router and setting the root password with passwd. You can do that after 10 reboots if you like... (your passwd file is stored in jffs by the time you set the password)
#######################
So what is the concept?
#######################
The concept is: -> We simply do not use the jffs flash memory part. <-
However this means, that everything we need to get the usb-stick mounted, needs to be readily avaible in the rom flash memory part. Therefore all that stuff needs to be included in openwrt-brcm47xx-squashfs.trx AND FLASHED TO THE ROUTER ALLTOGETHER.
If you have a clever box like the ASUS WL500-GP V1, you won't have to worry much about bricking your router, since you always have diag mode in case you flash some crap into the router. On routers which do not have something like this, I would not recommend messing around with the firmware this radically unless you know what you are doing.
Preparing a new firmware includes:
* kernel modules
* other packages
* modified scripts and config files
Since we want to mount by ID, we also need to include the ID of the USB-Stick in openwrt-brcm47xx-squashfs.trx.
In more detail: we need the following:
* kernel modules
* other packages
udev
mount-utils
* modified scripts and config files
-> To get all that stuff into the openwrt-brcm47xx-squashfs.trx image, you need to use the openwrt image builder.
###########
Preparation
###########
First you need to get kamikaze and compile it.
(Read through http://kamikaze.openwrt.org/docs/openwrt.html for more info)
svn checkout svn://svn.openwrt.org/openwrt/trunk kamikaze
cd kamikaze
make menuconfig
I recommend choosing Target Profile "No Wifi" and enabling "Choose all modules by default" under "Global build settings". However it does not really matter at this point.
Choose the correct Target System. In case of the WL-500GP it is "BCM947xx/953xx".
You might also want to enable IPV6...
You have to enable "Build the OpenWrt Image Builder"
Then exit and save your configuration.
You might also symlink-in some additional packages e.g:
./scripts/feeds update
./scripts/feeds install loop-aes
(linking in all feeds is probably not the brightest idea)
I recommend doing
make download V=99
before the actual "make".
Then build it (takes very long)
make V=99
(If you want to hack in some strange code from tutorials - this is the time. Do make V=99 again after you are done.)
Congratulations: you now have all files to build your image from
If you go to /bin/brcm47xx you will find the file "OpenWrt-ImageBuilder-brcm47xx-for-Linux-x86_64.tar.bz2"
Copy the file to a different location and unpack it with
tar xfvj OpenWrt-ImageBuilder-brcm47xx-for-Linux-x86_64.tar.bz2
Time to add some packages to the image:
Now we create a separate profile in the subdirectory:
cp target/linux/brcm47xx/profiles/110-None.mk target/linux/brcm47xx/profiles/myprofile.mk
And modify it a bit so that it look somwhat like this:
#
# Copyright (C) 2006-2008 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#
define Profile/myprofile
NAME:=No WiFi
PACKAGES:=kmod-usb-core kmod-usb-uhci kmod-usb2 e2fsprogs kmod-scsi-core kmod-usb-storage kmod-fs-ext2 kmod-fs-ext3 udev
endef
define Profile/myprofile/Description
Package set without WiFi support
endef
$(eval $(call Profile,myprofile))
All the above packages are necessary to mount an USB-stick by ID. Do not include packages which are not necessary for that! You can install everything else later when the image is flashed using opkg! Including Wifi.
Controlling the image-builder is pretty strange, so you might want to do "make help".
When you are done reading, do:
make image PROFILE="myprofile"
Be sure you have no typo in there, because the image-builder will build something anyway - just not what you want. Check the first lines of the output, whether the packages you want are listed.
Reissuing the same command will clean the build-dir and all subsequent modifications. So do "make build_image" to rebuild the image instead!
Udev is a bit of a strange program: it gives you the cool /dev/disk/ directory. However on openwrt it also wants to make your interface names unique (by renaming them). This totally confuses openwrt, so you have to get rid of this behaviour.
#Modification 1#
Simply delete the rules-file by doing
rm ./build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/lib/udev/rules.d/75-persistent-net-generator.rules
Time to modify something interesting now...
#####################
Modifying the scripts
#####################
Now we modify some files in the image-builder directory "/build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/".
Do not do "make clean" now, as this will delete your modified files.
#Modification 2#
First create a file called "stick_id" in
/build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/
with
echo "/dev/disk/by-id/usb-Kingston_DataTraveler_2.0_5B800A001388-0:0-part1" > ./build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/stick_id
Use the ID of your USB-Stick parition here!
(plug it into a linux system and watch /dev/disk/by-id)
(while you are at it, format the usb-stick partition with "mkfs.ext3")
#Modification 3#
Then create a file called "stick_prepare" in
/build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/
and put the following into it:
#!/bin/sh
. /etc/functions.sh
usbdev=$(cat /sbin/stick_id)
load_modules /etc/modules.d/*
##lsmod >>/tmp/bootinfo.txt
killall udevd
udevd --daemon
udevadm trigger
udevadm settle
COUNTER=0
while [ ! -b "$usbdev" ] && [ $COUNTER -lt 15 ]; do
##echo "mount sleep for $usbdev" >>/tmp/bootinfo.txt
sleep 1
let COUNTER=COUNTER+1
done
make the file executable with
chmod 755 /build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/stick_prepare
#Modification 4#
Then create a file called "stick_mount" in
/build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/
and put the following into it:
#!/bin/sh
usbdev=$(cat /sbin/stick_id)
if [ -z "${1}" ]; then
echo "where do you want to mount it?"
exit
fi
##ls -la /dev/ >> /tmp/bootinfo.txt
e2fsck -p "$usbdev"
mount "$usbdev" "${1}" -t ext3 -o barrier=1,noatime,nodiratime,data=ordered >>/tmp/bootinfo.txt
##echo "mount 1" >> /tmp/bootinfo.txt
make the file executable with
chmod 755 /build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/stick_mount
#Modification 5#
Now change the end of the following file accordingly:
/build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/firstboot
(delete or comment-out the minus-rows and add the plus-rows. And try to understand what you are doing )
....
# invoked as an executable
[ "${0##*/}" = "firstboot" ] && {
- [ -z "$mtdpart" ] && {
- echo "MTD partition not found."
- exit 1
- }
[ -z "$rom" ] && {
echo "You do not have a squashfs partition; aborting"
echo "(firstboot cannot be run on jffs2 based firmwares)"
exit 1
}
[ "$1" = "switch2jffs" ] && {
if grep -q mini_fo /proc/filesystems; then
- mount "$mtdpart" /rom/jffs -t jffs2 || exit
+ #mount the stick
+ /sbin/stick_prepare
+ /sbin/stick_mount /rom/jffs
# try to avoid fs changing while copying
mount -o remount,ro none / 2>&-
# copy ramoverlay to jffs2
echo -n "copying files ... "
cp -a /tmp/root/* /rom/jffs 2>&-
echo "done"
# switch back to squashfs (temporarily)
# and park the ramdisk ontop of /tmp/root
pivot /rom /mnt
mount -o move /mnt /tmp/root
# /jffs is the overlay
# /rom is the readonly
fopivot /jffs /rom
# try to get rid of /tmp/root
# this will almost always fail
umount /tmp/root 2>&-
else
# switch back to squashfs temporarily
pivot /rom /mnt
# get rid of the old overlay
umount -l /mnt
# another umount to get rid of the bind from /tmp/root
umount -l /mnt
- # initialize jffs2
- mount "$mtdpart" /jffs -t jffs2 || exit
+ #mount the stick
+ /sbin/stick_prepare
+ /sbin/stick_mount /jffs
# workaround to ensure that union can attach properly
sync
ls /jffs >/dev/null
# switch to the new (empty) jffs2
fopivot /jffs /rom 1
# copy ramoverlay to jffs2, must be done after switching
# to the new rootfs to avoid creating opaque directories
echo -n "copying files ... "
cp -a /tmp/root/* / >/dev/null 2>&1
sync
echo "done"
umount -l /jffs
umount -l /tmp/root
fi
exit 0
}
# script run manually
[ \! -z "$jffs" ] && {
echo "firstboot has already been run"
echo "jffs2 partition is mounted, only resetting files"
grep mini_fo /proc/filesystems >&-
[ $? != 0 ] && {
dupe $jffs $rom
exit 0
} || {
rm -rf $jffs/* 2>&-
mount -o remount $jffs / 2>&-
exit 0
}
}
- mtd erase "$partname"
- mount "$mtdpart" /jffs -t jffs2
+ #You should automatically format the usb-stick
+ #with ext3 here, however this is risky
+ #So better format the stick yourself on another linux box
+ #mount the stick
+ /sbin/stick_prepare
+ /sbin/stick_mount /jffs
fopivot /jffs /rom 1
}
#Modification 6#
And there is a second file you have to modify:
/build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/sbin/mount_root
#!/bin/sh
# Copyright (C) 2006 OpenWrt.org
. /etc/functions.sh
jffs2_ready () {
- mtdpart="$(find_mtd_part rootfs_data)"
- magic=$(hexdump $mtdpart -n 4 -e '4/1 "%02x"')
- [ "$magic" != "deadc0de" ]
+ /sbin/stick_prepare
+
+ #... and return whether it exists...
+ usbdev=$(cat /sbin/stick_id)
+ [ -e "$usbdev" ]
}
grep rootfs_data /proc/mtd >/dev/null 2>/dev/null && {
. /sbin/firstboot
- mtd unlock rootfs_data
jffs2_ready && {
echo "switching to jffs2"
- mount "$(find_mtd_part rootfs_data)" /jffs -t jffs2 && \
- fopivot /jffs /rom
+ /sbin/stick_mount /jffs
+ fopivot /jffs /rom
} || {
echo "jffs2 not ready yet; using ramdisk"
ramoverlay
}
} || {
mtd unlock rootfs
mount -o remount,rw /dev/root /
}
#Modification 7#
The last thing is to do is to remove the script for hotplugging block-devices (so that usb-sticks don't automount)
rm ./build_dir/target-mipsel_uClibc-0.9.30.1/root-brcm47xx/etc/hotplug.d/block/10-mount
#########
Finish it
#########
Well, all you have to do now is
make build_image
to update the image. Do not do "make clean" or "make image PROFILE="myprofile"" of course, or your hacks will be lost.
Then scp the image in /bin/brcm47xx to the router and flash it with "mtd -r write scp openwrt-brcm47xx-squashfs.trx linux" to your box. (be patient - booting for the first time takes a litte longer...)
Do that at your own risk of course. I personally would only flash a customized image like that, if the router has a diag-mode for recovery.
Telnet the router and set a password with "passwd"
Do "mount" and watch the output. It should look something like this:
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=14848k)
tmpfs on /dev type tmpfs (rw,relatime,size=512k)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/disk/by-id/usb-Kingston_DataTraveler_2.0_5B810A000A26-0:0-part1 on /jffs type ext3 (rw,noatime,nodiratime,errors=continue,barrier=1,data=ordered)
mini_fo:/jffs on / type mini_fo (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)
Reboot it with "reboot".
If you can login with ssh and your new password, you have won. Congratulations!
Start installing hundreds of packages with opkg.
###############################################################
Appendix1: Why you should not put unnecessary things into the image
###############################################################
Usually the Wifi kernel module is loaded at boot. However since wifi-modules are sometimes buggy, you might upgrade them from time to time without reflashing the router.
You might say now that you can simply update the package by "opkg remove kmod_madwifi" and "opkg install kmod_madwifi", because the new version is stored in JFFS. This however will not work as expected:
In our case the problem is that before your router mounts the modifyable JFFS, you make the router load all kernel modules which are in the firmware (with the "load_modules /etc/modules.d/*" line in the script)
(You need this for clever things like "ext2", which allow you to mount your stick before JFFS)
There is actually no good reason to include WIFI drivers directly into the firmware image, because WIFI is disabled anyway after flashing the router. And if you are clever you install packages like hostapt and wpa_supplicant before enabling wifi.
So why not install the wifi kernel module after flashing?
You could of course replace the line "load_modules /etc/modules.d/*" with several "insmod bla". However why should you?
Just include things you absolutely need to get JFFS working. Put the rest in JFFS! This applies to all modules!
(I tried creating a complementary "unload_modules" which was called after mounting in "firstboot" and "mount_root" - however this removes some modules which are not properly marked as "used". This causes you to loose the filesystems and wifi if it was up already.)
############################
Appendix2: Which filesystem?
############################
One thing before we start: Why ext3 and not ext2 or ext4?
*) In its current state we can exclude ext4 from our decision. Search the web for it - you will find out why...
*) Ext2 is exremely stable, however it has one disadvantage: It tends to cache everything for a long time -> If you pull the plug, it is very likely that not everything is committed.
Of course you can circumvent that by mounting ext2 with the option "sync". However this wears out the flash, since every single change is written for itself. And the usb-stick has to rewrite at least one eraseblock for each change.
(So if you do a loop and add a single character to a file a few million times, you will have seriously decreased the lifetime of your stick.)
*) Ext3 is the best choice, because it writes a journal for all changes. And every 5 seconds the journal is being committed to the actual filesystem. That means your filesystem is usually good 5 seconds after each write.
If you pulled the plug before the 5 second commit, the journal will be replayed when mounting it the next time.
However if you pull the plug when the journal is being written (very unlikely, but possible), the filesystem might get messy. Why? The journalling concept depends on the fact that the journal is written down linearly. However if a very clever harddisk does "out-of-order-writes" (change the order in which data is written down, in order to increase write-speed), the end of the journal might be written before the start of the journal is written. When this "journal" is replayed after reboot, your filesystem might get corrupted.
However your fellow filesystem developers addressed this issue by introducing "write barriers". These barriers are set by the ext3 kernel module and simply force the harddisk to write everything down when such a barrier is reached. This is a perfect tradeoff between write performance and data integrity. You can enable this with the "barrier=1" mount option.
Unfortunately is is not that simple with usb-sticks The sticks act like a harddisk, however how exactly data is read and written is a secret which is implemented in the controller-chip on the stick. The logic in the chip is pretty complex, because it distributes writes over the flash memory. The stick basically has its own "flash-filesystem" between you and the flash chip. If this wouldn't be happening, you could easily kill the stick by formatting it as ext2 and writing one inode-table 10000 times. (maybe appending one char to a file 10000 times)
This secret filesystem most likely has a small RAM in order to perform write operations.
So even if you set write barriers, and kill the power while it is writing, not everything from the small RAM may be written and you might damage the "secret" flash-filesystem of the stick. And this cannot be sorted out by a high level filesystem-check.
So to sum up:
Your box
Ext? filesystem
Filesystem cache (In Kernel Ram)
USB-Stick
(Harddisk-like write-cache)
Secret flash-filesystem <-> small Ram
Flash Memory
You can eliminate the "Filesystem cache (in Kernel Ram)" problem by using the right filesystem (ext3) and specifying clever mount options like "data=journal" or "data=ordered"
You can eliminate the "Harddisk-like write cache" problem of the stick by specifying the "barrier=1" mount option, or by disabling write caching of the device alltogether (e.g. with "hdparm -W" or so)
However you cannot influence the "small Ram" of the usb-stick flash file system.
(I just had an clever idea... maybe solder a capacitor onto the usb-stick, so that it has enough power to commit its small ram when the plug is pulled in an inconvient moment, hmmm... will have to think about that... Maybe there is already one on them...)
So, to sum up: Use ext3 with "barrier=1,data=ordered"
------------------
lg, Mr.M
------------------
References:
http://wiki.openwrt.org/oldwiki/replace … ernalmedia
https://lists.openwrt.org/pipermail/ope … 04892.html
I can't remember where I found the infos about the filesystems... however I guess they can be considered to be common knowledge you can find them on the net.
Edit: Included mounting with noatime, nodiratime,data=ordered
(Last edited by mr.m on 19 Dec 2009, 08:42)