openwrt can be built with different libcs. uClibc is the smallest, has the fewest features, and is the default libc. eglibc is larger and can be chosen from menuconfig. glibc is the largest (doesn't fit on many devices), is not supported in latest trunk, but is required for certain applications. eglibc is interesting in mid-range routers as a faster alternative to uClibc.
eglibc vs glibc was an interesting comparison between glibc and eglibc, with some mention of uClibc. In terms of final image size, glibc was largest, eglibc was 300K smaller, and uClibc 800K smaller (than glibc). (Note that the full image size was 10 MB, but this is very dependent on the set of packages). In terms of speed, there was no noticeable difference between glibc and eglibc, but uClibc was "substantially *slower*". (it's not clear however exactly how the speed measurements were done) Individual libc*.so sizes followed the trend, with glibc clocking in at 1474 KB, eglibc at 1375 KB and uClibc at 331 KB.
I was interested in comparing eglibc and uClibc in more detail. My router has a comfortable amount of flash, but I'm a little tight on memory so I wanted to make sure moving to eglibc wouldn't cost me too much RAM.
I did my tests using the "MIPS Malta CoreLV board (qemu)" target, Big Endian configuration, and ran them in qemu-system-mips for testing convenience. This is an ar71xx target and aside from a few drivers, should be a good model for other (real) ar71xx platforms. In particular I added a few IDE drivers to the kernel to support emulation but the userspace (libc, packages) should be the same as any other ar71xx target.
I selected a small set of packages that were interesting. The trunk version was r33867 plus some modifications just to the dropbear package to update it to 2012.55 (shouldn't affect anything outside the dropbear package). I used the default libc versions, so they're implicit in the trunk version, but I'll give them here for reference: uClibc 0.9.33.2, eglibc 2.15 with all options enabled (default).
I generated both a squashfs (for measuring variation in image size) and an ext4 image (for qemu). I started up the emulator, waited for it to settle down, then ran 'ps' to see process vsize and 'free' for overall memory usage.
Without further ado, here are the results:
Virtual Size
VSZ(uClibc) VSZ(eglibc) DELTA
1496 3420 +129% init
1492 3600 +141% /bin/ash --login
1496 3420 +129% init
1496 3420 +129% init
1500 3436 +129% /sbin/syslogd -C16
1480 3420 +131% /sbin/klogd
1184 2284 +93% /sbin/hotplug2 --override --persistent --set-rules-f
860 1968 +129% /sbin/ubusd
1504 2572 +71% /sbin/netifd
1184 2284 +93% /sbin/hotplug2 --override --persistent --set-rules-f
1484 3420 +130% /usr/sbin/telnetd -l /bin/login.sh
940 2200 +134% /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf
1492 3764 +152% /usr/sbin/ntpd -n -p 0.openwrt.pool.ntp.org -p 1.ope
1152 2360 +105% /usr/sbin/dropbear -P /var/run/dropbear.1.pid -p 22
0 0 - [ash]
1488 3600 +142% psMem Usage ('used' from output of 'free')
uClibc eglibc delta
14812 KB 17264 KB +17%Image Size (squashfs which is compressed, values in bytes)
uClibc eglibc delta
2'228'224 3'014'656 +35%Packages
uClibc eglibc delta [sizes in KB]
636 636 0% libopenssl_1.0.1c-1_malta_mips.ipk
228 1132 +396% libc_0.9.33.2-1_malta_mips.ipk
228 228 0% busybox_1.19.4-5_malta_mips.ipk
192 192 0% kmod-mac80211_3.3.8+2012-09-07-3_malta_mips.ipk
180 180 0% wpad-mini_20120910-1_malta_mips.ipk
108 108 0% kmod-ath9k-common_3.3.8+2012-09-07-3_malta_mips.ipk
108 108 0% ppp_2.4.5-6_malta_mips.ipk
96 96 0% kmod-cfg80211_3.3.8+2012-09-07-3_malta_mips.ipk
80 84 +5% dnsmasq_2.62-2_malta_mips.ipk
80 80 0% dropbear_2012.55-2_malta_mips.ipk
72 72 0% kmod-usb-core_3.3.8-1_malta_mips.ipk
60 60 0% opkg_618-2_malta_mips.ipk
56 56 0% kmod-ath9k_3.3.8+2012-09-07-3_malta_mips.ipk
44 44 0% zlib_1.2.7-1_malta_mips.ipk
40 40 0% iptables_1.4.10-4_malta_mips.ipk
40 40 0% iw_3.6-1_malta_mips.ipk
40 40 0% netifd_2012-10-19-a9fd870450e82f31cba681651656276dfc1c39ac_malta_mips.ipk
36 40 +11% libpthread_0.9.33.2-1_malta_mips.ipk
36 36 0% kmod-ipt-conntrack_3.3.8-1_malta_mips.ipk
36 36 0% libgcc_unknown-1_malta_mips.ipk
32 32 0% base-files_118-r33867_malta_mips.ipk
28 28 0% kmod-ipt-core_3.3.8-1_malta_mips.ipk
28 28 0% kmod-usb2_3.3.8-1_malta_mips.ipk
24 24 0% kmod-ppp_3.3.8-1_malta_mips.ipk
24 24 0% wireless-tools_29-5_malta_mips.ipk
20 20 0% hotplug2_1.0-beta-4_malta_mips.ipk
20 20 0% libuci_2012-03-28.1-1_malta_mips.ipk
16 16 0% firewall_2-53_all.ipk
16 16 0% kmod-crypto-aes_3.3.8-1_malta_mips.ipk
16 16 0% kmod-usb-ohci_3.3.8-1_malta_mips.ipk
16 16 0% libnl-tiny_0.1-3_malta_mips.ipk
12 12 0% kmod-ath_3.3.8+2012-09-07-3_malta_mips.ipk
12 12 0% kmod-ipt-nat_3.3.8-1_malta_mips.ipk
12 12 0% kmod-ipt-nathelper_3.3.8-1_malta_mips.ipk
12 12 0% libip4tc_1.4.10-4_malta_mips.ipk
12 12 0% libjson_0.9-2_malta_mips.ipk
12 12 0% libubox_2012-07-08-cca2ed6c8cdb3555fea43ff63ba1c9905dd7a164_malta_mips.ipk
12 12 0% libubus_2012-06-01-a62e2f8c15a65efb1a82404d8f31e3bfc1cbe7d9_malta_mips.ipk
12 12 0% libxtables_1.4.10-4_malta_mips.ipk
12 12 0% mtd_20_malta_mips.ipk
12 12 0% ppp-mod-pppoe_2.4.5-6_malta_mips.ipk
12 12 0% swconfig_10_malta_mips.ipk
12 12 0% uboot-envtools_2012.04.01-1_malta_mips.ipk
12 12 0% ubusd_2012-06-01-a62e2f8c15a65efb1a82404d8f31e3bfc1cbe7d9_malta_mips.ipk
8 20 +150% librt_0.9.33.2-1_malta_mips.ipk
8 12 +50% uci_2012-03-28.1-1_malta_mips.ipk
8 8 0% jshn_2012-07-08-cca2ed6c8cdb3555fea43ff63ba1c9905dd7a164_malta_mips.ipk
8 8 0% kmod-gpio-button-hotplug_3.3.8-1_malta_mips.ipk
8 8 0% kmod-nls-base_3.3.8-1_malta_mips.ipk
8 8 0% kmod-pppoe_3.3.8-1_malta_mips.ipk
8 8 0% libblobmsg-json_2012-07-08-cca2ed6c8cdb3555fea43ff63ba1c9905dd7a164_malta_mips.ipk
8 8 0% ubus_2012-06-01-a62e2f8c15a65efb1a82404d8f31e3bfc1cbe7d9_malta_mips.ipk
4 4 0% kernel_3.3.8-1-ea785713d2e509d1e9810d2967be628b_malta_mips.ipk
4 4 0% kmod-crypto-arc4_3.3.8-1_malta_mips.ipk
4 4 0% kmod-ledtrig-default-on_3.3.8-1_malta_mips.ipk
4 4 0% kmod-ledtrig-netdev_3.3.8-1_malta_mips.ipk
4 4 0% kmod-ledtrig-timer_3.3.8-1_malta_mips.ipk
4 4 0% kmod-ledtrig-usbdev_3.3.8-1_malta_mips.ipk
4 4 0% kmod-lib-crc-ccitt_3.3.8-1_malta_mips.ipk
4 4 0% kmod-pppox_3.3.8-1_malta_mips.ipkA few thoughts:
I wouldn't read too much into the vsize measurements. vsize measures the maximum amount of memory a process can access at the time it's measured, and some of this memory is shared between processes, so the number is larger than the actual slice of the 'used' memory taken by the process. It would be interesting to measure rss (resident set size aka working set) but my 'ps' command can't show that by default. It would also be much more useful to measure rss after some usage than after a fresh boot. The 'used' figures suggest that the RAM penalty with eglibc may be only ~17% so probably realistic for many devices.
As expected, the bulk of the size increase in the image (+~800KB) is due to libc. A few other packages grow slightly too, but the libc growth dwarfs the other packages. It's nice to see that individual packages aren't sensitive to the choice of libc (code is nicely modularized).
There is some potential for slimming down eglibc by disabling the many options under "eglibc configuration" but I have no indication of how big these savings could be.
Has anyone been running eglibc for a while on their routers? What version, what configuration, and what are your impressions?
Does anyone suspect what specifics of the eglibc implementation make it larger and faster than uClibc?
(Last edited by eigma on 25 Oct 2012, 07:25)
) benchmarks?