OpenWrt Forum Archive

Topic: luci stops running after a while; how can I restart it?

The content of this topic has been archived on 30 Mar 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

Post #1
jacobq
19 Apr 2017, 16:54

When my OpenWRT router (ar71xx running r49388) boots up everything seems to work fine. However, after a while (not sure when it happens, but right now it has been up 43 days and is happening now) I can't login to the web interface any more. When I go to the URL (192.168.1.1) Firefox says

> Bad Gateway
> The process did not produce any response

and Google Chrome spins for a while then says:

> This site can’t be reached
> 192.168.1.1 took too long to respond.

In the past when I've encountered this I've just rebooted the device, but right now I can't because it's in production, and I'd have to wait for scheduled downtime, so I'd like to learn how to manually fix whatever is breaking in this case.

When I attempt to run 

/ww_/cgi-bin/luci

(replace _ with w; stupid link blocker is tripping on it) I get the following output:


/usr/bin/lua: /ww_/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
stack traceback:
    /ww_/cgi-bin/luci:4: in main chunk
    [C]: ?


That file is just a few lines long:

#!/usr/bin/lua
require "luci.cacheloader"
require "luci.sgi.cgi"
luci.dispatcher.indexcache = "/tmp/luci-indexcache"
luci.sgi.cgi.run()

If I comment out or delete the 3rd line (dispatcher) I get the following error instead:

/usr/bin/lua: /ww_/cgi-bin/luci:4: attempt to index field 'sgi' (a nil value)
stack traceback:
    /ww_/cgi-bin/luci:4: in main chunk
    [C]: ?

so it seems like somehow required modules aren't getting loaded or something, but I don't know how to debug this further. Could anyone help me figure out what is wrong or where I should look next?

Post #2
hnyman
19 Apr 2017, 17:38

uhttpd is the actual www server process running in the router. look if that is still running.
If not, restart it with "/etc/init.d/uhttpd restart"
And you should look into system log messages related to uhttpd.

LuCI itself is just lua scripts & html.

Post #3
jacobq
19 Apr 2017, 18:01

I forgot to mention that I already tried stopping and starting uhtt pd (how were you able to post that with setting off the stupid/broken link blocker?) to no effect. How can I view log messages though? I'm used to seeing something like 

/var/log/messages

or 

/var/log/syslog

but all I see in that folder is:

# ls -alh /var/log
drwxr-xr-x    3 root     root         100 Mar  1 17:04 .
drwxrwxrwt   17 root     root         480 Apr 19 10:29 ..
drwxr-xr-x    2 root     root          60 Mar  1 17:04 ddns
-rw-r--r--    1 root     root           0 Mar  1 17:04 lastlog
-rw-r--r--    1 root     root           0 Mar  1 17:04 wtmp
Post #4
tmo26
19 Apr 2017, 18:45
jacobq wrote:

How can I view log messages though?

Try:

logread
dmesg
Post #5
jacobq
19 Apr 2017, 18:54

Thank you for the logread tip. (I knew about dmesg, but it looks like it only has start-up log messages)

I didn't see any messages from uhtt pd but did see some other messages which indicated that there is no space left in /tmp! I have a 128GB USB flash storage device attached to this unit. Is there any way I can use that to increase the capacity of tmp?

...
Wed Apr 19 12:46:40 2017 daemon.err dnsmasq-dhcp[1443]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
Wed Apr 19 12:46:42 2017 daemon.info dnsmasq-dhcp[1443]: DHCPINFORM(br-lan) 192.168.1.153 00:25:64:88:b9:80 
Wed Apr 19 12:46:42 2017 daemon.info dnsmasq-dhcp[1443]: DHCPACK(br-lan) 192.168.1.153 00:25:64:88:b9:80 (host name 1)
Wed Apr 19 12:46:42 2017 daemon.err dnsmasq-dhcp[1443]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
Wed Apr 19 12:47:39 2017 daemon.info dnsmasq-dhcp[1443]: DHCPINFORM(br-lan) 192.168.1.133 00:25:64:88:e5:a3 
Wed Apr 19 12:47:39 2017 daemon.info dnsmasq-dhcp[1443]: DHCPACK(br-lan) 192.168.1.133 00:25:64:88:e5:a3 (hostname 2)
Wed Apr 19 12:47:39 2017 daemon.err dnsmasq-dhcp[1443]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
Wed Apr 19 12:47:48 2017 daemon.info dnsmasq-dhcp[1443]: DHCPINFORM(br-lan) 192.168.1.153 00:25:64:88:b9:80 
Wed Apr 19 12:47:48 2017 daemon.info dnsmasq-dhcp[1443]: DHCPACK(br-lan) 192.168.1.153 00:25:64:88:b9:80 (hostname 1)
Wed Apr 19 12:47:48 2017 daemon.err dnsmasq-dhcp[1443]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
Wed Apr 19 12:48:48 2017 daemon.err dnsmasq-dhcp[1443]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
Wed Apr 19 12:49:28 2017 daemon.info dnsmasq-dhcp[1443]: DHCPINFORM(br-lan) 192.168.1.153 00:25:64:88:b9:80 
Wed Apr 19 12:49:28 2017 daemon.info dnsmasq-dhcp[1443]: DHCPACK(br-lan) 192.168.1.153 00:25:64:88:b9:80 (hostname 1)
Wed Apr 19 12:49:28 2017 daemon.err dnsmasq-dhcp[1443]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
root@OpenWrt-4106:/# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 2.0M      2.0M         0 100% /rom
tmpfs                    61.3M     61.3M         0 100% /tmp
/dev/sda1               108.1G    694.8M    102.0G   1% /overlay
overlayfs:/overlay      108.1G    694.8M    102.0G   1% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda3                 3.6G      7.4M      3.4G   0% /mnt/sda3
root@OpenWrt-4106:/#
Post #6
hnyman
19 Apr 2017, 19:03
jacobq wrote:

Is there any way I can use that to increase the capacity of tmp?

Buy a device with larger RAM amount...
tmp = ramdisk

The core issue is that something in your system fills up the /tmp directory.
Maybe some logging from a package?

In a typical router there is only a few megabytes, even after running some time.

root@OpenWrt2:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.0M      5.0M         0 100% /rom
tmpfs                    29.5M      1.5M     28.0M   5% /tmp
...

My router has 1.5 MB data in /tmp, while your router has 61 MB data there.

Ps.
One alternative might be to manually mount /tmp into your large storage device, but it seems overkill compared to preventing the actual disk space usage there.

Pps.

(how were you able to post that with setting off the stupid/broken link blocker?)

Probaly due to having posted ~2900 messages. You have the "new user" spam limitations.

(Last edited by hnyman on 19 Apr 2017, 19:06)

Post #7
jacobq
19 Apr 2017, 19:08

I found that a 60MB+ openvpn log file was filling up all the tmp space, so I moved it out of there (will see about fixing log settings later). So there should be plenty of space there, but it's acting like it wasn't freed. When I run `df /tmp` I see 100% use, but when I run `du -sh /tmp` I see 84.0K. Also, when I try to make a new file (e.g. with text editor or `cat > /tmp/file.txt`) I get a `write error: No space left on device` message. Any idea what I might need to do to get it to realize that there is free space there now?

Post #8
jacobq
19 Apr 2017, 19:10

Also, when I run `free` it looks like there is memory available...

             total       used       free     shared    buffers     cached
Mem:        125524     122176       3348      62760       7232      90988
-/+ buffers/cache:      23956     101568
Swap:      2097148          0    2097148
Post #9
jacobq
19 Apr 2017, 19:16

Nevermind; I figured out the free space issue based on hXXp://askubuntu.com/questions/209934/why-do-df-and-du-show-different-output
I ran `/etc/init.d/openvpn restart` so that it would close the log file and allow the space to be freed.
Unfortunately, this hasn't solved the luci problem yet. I restarted uht tpd again and now I see that error message I got when running luci from the command line in the log messages:

Wed Apr 19 13:15:17 2017 daemon.err uhttpd[9902]: /usr/bin/lua: /www/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
Wed Apr 19 13:15:17 2017 daemon.err uhttpd[9902]: stack traceback:
Wed Apr 19 13:15:17 2017 daemon.err uhttpd[9902]:     /www/cgi-bin/luci:4: in main chunk
Wed Apr 19 13:15:17 2017 daemon.err uhttpd[9902]:     [C]: ?
Wed Apr 19 13:15:18 2017 daemon.err uhttpd[9902]: /usr/bin/lua: /www/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
Wed Apr 19 13:15:18 2017 daemon.err uhttpd[9902]: stack traceback:
Wed Apr 19 13:15:18 2017 daemon.err uhttpd[9902]:     /www/cgi-bin/luci:4: in main chunk
Wed Apr 19 13:15:18 2017 daemon.err uhttpd[9902]:     [C]: ?
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]: /usr/bin/lua: /www/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]: stack traceback:
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]:     /www/cgi-bin/luci:4: in main chunk
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]:     [C]: ?
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]: /usr/bin/lua: /www/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]: stack traceback:
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]:     /www/cgi-bin/luci:4: in main chunk
Wed Apr 19 13:15:19 2017 daemon.err uhttpd[9902]:     [C]: ?
Wed Apr 19 13:15:20 2017 daemon.info dnsmasq-dhcp[1443]: DHCPINFORM(br-lan) 192.168.1.153 00:25:64:88:b9:80 
Wed Apr 19 13:15:20 2017 daemon.info dnsmasq-dhcp[1443]: DHCPACK(br-lan) 192.168.1.153 00:25:64:88:b9:80 JOHN-OFFICE
Wed Apr 19 13:15:20 2017 daemon.err uhttpd[9902]: /usr/bin/lua: /www/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
Wed Apr 19 13:15:20 2017 daemon.err uhttpd[9902]: stack traceback:
Wed Apr 19 13:15:20 2017 daemon.err uhttpd[9902]:     /www/cgi-bin/luci:4: in main chunk
Wed Apr 19 13:15:20 2017 daemon.err uhttpd[9902]:     [C]: ?
Wed Apr 19 13:15:26 2017 daemon.err uhttpd[10154]: /usr/bin/lua: /www/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
Wed Apr 19 13:15:26 2017 daemon.err uhttpd[10154]: stack traceback:
Wed Apr 19 13:15:26 2017 daemon.err uhttpd[10154]:     /www/cgi-bin/luci:4: in main chunk
Wed Apr 19 13:15:26 2017 daemon.err uhttpd[10154]:     [C]: ?
Wed Apr 19 13:15:30 2017 daemon.err uhttpd[10154]: /usr/bin/lua: /www/cgi-bin/luci:4: attempt to index field 'dispatcher' (a nil value)
Wed Apr 19 13:15:30 2017 daemon.err uhttpd[10154]: stack traceback:
Wed Apr 19 13:15:30 2017 daemon.err uhttpd[10154]:     /www/cgi-bin/luci:4: in main chunk
Wed Apr 19 13:15:30 2017 daemon.err uhttpd[10154]:     [C]: ?

(Last edited by jacobq on 19 Apr 2017, 19:40)

Post #10
rlei
25 May 2017, 17:25

@jacobq I just solved a very similar issue to yours and I hope my findings might be helpful to you:

Basically when /tmp runs out of space, luci scripts might have already created some corrupted code cache file in /tmp. All you need to do is to remove /tmp/luci-indexcache and /etc/luci-modulecache/* so that luci can recreate all cache on next visit. And the error will be gone.

Post #11
jacobq
25 May 2017, 17:43

@rlei, thanks for the tip. Unfortunately, I do not see any of those files in existence on my router. 

find /|grep -i luci|grep -i cache

returns no results as well.

(Last edited by jacobq on 25 May 2017, 17:43)

The discussion might have continued from here.