Hi,

in openwrt package trunk you'll find an updated adblock package (plus LuCI companion/configuration package).

release 2.0.x

• support of 21 domain blacklist sites

• zero-conf like automatic installation & setup

• simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that

• full IPv4 and IPv6 support

• supports a wide range of router modes, even AP modes are supported

• support different download tools like wget (default), aria2c, uclient-fetch, curl

• each blocklist source will be updated and processed separately

• overall duplicate removal in separate adblock lists

• adblock source list parsing by fast & flexible regex rulesets

• additional white- and blacklist support for manual overrides

• quality checks during & after update of adblock lists to ensure a reliable dnsmasq service

• procd based init system support (start/stop/restart/reload/suspend/resume)

• procd based hotplug support, the adblock start will be triggered by interface triggers

• suspend & resume adblock actions temporarily without block list reloading

• query function to quickly identify blocked (sub-)domains, i.e. for whitelisting

• automatic block list backup & restore, backups will be (de-)compressed and restored on the fly

• add new adblock sources on your own via uci config

• LuCI configuration frontend (provided by hnyman)

update 2.1.x

• add initial unbound support (experimental, see online doc)

• automatically switch between wget & uclient-fetch

update 2.3.x

• automatically selects dnsmasq or unbound as dns backend

• add the new 'adguard' source, a combined/quite effective block list

• many improvements & fixes

update 2.4.x

• add tld compression ("top level domain compression")

The new "top level domain compression" removes up to 40 thousand needless host entries from the block lists and lowers the memory footprint for the dns backends by 8-10 MByte.

For Designated Driver and Chaos Calmer package installation details, please check the documentation link below!
A big 'thank you' to all beta testers and a special one to Hannu Nyman for his great adblock LuCI frontend!

Have fun!
Dirk

Link to the latest adblock documentation

(Last edited by dibdot on 3 Mar 2017, 18:45)

Script and documentation has been updated. It's tested against trunk only (min. version r47025). Prerequisites will be checked automatically during script start. If all goes well, it will be look like this excerpt:

root@pi2wrt:/usr/bin# logread -e "adblock"
Fri Sep 25 14:12:39 2015 user.notice adblock[3295]: start adblock processing (0.9.10pre)
Fri Sep 25 14:12:39 2015 user.notice adblock[3295]: environment check finished
Fri Sep 25 14:12:39 2015 user.notice adblock[3295]: get wan/update interface: wlan1, after 0 loops
Fri Sep 25 14:13:32 2015 user.notice adblock[3295]: get ntp time sync: 0.openwrt.pool.ntp.org, after 1 loops
Fri Sep 25 14:13:38 2015 user.notice adblock[3295]: shallalist archive download finished
Fri Sep 25 14:13:42 2015 user.notice adblock[3295]: shallalist archive extraction finished
Fri Sep 25 14:13:42 2015 user.notice adblock[3295]: shallalist (pre-)processing finished (adv costtraps downloads spyware tracker warez)
Fri Sep 25 14:13:42 2015 user.notice adblock[3295]: source download finished (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=one-line&showintro=0&mimetype=plaintext, 2422 entries)
Fri Sep 25 14:13:44 2015 user.notice adblock[3295]: source download finished (http://mirror1.malwaredomains.com/files/justdomains, 10826 entries)
Fri Sep 25 14:13:50 2015 user.notice adblock[3295]: source download finished (https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist, 521 entries)
Fri Sep 25 14:13:57 2015 user.notice adblock[3295]: source download finished (https://feodotracker.abuse.ch/blocklist/?download=domainblocklist, 0 entries)
Fri Sep 25 14:14:03 2015 user.notice adblock[3295]: source download finished (https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist, 12 entries)
Fri Sep 25 14:14:05 2015 user.notice adblock[3295]: source download finished (http://www.dshield.org/feeds/suspiciousdomains_Low.txt, 4542 entries)
Fri Sep 25 14:14:05 2015 user.notice adblock[3295]: source download finished (file:////mnt/sda1/adblock/adblock.blacklist, 3 entries)
Fri Sep 25 14:14:07 2015 user.notice adblock[3295]: source download finished (file:////tmp/tmp.cIKpEp/shallalist.txt, 36901 entries)
Fri Sep 25 14:14:20 2015 user.notice adblock[3295]: new block list with 52109 domains loaded, backup generated
Fri Sep 25 14:14:21 2015 user.notice adblock[3295]: finish domain adblock processing (0.9.10pre)

Have fun!
Dirk

(Last edited by dibdot on 25 Sep 2015, 13:22)

The release base of R47025 is for DD Trunk.  Do you plan to test this on the CC Final which I believe is this:
CC 15.05 Final   R46767      2015 September 10
https://forum.openwrt.org/viewtopic.php?id=59548

Regarding the prerequisites, I have a PC_ENGINES ALIX with 256MB memory and what ever size CF card I need.  Do I still need the external USB drive, or can this work in the available memory.  Alternatively, can I add sdx3 to my CF card and use this?

I've pushed 0.9.10 few minutes ago. Please take this version for tests with CC - it includes wget support for https sites (see openwrt ticket #19621 for details). In the script change the environment block to your needs, regarding CC support and space requirements you should check (space in kilobytes).

# minimal values for environment checks
#
min_release=47025
min_adb_space=100000
min_tmp_space=80000

Under which menu will I find this package?

please check git link in the first post

new pre-release 0.9.11 available, with the following changes:

* add support for two new blacklist sites (spam404.com & winhelp.mvps2002.org)
* add support for multiple ntp pools
* fix minor domain query glitches
* updated documentation

Have fun!
Dirk

Edit: Removed link to private git repo, cause package is now in trunk.

(Last edited by dibdot on 15 Nov 2015, 16:30)

Sorry, I can't test until the package is ready packaged for OpenWRT.

The new adblock package for openwrt (network/adblock) is now in trunk and ready for use. It will take a couple days for the package to get built by the buildbots, but it can be compiled now.

Feel free to test, ask questions or make suggestions.

Have fun!
Dirk

Grabbed a zip from the link above and looked at the code/configs. Quite a few of the blacklist URLs in the config are 404/not available.

Made me wonder how thoroughly was it tested before being published.

@stangri: which urls not working?
Just a rough guess: you didn't remove the "&ruleset=..." part during local testing in your browser!?

Thu Oct 29 05:40:55 2015 user.notice adblock[7693]: domain adblock processing started (0.11.0)
Thu Oct 29 05:40:55 2015 user.notice adblock[7693]: wan update check will be disabled
Thu Oct 29 05:40:55 2015 user.notice adblock[7693]: backup/restore will be disabled
Thu Oct 29 05:40:55 2015 user.notice adblock[7693]: error logging will be disabled
Thu Oct 29 05:40:55 2015 user.notice adblock[7693]: dns query logging will be disabled
Thu Oct 29 05:41:06 2015 user.notice adblock[7693]: get ntp time sync (ntp0.fau.de ntp1.fau.de ntp2.fau.de ntp3.fau.de), after 0 loops
Thu Oct 29 05:41:33 2015 user.notice adblock[7693]: shallalist archive download finished
Thu Oct 29 05:41:36 2015 user.notice adblock[7693]: shallalist archive extraction finished
Thu Oct 29 05:41:36 2015 user.notice adblock[7693]: shallalist (pre-)processing finished (adv costtraps downloads spyware tracker warez)
Thu Oct 29 05:41:37 2015 user.notice adblock[7693]: source download finished (http://pgl.yoyo.org/adservers/serverlist.php?hostformat=one-line&showintro=0&mimetype=plaintext, 2430 entries)
Thu Oct 29 05:41:40 2015 user.notice adblock[7693]: source download finished (http://mirror1.malwaredomains.com/files/justdomains, 12946 entries)
Thu Oct 29 05:41:41 2015 user.notice adblock[7693]: source download finished (https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist, 550 entries)
Thu Oct 29 05:41:44 2015 user.notice adblock[7693]: source download finished (https://feodotracker.abuse.ch/blocklist/?download=domainblocklist, 0 entries)
Thu Oct 29 05:41:46 2015 user.notice adblock[7693]: source download finished (https://palevotracker.abuse.ch/blocklists.php?download=domainblocklist, 11 entries)
Thu Oct 29 05:41:48 2015 user.notice adblock[7693]: source download finished (http://www.dshield.org/feeds/suspiciousdomains_Low.txt, 4542 entries)
Thu Oct 29 05:41:51 2015 user.notice adblock[7693]: source download finished (file:////tmp/tmp.fJaacL/shallalist.txt, 36937 entries)
Thu Oct 29 05:41:52 2015 user.notice adblock[7693]: source download finished (http://spam404bl.com/spam404scamlist.txt, 5189 entries)
Thu Oct 29 05:41:59 2015 user.notice adblock[7693]: source download finished (http://winhelp2002.mvps.org/hosts.txt, 13852 entries)
Thu Oct 29 05:41:59 2015 user.notice adblock[7693]: source download finished (file:///etc/adblock/adblock.blacklist, 1 entries)
Thu Oct 29 05:42:15 2015 user.notice adblock[7693]: new block list with 71167 domains loaded
Thu Oct 29 05:42:15 2015 user.notice adblock[7693]: domain adblock processing finished (0.11.0)

You're right, I didn't! Now that I did all the URLs in your list are working with a browser.

Thanks!

Hi,

new release 0.20.2 is in trunk:

* add uci support
* add dynamic uhttpd instance support (no longer rely on uhttpd config changes)
* bugfix: busybox "tr" does not support character classes by default and generates (partly) odd domain names.
* fix for possible query log config change (enabled => disabled)

Next to come ...

* bugfixes
* LUCI gui

Have fun!
Dirk

Please note: If you've already tested an older release before, please remove adblock uhttp config changes (/etc/uhttpd) manually.

(Last edited by dibdot on 15 Nov 2015, 16:39)

@dibdot

Could you clarify the config examples and README.md a bit especially regarding the network config options, as your examples do not really give much info about the actual goals. (There are so many router models that e.g. the default interface names and number vary a lot.)

Some questions about the model config:
- adb_ip "192.168.2.1" needs to be from a different different subnet than the normal LAN (typically 192.168.1.x). Right?
- To which interface does adb_dev "eth0" relate? wan, lan or something else/unused? E.g. I have by default lan as eth0.1 and wan as eth1. Should it be eth0.1 for me?
- How should adb_domain be selected? some common domain in local country?

Some feedback:
* I don't understand the need for ntp functionality. Openwrt already has normally ntp embedded in busybox, so having ntp launched here looks unnecessary.
* The minimum diskspace requirement 100 MB looks quite large. Probably some 95% of users have less RAM (and thus for ramdisk).

(Last edited by hnyman on 20 Nov 2015, 14:46)

@hnyman

Currently I'll prepare next release 0.21.0 with better documentation ...

adb_ip: needs to be a different subnet from the normal LAN, it's used by the adblock network interface and uhttpd instance.
adb_dev: should point to the default lan interface (check /sys/class/net/...), in your case I would suggest to use eth0.1.
adb_domain: this domain will be used for the final nslookup check (after dnsmasq reload with the newly generated blacklist). It should be an "always accessible" domain and it must not point to your adb_ip during nslookup check. The newly generated blacklist will be discarded, if this check failed.
adb_minspace: sorry, clearly a typo ... I've reduced this to 15MB in the default config.

The ntp check is only required for additional logfile logging (by default the script only logs to syslog fifo), to make sure to log the right date & time and for logfile housekeeping (delete logfiles older than n days, see adb_queryhistory).

Thanks
Dirk

So far I have not been able to got it to work. Two error messages pop up when starting the script. It gets partially initialised, so that dns probably works ok while the web server does not start ok.
...
EDIT
the error message is coming from the uhttpd startup. Interestingly, "ifstatus adblock" shows the new "adblock", but old "ifconfig" does not show it. Is that expected?

(Last edited by hnyman on 29 Feb 2016, 10:18)

@hnyman

By default, uHTTPd is bind to 0.0.0.0 (all ports, which also includes the WAN port of your router. To bind uHTTPd to the standard LAN port only you have to change the listen_http and listen_https options to your LAN IP address.

After that change try it again please.

Looks like that helped. I haven't yet tested widely, but now there are two uhttpd processes.
And trying a few blocked sites in web browser does not lead into LuCI anymore. Looks good so far.

(Last edited by hnyman on 29 Feb 2016, 10:18)

The port 80 definition should be sufficient. All ad related http traffic will be redirected to the adblock interface, for "https only" sites like facebook you'll get an "not found" - no timeouts (it's not possible to "fake" certificates for adblock purposes).

Is option adb_minspace "20000" for root? Seems like most devices wouldn't have nearly 20 MB free/total storage. If so, what's the best way to use /tmp?

Looks like that diskspace or actually free RAM is a really critical thing:

I can download all other domain lists except "shallalist". Trying to download just the shallalist (with current default selections) kills my router, as the memory/disk space requirement grows to over 30 MB at some point and the router dies

If I activate all other feeds, the resulting file is about 1.6 MB. So that is much.

But the critical thing is actually memory consumption during the update process.

Looking with ps (or htop) shows that "sort -u" command requires ~27 MB RAM at the peak:

root@OpenWrt:~# ps -w | grep sort
 6031 root     26652 R    sort -u
 6052 root      1192 S    grep sort

root@OpenWrt:~# which sort
/usr/bin/sort

root@OpenWrt:~# ls -l /usr/bin/sort
lrwxrwxrwx    1 root     root            17 Nov 21 11:47 /usr/bin/sort -> ../../bin/busybox

That might be due to weak "sort" implementation in busybox. Or something.

But that might form a problem for many users, as quite many routers have less RAM than mine (64 MB).

EDIT:
I installed GNU sort (package "coreutils-sort") and the peak memory consumption dropped to 4.4 MB ;-)

(But shallalist seems to take too much diskspace or RAM already before sort, so that does not work even then.)

(Last edited by hnyman on 21 Nov 2015, 21:12)

@roger_/hnyman: for best results please resize your tmp partition to 256MB. You'll find an example configuration in /etc/adblock/samples/rc.local.sample.

Yes, that is naturally possible, but that requires attaching a storage device to the router to have real effect on disk space.

And based on my debugging, it is not about disk space (in /tmp), but about actual RAM used by the process:

[ 3798.533895] Out of memory: Kill process 9406 (sort) score 408 or sacrifice child
[ 3798.541500] Killed process 9406 (sort) total-vm:26728kB, anon-rss:25572kB, file-rss:0kB

EDIT:
I tested the mount command in samples to lie about the tmpfs size. That brought no help, although "df" now claims /tmp size to be 256 MB.
Still the actual RAM gets consumed and router does oom:

[ 4692.225439] Out of memory: Kill process 10212 (sort) score 408 or sacrifice child
[ 4692.233489] Killed process 10212 (sort) total-vm:26604kB, anon-rss:25448kB, file-rss:104kB

(Last edited by hnyman on 21 Nov 2015, 22:39)

@hnyman: which router model did you use for testing? Maybe your router is to weak for shallalist processing ... I've tested the script with all sources successfully on wdr4300, wdr4900, dir-835 and a raspi2. All routers have at least 128MB ram ...

Today I used with WNDR3700v2 that has 64 MB RAM and is capable of running ~90 Mbit/s traffic with QoS enabled. Yesterday I was using WNDR3800 with 128 MB RAM, otherwise similar.

The router itself is capable of running adblock itself just nicely, but the 'sort' utility from busybox seems to choke with ~41000 lines to be sorted (other lists without shallalist). Memory consumption explodes for a few seconds to 26-27 MB, which I do not understand at all. The list itself is just 1.5 MB or so, so there should be no need to that kind of memory consumption.

Like I said, installing GNU sort (coreutils-sort package) helps. It takes only some 4.5 MB RAM at the peak.

The problem is not due to your script itself, but merely the RAM consumption by the commands it runs.
I edited the script, and saved the downloaded blocklist input to a file and tried to sort that. The same 26+ MB. So, it is not even about running too many piped things.

In the Openwrt router database, ~40 routers and ~40 computer boards out of total 488 have 128+ MB RAM. Probably the majority of Openwrt users use something with only 32-64 MB RAM.

You might maybe mention in the notes/guide something about "high memory consumption at the update phase if all blocklists are enabled", so that users will not run into unexpected problems if they try to download all the lists. One can easily avoid the problem by either downloading only one of the two large lists (malwaredomains & winhelp) or by installing GNU sort utility. (And of course, if the router has 128+ MB RAM, there is likely no problem at any phase.)  My point is merely to decrease the user frustration by providing them info about the expected resource consumption and possible mitigations (do not download all lists, install GNU sort).