OpenWrt Forum Archive

Topic: Update Local Host with mvps block list, Merge, clean with script

The content of this topic has been archived on 7 Apr 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

I am using the mvps.com list for blocking marketing servers, the script I created does the following.

Downloads http://winhelp2002.mvps.org/hosts.txt file and created a local file with a clean version of the hosts file.

Takes the existing local hosts file and merges it with the new downloaded hosts file.

Removed duplicates

Additional option to remove specific entries in case you run into problems.

Copies it to the /etc/hosts file, and loads the settings.

I have it scheduled to run once a month via cronjob

wget -O - http://winhelp2002.mvps.org/hosts.txt | grep 127.0.0.1 | sed -e 's/[[:space:]]*#.*$//' | sed -e '2,$s/  / /g' | tr -d '\015\032' > /tmp/hosts.tmp

cat /etc/hosts | sed -e 's/[[:space:]]*#.*$//' | sed -e '2,$s/  / /g' | tr -d '\015\032' >> /tmp/hosts.tmp

awk '!x[$0]++' /tmp/hosts.tmp > /tmp/hosts

cat /tmp/hosts | grep -v 'item_to_remove_from_list' > /tmp/hosts.tmp

rm /tmp/hosts

mv /tmp/hosts.tmp /etc/hosts

killall -1 dnsmasq

The first 2 lines is supposed to be only 1 line, but shows up as 2 because it does not fit on the forum screen.

Create a bash file, for example you can do vi /etc/updatehostfile

With VI, remember to press the I on the keyboard to add content to the file.

Paste the content above, remember that the first 2 lines are supposed to be only 1.

The line cat /tmp/hosts | grep -v 'item_to_remove_from_list' > /tmp/hosts.tmp is for removing items off the new host file, I am using this because the full list is causing a small problem with a service I use, so I am using this command to remove the item off the list. Within the ' ' you can add the server name you want to remove, for example 'something.com', to add more than 1 server do 'something1.com|something2.com' and add as many as you want.

To save in VI, press the ESC key, then wq!

You have to make the file executable, so do a chmod +x /etc/updatehostfile

Now test it ./etc/updatehostfile screen might look like it stops for about 5 seconds as it cleans up the file.

To schedule it, run the command crontab -e

Add the line below, remember that in VI you have to press the I key to start entering text into the file

The command I am using is for running the script on the first day of every month.

00 00 1 * * /etc/updatehostsfile

The discussion might have continued from here.