OpenWrt Forum Archive

Topic: Fail2ban replacement and RBL firewall sync'ing - in lightweight ash

The content of this topic has been archived between 5 Apr 2018 and 29 Apr 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

I have two scripts that I've been working on ready for some feedback if anyone is interested. 

The first is called bearDropper, and is essentially a minimal, lightweight dropbear (ssh) log examiner that blocks ssh brute force attacks.  It hooks itself into iptables, is very configurable using uci and/or command line options.  The goal was to make it simple to use but very reliable and configurable - let me know if you think I succeeded.  It started as a rewrite of dropBrute, but has many other features:

  • Written in busybox ash with no dependencies outside of stock Chaos Calmer

  • Maintains a state database, periodically writes it to tmpfs and (optionally) persistent storage

  • Persistent storage writes are throttled (default once a day) and are disabled by default

  • Periodically syncs the state database to a dedicated iptables chain

  • Periodically expires entries from the iptables chain

  • Self installs into iptables (default hook is into input_wan_rule) - easily disabled or modified in config

  • Uses native uci config, and runs via a procd init script (continuous run)

  • For those who like options, it can also run in a few single-run (non-continuous) modes, examining the entire syslog ring buffer, just today's entries (like dropBrute), or even arbitrary intervals.

The second is called sub2rbl, and it's a very simple, lightweight script to retrieve RBLs, compile them into an ipset and automatically hook into iptables with a firewall rule (default in input_wan_rule).  It also uses uci (/etc/config/sub2rbl), with command line overrides.  Dependencies are ipset, curl and openssl-util (to retrieve RBLs via https).

  • Supports IP based and CIDR (net) based RBLs

  • Default config uses OpenBL, blocklist.de, Dragon Research and SpamHaus DROP/EDROP RBLs

  • Whitelist support based on a uci config list (see config file for details)

  • Simple installation, see the github project page for instructions

I'm looking forward to hearing some feedback (and bug reports) smile

Rob

(Last edited by robzr on 19 Jan 2016, 00:13)

Thanks for sharing your work! This is really helpful for OpenWrt community since fail2ban relies on Python. I'll definitely try it when I have time.

Besides, I'm running a proxy server on my router and occasionally there are failed attempts from unknown IP addresses shown in the log. It looks like:

Sat Jan 16 20:22:50 2016 kern.err /usr/bin/ss-server[875]: failed to handshake with 185.35.62.11

I guess I can customize your scripts a little bit to ban the brute force attackers too?

Thanks! Will test sub2rbl.

wyf88 wrote:

Thanks for sharing your work! This is really helpful for OpenWrt community since fail2ban relies on Python. I'll definitely try it when I have time.

Besides, I'm running a proxy server on my router and occasionally there are failed attempts from unknown IP addresses shown in the log. It looks like:

Sat Jan 16 20:22:50 2016 kern.err /usr/bin/ss-server[875]: failed to handshake with 185.35.62.11

I guess I can customize your scripts a little bit to ban the brute force attackers too?

NP, I hope it is useful, my last attempt at this (dropBrute) I wasn't very happy with, so I'm hoping this one is easier to maintain and more flexible.  And on that note, I made a couple small changes to the script that make it more flexible in regards to log filtering, and I tried it with the logline you posted.  Download the new version in github and add this to your /etc/config/bearDropper

        option  regexLogString          '^[a-zA-Z ]* [0-9: ]* (authpriv.warn dropbear|kern.err /usr/bin/ss-server)\['

It works with my testing but I had to use a dummy input file, so let me know how it works for you.  If ss-server posts a lot to kern.err it may catch too much junk, which should just result in "bearDropper... malformed line (...)" entries in logger, if so let me know and I can figure out a better regex.  I was also toying with the idea of making the regexLogString a list so you could specify as many different regex's as you like, but it would make building the sed statement a little tricky.  Might be worth doing though...

bouwew wrote:

Thanks! Will test sub2rbl.

Excellent!  You'll be my first guinea pig smile  Please let me know how it works for you.

Rob

robzr wrote:

Excellent!  You'll be my first guinea pig smile  Please let me know how it works for you.

Rob

I installed the tools as described on github.

After that, I did:

/etc/init.d/cron reload

to make the changes to the root-file active, and

/usr/sbin/sub2rbl

to execute the tool for the first time.
No message after this, so I assume all is OK.

Do you want me to test anything specific?

(Last edited by bouwew on 17 Jan 2016, 12:33)

bouwew wrote:

...

/usr/sbin/sub2rbl

to execute the tool for the first time.
No message after this, so I assume all is OK.

Do you want me to test anything specific?

So far so good - to verify it is running and loading RBLs, run "logread -e sub2rbl"

root@gw:~# logread -e sub2rbl
Sun Jan 17 00:00:00 2016 cron.info crond[871]: USER root pid 10154 cmd /usr/sbin/sub2rbl
Sun Jan 17 00:00:03 2016 authpriv.notice sub2rbl[10155]: RBL (https://www.openbl.org/lists/base_30days.txt) added 1936 entries
Sun Jan 17 00:00:06 2016 authpriv.notice sub2rbl[10155]: RBL (https://lists.blocklist.de/lists/ssh.txt) added 929 entries
Sun Jan 17 00:00:07 2016 authpriv.notice sub2rbl[10155]: RBL (https://lists.blocklist.de/lists/strongips.txt) added 78 entries
Sun Jan 17 00:00:09 2016 authpriv.notice sub2rbl[10155]: RBL (https://www.dragonresearchgroup.org/insight/sshpwauth.txt) added 320 entries

You can verify the total number of IPs that have been slurped into the ipset from the RBLs:

root@gw:~# ipset list sub2rbl | wc -l
3341

Finally, to see how many connection attempts have been stopped, look in the packets column in the iptables rule.  These numbers won't get reset each time sub2rbl is run - but it will get reset if you reboot or reload your firewall.

root@gw:~# iptables -nvL input_wan_rule
Chain input_wan_rule (1 references)
 pkts bytes target     prot opt in     out     source               destination               
  449 25969 DROP     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set sub2rbl src

Rob

What is rbl? what is it for?

tapper wrote:

What is rbl? what is it for?

Real-time Black-hole List - came out of the spam prevention world, the idea being that when end-nodes receive spam (or ssh brute force attempts in this case), they upload the data on where those attacks are coming from to a central server, which then analyzes it, looks for patterns, and then publishes a list of attacker IPs.  End-nodes can then download these lists (RBLs) and actively block them before they've even been attacked by them.  There is of course a possiblity of false-positive, and once the attackers get effectively blocked they will move onto other IPs, therefore the lists must constantly be updated (that's the Real-time in RBL), as well as expiring the old entries.  sub2rbl does all that smile  I wrote it with the intention of blocking ssh brute force attacks (since that is what OpenWRT is usually most vulnerable to, as ssh is often the only service hosted on a router), but you can use it to subscribe to any RBL you like, and the default firewall rules block all services, so effectively it can also be used to prevent web or email (or any sort of) attacks as well.  The default RBLs in the config file are focused primarily on ssh brute force attacks though.

Rob

Hi Rob,

Thanks for the extra info!
It's running fine in my case.
I see no more attempt in my log trying to get into my VPN smile
And 39 blocked attempts since yesterday's reboot.

I did a bit of work on sub2rbl - it now includes support for net based RBLs, and the config file includes a couple - the SpamHaus DROP & EDROP RBLs, which are composed of IP ranges hijacked/used by spammers and cyber-criminals.  To keep it memory efficient, sub2rbl will now add a hash:net based ipset if the net-based RBLs are used in addition to the standard hash:ip set.  They could all go into a single hash:net set, but they use a bit more memory than the hash:ip sets, so I think it was worth a little extra logic to implement this way.

Rob

Hi Rob. Quick question;  does the stock bearDropper config file contain the settings that are turned on by default, or do I need to remove the comments in order for it to work?  Nice work by the way.

Thanks,

Mike

mmcneil wrote:

Hi Rob. Quick question;  does the stock bearDropper config file contain the settings that are turned on by default, or do I need to remove the comments in order for it to work?  Nice work by the way.

Thanks Mike.  The config file entries that are commented out are the same as the hardcoded defaults.  That's kind of confusing, I should just uncomment them since the end result is the same, and that'd be simpler.  BTW at any time you can run bearDropper -h and look at the (def:...) values, they'll show you what it would run as based on the current config and command line even (just put -h at the end).

Rob

Updated the bearDropper config - it was a little out of sync, now it's hopefully a little more intuitive.  Also fixed a bug in the expiration logic - I'd recommend anyone using it upgrades.

Rob

robzr wrote:
mmcneil wrote:

Hi Rob. Quick question;  does the stock bearDropper config file contain the settings that are turned on by default, or do I need to remove the comments in order for it to work?  Nice work by the way.

Thanks Mike.  The config file entries that are commented out are the same as the hardcoded defaults.  That's kind of confusing, I should just uncomment them since the end result is the same, and that'd be simpler.  BTW at any time you can run bearDropper -h and look at the (def:...) values, they'll show you what it would run as based on the current config and command line even (just put -h at the end).

Rob

Thanks Rob.  Keep up the good work.

Mike

This is the cat's meow!

Hi Rob.  Sorry to bug you but I'm getting the following error when trying to use sub2rbl:

Creating ipset sub2rbl_swing
Retrieving RBL (https://www.openbl.org/lists/base_30days.txt)
Error: retrieving RBL(https://www.openbl.org/lists/base_30days.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://lists.blocklist.de/lists/ssh.txt)
Error: retrieving RBL(https://lists.blocklist.de/lists/ssh.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://lists.blocklist.de/lists/strongips.txt)
Error: retrieving RBL(https://lists.blocklist.de/lists/strongips.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://www.dragonresearchgroup.org/ins … pwauth.txt)
Error: retrieving RBL(https://www.dragonresearchgroup.org/insight/sshpwauth.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Creating ipset sub2rbl-net_swing
Retrieving RBL (https://www.spamhaus.org/drop/drop.txt)
Error: retrieving RBL(https://www.spamhaus.org/drop/drop.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://www.spamhaus.org/drop/edrop.txt)
Error: retrieving RBL(https://www.spamhaus.org/drop/edrop.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Swinging set sub2rbl
Destroying ipset sub2rbl_swing
Swinging set sub2rbl-net
Destroying ipset sub2rbl-net_swing

Any idea as to what I'm doing wrong?

Thanks,

Mike

mmcneil wrote:

Retrieving RBL (https://www.openbl.org/lists/base_30days.txt)
Error: retrieving RBL(https://www.openbl.org/lists/base_30days.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED

Hi Mike - as anomeome mentioned, you need the root certificates, 'opkg install ca-certificates' should do it, let me know if that doesn't fix it.

Rob

robzr wrote:
mmcneil wrote:

Retrieving RBL (https://www.openbl.org/lists/base_30days.txt)
Error: retrieving RBL(https://www.openbl.org/lists/base_30days.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED

Hi Mike - as anomeome mentioned, you need the root certificates, 'opkg install ca-certificates' should do it, let me know if that doesn't fix it.

Rob

I'm using my own build from trunk and I have ca-certs built in already:

lnxs-rtr:~# opkg install ca-certificates                 
Package ca-certificates (20151214) installed in root is up to
date.

curl -o /etc/ssl/certs/ca-certificates.crt http://curl.haxx.se/ca/cacert.pem
Solved the issue for me. Where does the opkg install / or build place the cert file | contents?

mmcneil wrote:

lnxs-rtr:~# opkg install ca-certificates                 
Package ca-certificates (20151214) installed in root is up to
date.

Two questions - What's the output of "ls /etc/ssl/certs/????????.? | wc -l"  (checking the ca-certificate hash links - it's 180 on my ca-certificates 20150426 installation.

Does "curl --capath=/etc/ssl/certs https://lists.blocklist.de/lists/ssh.txt" work?

Rob

robzr wrote:
mmcneil wrote:

lnxs-rtr:~# opkg install ca-certificates                 
Package ca-certificates (20151214) installed in root is up to
date.

Two questions - What's the output of "ls /etc/ssl/certs/????????.? | wc -l"  (checking the ca-certificate hash links - it's 180 on my ca-certificates 20150426 installation.

Does "curl --capath=/etc/ssl/certs https://lists.blocklist.de/lists/ssh.txt" work?

Rob

@anomeome - certs are installed in /etc/ssl/certs

@rob:

1. ls /etc/ssl/certs/????????.? | wc -l
173


2. curl --capath /etc/ssl/certs https://lists.blocklist.de/lists/ssh.txt
curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED

I ran 'curl -o /etc/ssl/certs/ca-certificates.crt http://curl.haxx.se/ca/cacert.pem' but I'm still getting errors.

Thanks,

Mike

I removed the *.pem file from the above curl, got the ca-cert*.ipk from the last successful trunk build (20151214), yields 173 certs as per the above wc command, and bounced sub2rbl successfully. So either way be good as far as the certs are concerned.

Does curl -V indicate your compile supports SSL?

Edit:
I doubt this ^^^^ but can you check? You do not need a CA.

(Last edited by anomeome on 21 Jan 2016, 23:46)

anomeome wrote:

I removed the *.pem file from the above curl, got the ca-cert*.ipk from the last successful trunk build (20151214), yields 173 certs as per the above wc command, and bounced sub2rbl successfully. So either way be good as far as the certs are concerned.

Does curl -V indicate your compile supports SSL?


1. root@lnxs-rtr:~# curl -o /etc/ssl/certs/ca-certificates.crt http://curl.haxx.se/ca/cacert
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   269  100   269    0     0    672      0 --:--:-- --:--:-- --:--:--   675
root@lnxs-rtr:~#


2. root@lnxs-rtr:~# ls /etc/ssl/certs/????????.? | wc -l
173


3. root@lnxs-rtr:~# opkg install ca-certificates
Package ca-certificates (20151214) installed in root is up to date.


4. root@lnxs-rtr:~# sub2rbl -l 3 -f stdout
Creating ipset sub2rbl_swing
Retrieving RBL (https://www.openbl.org/lists/base_30days.txt)
Error: retrieving RBL(https://www.openbl.org/lists/base_30days.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://lists.blocklist.de/lists/ssh.txt)
Error: retrieving RBL(https://lists.blocklist.de/lists/ssh.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://lists.blocklist.de/lists/strongips.txt)
Error: retrieving RBL(https://lists.blocklist.de/lists/strongips.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://www.dragonresearchgroup.org/ins … pwauth.txt)
Error: retrieving RBL(https://www.dragonresearchgroup.org/insight/sshpwauth.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Creating ipset sub2rbl-net_swing
Retrieving RBL (https://www.spamhaus.org/drop/drop.txt)
Error: retrieving RBL(https://www.spamhaus.org/drop/drop.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Retrieving RBL (https://www.spamhaus.org/drop/edrop.txt)
Error: retrieving RBL(https://www.spamhaus.org/drop/edrop.txt): curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
Swinging set sub2rbl
Destroying ipset sub2rbl_swing
Swinging set sub2rbl-net
Destroying ipset sub2rbl-net_swing

5. root@lnxs-rtr:~# curl -V
curl 7.43.0 (arm-openwrt-linux-gnu) libcurl/7.43.0 mbedTLS/1.3.15
Protocols: file ftp ftps http https
Features: IPv6 Largefile SSL



Do I need to setup a CA on the openwrt router?


Thanks,

Mike

(Last edited by mmcneil on 21 Jan 2016, 23:51)

I haven't experienced this, but it may be a known issue with newer curl builds on OpenWRT: https://dev.openwrt.org/ticket/19621

From what it looks like to me, curl (or rather the mbedTLS library, formerly known as PolarSSL) is not recognizing the root cert bundle and/or directory.  The ca-certificates package loads a directory with root certs and then generates hash links to the cert files.  The other method that people seem to use is to install a single cert bundle (one file with all the certs concatenated), which is what the http://curl.haxx.se/ca/cacert.pem file is.  The compiled in options in curl/mbedTLS should read at least the directory (maybe the file too).  Try this: "curl --cafile /etc/ssl/certs/ca-certificates.crt https://lists.blocklist.de/lists/ssh.txt"

If that still fails, try uninstalling the ca-certificates bundle; make sure that the only file in /etc/ssl/certs is ca-certificates.crt (the bundle), and then try it again.  Depending on the precedence that mbedTLS loads the cert files/bundles with, you might have a conflict there.

As a final band-aid you can also try "curl --insecure https://lists.blocklist.de/lists/ssh.txt" and see if that lets you sidestep the cert issue until this gets fixed.

Rob