Blocking bots with fail2ban

Develop the right mindset for Rails security

Avoid shipping vulnerable code by learning how to prevent security issues in your Rails applications.

Get the course for $99

If you run your own servers, you will surely get a lot of weird hackbot requests, searching for accidentally accessible .env files and such. While these bots are generally harmful, unless you have an issue, I prefer to block them so they don’t spam my logs. If it is a Rails app, I use rack-attack, but I also have some jekyll and other stuff and for those, fail2ban is a great tool to solve this issue. Fail2ban is an intrusion prevention tool, that scans your log files and based on rules, blocks clients on the firewall. It is available on most Linux distributions, and you can install it with your package manager, you can check the installation instructions on the projects documentation. Once installed and configured(you need setup the init service), we can add a “jail”, to block these bots to the /etc/fail2ban/jail.local config file:

[nginx-hack-bots]
enabled  = true
port     = http,https
filter   = hack-bot-paths
logpath  = /var/log/nginx/access.log
bantime  = 86400
findtime = 3600
maxretry = 1

We tell fail2ban to enable this jail on http and https, use the hack-bot-paths filter(we will create this in the next step), watch the nginx access log file, ban offenders for 1 day(I like this because the IP addresses are recycled and might be used by a legitimate client ion the future). findtime is irrelevant, because maxretry is set to 1, so any hit immediately bans the IP, but if you would want a jail to ban only after elevated number of requests, you could set the period for that with findtime. Next step, we need to create the filter in /etc/fail2ban/filters.d/hack-bot-paths.conf:

[Definition]
failregex = ^<HOST> -.*"GET (?:/wp-admin|/wp-login|/old|/wp|/wordpress|/backup|sftp-config\.json|/version|/ga\.js|/geoserver|/solr|/sendrid\.env|/docker-compose\.yml|\.env).* HTTP/.*" \d{3}
ignoreregex =

After this, you just need to restart the fail2ban service and you are done. Additionally, I recommend to check the available jails in /etc/fail2ban/jail.conf and enable the ones youn find relevant for your setup in jail.local.

And if you don’t want to mess around with this, Cloudflare can be used to block these, just make sure your server is not accessible from outside of Cloudflare.