This article is a how-to guide on installing Fail2Ban to block attacking hosts using a null route or blackhole routes. This can help mitigate brute force attacks on Zimbra. Especially brute force attacks on SMTP are very common.
Prerequisite:
Fail2ban has been tested in combination with netfilter-persistent and iptables. If you use ufw or firewalld you may see errors when trying to ban/unban such as ERROR Failed to execute ban jail … action 'route'. This article has been validated using a set-up installed using https://github.com/Zimbra/zinstaller which you can use to test fail2ban before applying to your production environment.
It is required the OIP configuration must be done before configuring
Fail2Ban service.
For a Single-Server Setup:
If you are running nginx on the same node as the mailstore, you will
need to add both 127.0.0.1 and the real IP address of that node:
sudo -u zimbra -
zmprov mcf +zimbraMailTrustedIP 127.0.0.1 +zimbraMailTrustedIP {IP of Server}
zmcontrol restart
For a Multi-Server Setup:
sudo -u zimbra -
zmprov mcf +zimbraHttpThrottleSafeIPs {IP of Mailbox-1}
zmprov mcf +zimbraHttpThrottleSafeIPs {IP of Mailbox-2}
zmprov mcf +zimbraMailTrustedIP {IP of Proxy-1}
zmprov mcf +zimbraMailTrustedIP {IP of Proxy-2}
zmcontrol restart
Installation and Configuration of Fail2Ban
1) Install Fail2Ban Package
On RHEL/CentOS 7/8:
yum install epel-release -y yum install fail2ban -y
On Ubuntu 18/20:
apt-get clean all ; apt-get update apt-get install fail2ban -y
2) Create a file /etc/fail2ban/jail.local and it will
override the default conf file /etc/fail2ban/jail.conf.
Add the local IP address of the Zimbra server in ignoreip =. You
can also add other IP addresses to ignore from Fail2Ban checking.
On a multi-server setup, add all server’s IP in ignoreip list.
nano /etc/fail2ban/jail.local
[DEFAULT] # "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. # Fail2ban will not ban a host which matches an address in this list. # Several addresses can be defined using space (and/or comma) separator. #ignoreip = 127.0.0.1/8 ::1 10.137.26.29/32 ignoreip = 127.0.0.1/8 IP-ADDRESS-OF-ZIMBRA-SERVER/32 banaction = route
3) Create a jail file for Zimbra services.
nano /etc/fail2ban/jail.d/zimbra.local
[zimbra-smtp] enabled = true filter = zimbra-smtp port = 25,465,587 logpath = /var/log/zimbra.log maxretry = 3 findtime = 86400 bantime = 86400 action = route [zimbra-web] enabled = true filter = zimbra-web port = 80,443,7071,9071 logpath = /opt/zimbra/log/mailbox.log maxretry = 5 findtime = 86400 bantime = 86400 action = route
Update: This article uses a regular expression that should work on most Zimbra deployments. To avoid double banning/unbanning which may lead to unpredictable results and errors this article combines the WebUI and Admin WebUI into a single jail called zimbra-web. This does mean that for most deployments a failed login will be counted double. So maxretry = 5 actually means you can try 3 times before being banned.
| Property | Description |
|---|---|
|
ignoreip |
This parameter identifies IP address that should be ignored by the banning system. By default, this is just set to ignore traffic coming from the machine itself, which is a pretty good setting to have. |
|
banaction |
This sets the action that will be used when the threshold is reached. There is actually the name of a file located in ’`/etc/fail2ban/action.d/’’ which calls the configured action using the .conf file. Here we configured route which calls route.conf to handle the routing table manipulation to ban an IP address. |
|
findtime |
This parameter sets the window that fail2ban will pay attention to when looking for repeated failed authentication attempts. The default is set to 600 seconds (10 minutes again), which means that the software will count the number of failed attempts in the last 10 minutes. |
|
bantime |
This parameter sets the length of a ban, in seconds. |
|
maxretry |
This sets the number of failed attempts that will be tolerated within the findtime window before a ban is instituted. |
4) [Optional]
If you want to apply Fail2Ban for SSH then create jail file
sshd.local.
(No need to create filter rules for SSH, Fail2ban by default shipped
with filter rules for SSH)
On Ubuntu systems, SSH jail is by default enabled within the jail file
“/etc/fail2ban/jail.d/defaults-debian.conf”.
nano /etc/fail2ban/jail.d/sshd.local
[sshd] enabled = true port = 22 maxretry = 3 findtime = 600 bantime = 3600
5) Create filters for Zimbra services.
nano /etc/fail2ban/filter.d/zimbra-web.conf
[Definition] failregex = .*ip=<HOST>;.*authentication failed for .*$ ignoreregex =
nano /etc/fail2ban/filter.d/zimbra-smtp.conf
[Definition]
failregex = postfix\/submission\/smtpd\[\d+\]: warning: .*\[<HOST>\]: SASL \w+ authentication failed: authentication failure$
postfix\/smtps\/smtpd\[\d+\]: warning: .*\[<HOST>\]: SASL \w+ authentication failed: authentication failure$
ignoreregex =
6) Restart the Fail2ban service and enable it to start after system
reboot.
systemctl restart fail2ban systemctl status fail2ban systemctl enable fail2ban
7) Check the status of the Fail2Ban jails.
fail2ban-client status
The result should be similar to this:
[root@centos8 ~]# fail2ban-client status Status |- Number of jail: 3 `- Jail list: sshd, zimbra-smtp, zimbra-web [root@centos8 ~]# [root@centos8 ~]# fail2ban-client status sshd Status for the jail: sshd |- Filter | |- Currently failed: 0 | |- Total failed: 14 | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd `- Actions |- Currently banned: 1 |- Total banned: 2 `- Banned IP list: 10.137.26.29
8) Check banned IP in routing table.
ip r
route -n
The result should be similar to this:
[root@centos8 ~]# ip r default via 10.0.10.1 dev ens3 10.0.10.0/24 dev ens3 proto kernel scope link src 10.0.10.67 unreachable 10.137.26.29 [root@centos8 ~]# [root@centos8 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.0.10.1 0.0.0.0 UG 0 0 0 ens3 10.0.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ens3 10.137.26.29 - 255.255.255.255 !H 0 - 0 - [root@centos8 ~]#
9) Ban and unban an IP manually.
Ban an IP address.
fail2ban-client set "Jail-Name" banip "IP-Address"
Example:
fail2ban-client set sshd banip 10.137.26.29
Unban an IP address.
fail2ban-client set "Jail-Name" unbanip "Banned IP-Address"
Example:
[root@centos8 ~]# fail2ban-client set sshd unbanip 10.137.26.29
Unban everyone.
Can be useful when something goes wrong with creating new RegEx filter:
fail2ban-client unban --all
Debugging of Fail2Ban:
The loglevel and target are configured in
/etc/fail2ban/fail2ban.conf you can also obtain the log level
and log target by running:
fail2ban-client get loglevel fail2ban-client get logtarget
To watch the log for debugging purpose you can run:
tail -f $(fail2ban-client get logtarget | grep "\`" | awk '{ print $2; }')
Fail2ban works by parsing log files using regular expressions, you can test the regular expression by using fail2ban-regex like this:
fail2ban-regex /opt/zimbra/log/mailbox.log /etc/fail2ban/filter.d/zimbra-web.conf
Multi server and centralized syslog
Fail2ban is designed to work on the local server. So it does it’s ban actions on the same server where it reads the logs. This can be a problem if you run Zimbra in a multi server scenario, where you can read the logs on the mailbox server, but want to apply the ban on the proxy server.
In addition you may want to use a centralized logging server and if you decide to ban a bad actor, deny access to all servers in your environment.
To do this you would need to create a custom fail2ban action. And set up SSH public key authentication so the server where you run fail2ban can connect to the server where the ban action needs to be applied. This article is not meant to cover all possible scenarios, but to get you started here is a basic example:
Create a new action by copying the default route action:
cp /etc/fail2ban/action.d/route.conf /etc/fail2ban/action.d/remote-route.conf
Next replace the local ip route command with an SSH command to run remotely, from this:
[Definition] actionban = ip route add <blocktype> <ip> actionunban = ip route del <blocktype> <ip> actioncheck = actionstart = actionstop = [Init] # Option: blocktype # Note: Type can be blackhole, unreachable and prohibit. Unreachable and prohibit correspond to the ICMP reject messages. # Values: STRING blocktype = unreachable
To this:
[Definition] actionban = ssh root@remote-server -C ip route add <blocktype> <ip> actionunban = ssh root@remote-server -C ip route del <blocktype> <ip> actioncheck = actionstart = actionstop = [Init] blocktype = unreachable
Then configure fail2ban to use the new action, in /etc/fail2ban/jail.local and /etc/fail2ban/jail.d/zimbra.local change
banaction = route ... action = route
to:
banaction = remote-route ... action = remote-route
Please note that this is meant to be a simple example to get you started, it is probably best NOT to use the root account on the remote server. But for testing this is the easiest. Once you have an idea of how it works, you will probably want to wrap the remote banaction into a script and use sudo on an account with limited access.
Which would lead to something like:
[Definition] actionban = /usr/local/sbin/my-banaction-script <blocktype> <ip> actionunban = /usr/local/sbin/my-unbanaction-script <blocktype> <ip> ...
Then in /usr/local/sbin/my-banaction-script you could run the banaction to any number over servers over SSH, something like:
#!/bin/bash ssh banuser@remote-proxy1 -C sudo ip route del $1 $2 & ssh banuser@remote-proxy2 -C sudo ip route del $1 $2 & ...etc
Thanks so much! Very useful!
This is great. Thank you for such a useful and precise write up.
I have a couple of questions. I see that this protects against brute force SMTP, Webmail, and AdminWeb console attacks.
Can this block against failed IMAP connections?
How is this different of more beneficial than the DOS filter settings that are built into Zimbra such as
zmprov gcf zimbraHttpDosFilterDelayMillis
zmprov gcf zimbraHttpDosFilterMaxRequestsPerSec
zmprov gcf zimbraInvalidLoginFilterDelayInMinBetwnReqBeforeReinstating
zmprov gcf zimbraInvalidLoginFilterMaxFailedLogin
zmprov gcf zimbraInvalidLoginFilterReinstateIpTaskIntervalInMin
zmprov gcf zimbraHttpThrottleSafeIPs
With the DOS filter, we can whitelist our internal sub-nets when end users type their passwords wrong too many times on a Monday morning.
Can we whitelist internal subnets as well for this?
Thanks again.
Hello David, Yes there is some feature overlap between Zimbra DoS filter and Fail2ban. So depending on the scenario you can choose either one.
Hi
After running the above, I got an issue with:
2022-09-25 16:35:21,794 fail2ban.actions [27540]: ERROR Failed to execute ban jail ‘zimbra-smtp’ action ‘route’ info ‘ActionInfo({‘ip’: ‘************’, ‘fid’: <function at 0x7f4e74c830c8>, ‘family’: ‘inet4’, ‘raw-ticket’: <function at 0x7f4e74c83668>})’: Error banning ************
2022-09-25 16:35:21,794 fail2ban.actions [27540]: NOTICE [zimbra-smtp] Restore Ban ************
2022-09-25 16:35:21,798 fail2ban.utils [27540]: ERROR 7f4e745089e0 — exec: ip route add unreachable ************
2022-09-25 16:35:21,798 fail2ban.utils [27540]: ERROR 7f4e745089e0 — stderr: ‘/bin/sh: ip: command not found’
2022-09-25 16:35:21,798 fail2ban.utils [27540]: ERROR 7f4e745089e0 — returned 127
2022-09-25 16:35:21,799 fail2ban.utils [27540]: INFO HINT on 127: “Command not found”. Make sure that all commands in ‘ip route add unreachable ************’ are in the PATH of fail2ban-server process (grep -a PATH= /proc/`pidof -x fail2ban-server`/environ). You may want to start “fail2ban-server -f” separately, initiate it with “fail2ban-client reload” in another shell session and observe if additional informative error messages appear in the terminals.
2022-09-25 16:35:21,799 fail2ban.actions [27540]: ERROR Failed to execute ban jail ‘zimbra-smtp’ action ‘route’ info ‘ActionInfo({‘ip’: ‘************’, ‘fid’: <function at 0x7f4e74c830c8>, ‘family’: ‘inet4’, ‘raw-ticket’: <function at 0x7f4e74c83668>})’: Error banning ************
[root@mail fail2ban]# ip r
default via ************ dev eth0 proto static metric 100
unreachable ************
unreachable ************
************ dev eth0 proto kernel scope link src ************ metric 100
[root@mail fail2ban]# ip route add unreachable ************
RTNETLINK answers: File exists
[root@mail fail2ban]#
[root@mail fail2ban]# grep -a PATH= /proc/`pidof -x fail2ban-server`/environ
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binPYTHONNOUSERSITE=1
[root@mail fail2ban]# whereis ip
ip: /usr/sbin/ip /usr/share/man/man8/ip.8.gz
[root@mail fail2ban]#
What Zimbra and OS version are you running?
can I use it on zimbra 7.2.7, Centos 5.8 (Final)?
The version of your operating system and Zimbra is long out of support, none of the steps provided in the blog have been tested on your version.
Hi Barry,
I’ve setted up this conf on a new Z9 last version ubuntu 20,
I encounter a problem where F2B can ban but not unban :
NOTICE [zimbra-admin] Unban 221.215.87.163
ERROR 7fb9782fb750 — exec: ip route del unreachable 221.215.87.163
ERROR 7fb9782fb750 — stderr: ‘RTNETLINK answers: No such process’
ERROR 7fb9782fb750 — returned 2
ERROR Failed to execute unban jail ‘zimbra-admin’ action ‘route’ info ‘ActionInfo({‘ip’: ‘221.215.87.163’, ‘family’: ‘inet4’, ‘fid’: <function Actions.ActionInfo. at 0x7fb9797a5c10>, ‘raw-ticket’: <function Actions.ActionInfo. at 0x7fb9797a6310>})’: Error unbanning 221.215.87.163
Doing the unban manually has root user is working
Did you encounter a problem like this ?
Regards
I noticed I actually see a similar message when banning but not when unbanning, but both banning and unbanning works for me. A likely (though I can’t be sure) cause of this is a difference in the firewall used in your set-up.
What firewall are you using? Perhaps try with iptables-persistent install via apt install netfilter-persistent iptables
systemctl stop iptablessystemctl disable iptables
systemctl mask iptables
systemctl stop ufw
systemctl disable ufw
systemctl mask ufw
systemctl stop firewalld
systemctl disable firewalld
systemctl mask firewalld
Then define firewall rules as in https://github.com/Zimbra/zinstaller/blob/main/zinstaller#L229 (basically by test environment is set-up by using zinstaller)
Try all this on a test environment first, please let me know if this fixes your issue.
I have added an update to the article as to avoid the issue. Basically merged the zimbra-admin and the zimbra-web into a single jail.