Configuring Fail2Ban on Zimbra

This article is a how-to guide on installing Fail2Ban to block attacking
hosts using a null route or blackhole routes. This can help mitigate
brute force attacks on Zimbra. Especially brute force attacks on SMTP
are very common.

Prerequisite:

It is required the OIP configuration must be done before configuring
Fail2Ban service.

For a Single-Server Setup:
If you are running nginx on the same node as the mailstore, you will
need to add both 127.0.0.1 and the real IP address of that node:

sudo -u zimbra -
zmprov mcf +zimbraMailTrustedIP 127.0.0.1 +zimbraMailTrustedIP {IP of Server}
zmcontrol restart

For a Multi-Server Setup:

sudo -u zimbra -
zmprov mcf +zimbraHttpThrottleSafeIPs {IP of Mailbox-1}
zmprov mcf +zimbraHttpThrottleSafeIPs {IP of Mailbox-2}
zmprov mcf +zimbraMailTrustedIP {IP of Proxy-1}
zmprov mcf +zimbraMailTrustedIP {IP of Proxy-2}
zmcontrol restart

Installation and Configuration of Fail2Ban

1) Install Fail2Ban Package

On RHEL/CentOS 7/8:

yum install epel-release -y
yum install fail2ban -y

On Ubuntu 18/20:

apt-get clean all ; apt-get update
apt-get install fail2ban -y

2) Create a file /etc/fail2ban/jail.local and it will override the
default conf file /etc/fail2ban/jail.conf.
Add the local IP address of the Zimbra server in ignoreip =. You can
also add other IP addresses to ignore from Fail2Ban checking.
On a multi-server setup, add all server’s IP in ignoreip list.

nano /etc/fail2ban/jail.local 

[DEFAULT]
# "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban will not ban a host which matches an address in this list.
# Several addresses can be defined using space (and/or comma) separator.
#ignoreip = 127.0.0.1/8 ::1 10.137.26.29/32
ignoreip = 127.0.0.1/8 "IP-ADDRESS-OF-ZIMBRA-SERVER/32"

banaction = route

3) Create a jail file for Zimbra services.

nano /etc/fail2ban/jail.d/zimbra.local

[zimbra-smtp]
enabled = true
filter = zimbra-smtp
port = 25,465,587
logpath = /var/log/zimbra.log
maxretry = 3
findtime = 86400
bantime = 86400
action = route

[zimbra-webmail]
enabled = true
filter = zimbra-webmail
port = 80,443
logpath = /opt/zimbra/log/mailbox.log
maxretry = 3
findtime = 86400
bantime = 86400
action = route

[zimbra-admin]
enabled = true
filter = zimbra-admin
port = 7071,9071
logpath = /opt/zimbra/log/mailbox.log
maxretry = 3
findtime = 86400
bantime = 86400
action = route
Property Description

ignoreip

This parameter identifies IP address that should be ignored
by the banning system. By default, this is just set to ignore traffic
coming from the machine itself, which is a pretty good setting to have.

banaction

This sets the action that will be used when the threshold is
reached. There is actually the name of a file located in
’`/etc/fail2ban/action.d/’’ which calls the configured action using the
.conf file. Here we configured route which calls route.conf to handle
the routing table manipulation to ban an IP address.

findtime

This parameter sets the window that fail2ban will pay
attention to when looking for repeated failed authentication attempts.
The default is set to 600 seconds (10 minutes again), which means that
the software will count the number of failed attempts in the last 10
minutes.

bantime

This parameter sets the length of a ban, in seconds.

maxretry

This sets the number of failed attempts that will be
tolerated within the findtime window before a ban is instituted.

4) [Optional]
If you want to apply Fail2Ban for SSH then create jail file
sshd.local.
(No need to create filter rules for SSH, Fail2ban by default shipped
with filter rules for SSH)
On Ubuntu systems, SSH jail is by default enabled within the jail file
“/etc/fail2ban/jail.d/defaults-debian.conf”.

nano /etc/fail2ban/jail.d/sshd.local

[sshd]
enabled = true
port = 22
maxretry = 3
findtime = 600
bantime = 3600

5) Create filters for Zimbra services.

nano /etc/fail2ban/filter.d/zimbra-webmail.conf 

[Definition]
failregex = .*oip=<HOST>;.*authentication failed for .*$

ignoreregex =

nano /etc/fail2ban/filter.d/zimbra-smtp.conf

[Definition]
failregex = postfix\/submission\/smtpd\[\d+\]: warning: .*\[<HOST>\]: SASL \w+ authentication failed: authentication failure$
            postfix\/smtps\/smtpd\[\d+\]: warning: .*\[<HOST>\]: SASL \w+ authentication failed: authentication failure$

ignoreregex =

nano /etc/fail2ban/filter.d/zimbra-admin.conf

[Definition]
failregex = .*ip=<HOST>;.*authentication failed for .*$

ignoreregex =

6) Restart the Fail2ban service and enable it to start after system
reboot.

systemctl restart fail2ban
systemctl status fail2ban
systemctl enable fail2ban

7) Check the status of the Fail2Ban jails.

fail2ban-client status

The result should be similar to this:

[root@centos8 ~]# fail2ban-client status
Status
|- Number of jail:      4
`- Jail list:   sshd, zimbra-admin, zimbra-smtp, zimbra-webmail
[root@centos8 ~]#
[root@centos8 ~]# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     14
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 1
   |- Total banned:     2
   `- Banned IP list:   10.137.26.29

8) Check banned IP in routing table.

ip r

route -n

The result should be similar to this:

[root@centos8 ~]# ip r
default via 10.0.10.1 dev ens3
10.0.10.0/24 dev ens3  proto kernel  scope link  src 10.0.10.67
unreachable 10.137.26.29
[root@centos8 ~]#
[root@centos8 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.10.1       0.0.0.0         UG    0      0        0 ens3
10.0.10.0       0.0.0.0         255.255.255.0   U     0      0        0 ens3
10.137.26.29    -               255.255.255.255 !H    0      -        0 -
[root@centos8 ~]#

9) Ban and unban an IP manually.

Ban an IP address.

fail2ban-client set "Jail-Name" banip "IP-Address"

Example:

fail2ban-client set sshd banip 10.137.26.29

Unban an IP address.

fail2ban-client set "Jail-Name" unbanip "Banned IP-Address"

Example:

[root@centos8 ~]# fail2ban-client set sshd unbanip 10.137.26.29

Unban everyone.

Can be useful when something goes wrong with creating new RegEx filter:

fail2ban-client unban --all

Debugging of Fail2Ban:

The loglevel and target are configured in /etc/fail2ban/fail2ban.conf
you can also obtain the log level and log target by running:

fail2ban-client get loglevel
fail2ban-client get logtarget

To watch the log for debugging purpose you can run:

tail -f $(fail2ban-client get logtarget | grep "\`" | awk '{ print $2; }')

Fail2ban works by parsing log files using regular expressions, you can
test the regular expression by using fail2ban-regex like this:

fail2ban-regex /opt/zimbra/log/mailbox.log /etc/fail2ban/filter.d/zimbra-webmail.conf

Multi server and centralized syslog

Fail2ban is designed to work on the local server. So it does it’s ban
actions on the same server where it reads the logs. This can be a
problem if you run Zimbra in a multi server scenario, where you can read
the logs on the mailbox server, but want to apply the ban on the proxy
server.

In addition you may want to use a centralized logging server and if you
decide to ban a bad actor, deny access to all servers in your
environment.

To do this you would need to create a custom fail2ban action. And set up
SSH public key authentication so the server where you run fail2ban can
connect to the server where the ban action needs to be applied. This
article is not meant to cover all possible scenarios, but to get you
started here is a basic example:

Create a new action by copying the default route action:

cp /etc/fail2ban/action.d/route.conf /etc/fail2ban/action.d/remote-route.conf

Next replace the local ip route command with an SSH command to run
remotely, from this:

[Definition]
actionban   = ip route add <blocktype> <ip>
actionunban = ip route del <blocktype> <ip>
actioncheck =
actionstart =
actionstop =

[Init]

# Option:  blocktype
# Note:    Type can be blackhole, unreachable and prohibit. Unreachable and prohibit correspond to the ICMP reject messages.
# Values:  STRING
blocktype = unreachable

To this:

[Definition]
actionban   = ssh root@remote-server -C ip route add <blocktype> <ip>
actionunban = ssh root@remote-server -C ip route del <blocktype> <ip>
actioncheck =
actionstart =
actionstop =

[Init]
blocktype = unreachable

Then configure fail2ban to use the new action, in
/etc/fail2ban/jail.local and /etc/fail2ban/jail.d/zimbra.local
change

banaction = route
...
action = route

to:

banaction = remote-route
...
action = remote-route

Please note that this is meant to be a simple example to get you
started, it is probably best NOT to use the root account on the remote
server. But for testing this is the easiest. Once you have an idea of
how it works, you will probably want to wrap the remote banaction into a
script and use sudo on an account with limited access.

Which would lead to something like:

[Definition]
actionban   = /usr/local/sbin/my-banaction-script <blocktype> <ip>
actionunban = /usr/local/sbin/my-unbanaction-script <blocktype> <ip>
...

Then in /usr/local/sbin/my-banaction-script you could run the
banaction to any number over servers over SSH, something like:

#!/bin/bash

ssh banuser@remote-proxy1 -C sudo ip route del $1 $2 &
ssh banuser@remote-proxy2 -C sudo ip route del $1 $2 &
...etc

, , ,

5 Responses to Configuring Fail2Ban on Zimbra

  1. marco August 29, 2022 at 1:29 PM #

    Thanks so much! Very useful!

  2. David Killingsworth August 31, 2022 at 7:49 AM #

    This is great. Thank you for such a useful and precise write up.

    I have a couple of questions. I see that this protects against brute force SMTP, Webmail, and AdminWeb console attacks.

    Can this block against failed IMAP connections?

    How is this different of more beneficial than the DOS filter settings that are built into Zimbra such as
    zmprov gcf zimbraHttpDosFilterDelayMillis
    zmprov gcf zimbraHttpDosFilterMaxRequestsPerSec
    zmprov gcf zimbraInvalidLoginFilterDelayInMinBetwnReqBeforeReinstating
    zmprov gcf zimbraInvalidLoginFilterMaxFailedLogin
    zmprov gcf zimbraInvalidLoginFilterReinstateIpTaskIntervalInMin
    zmprov gcf zimbraHttpThrottleSafeIPs

    With the DOS filter, we can whitelist our internal sub-nets when end users type their passwords wrong too many times on a Monday morning.

    Can we whitelist internal subnets as well for this?

    Thanks again.

    • Barry de Graaff September 4, 2022 at 11:35 PM #

      Hello David, Yes there is some feature overlap between Zimbra DoS filter and Fail2ban. So depending on the scenario you can choose either one.

  3. Cosmin September 25, 2022 at 7:48 AM #

    Hi

    After running the above, I got an issue with:

    2022-09-25 16:35:21,794 fail2ban.actions [27540]: ERROR Failed to execute ban jail ‘zimbra-smtp’ action ‘route’ info ‘ActionInfo({‘ip’: ‘************’, ‘fid’: <function at 0x7f4e74c830c8>, ‘family’: ‘inet4’, ‘raw-ticket’: <function at 0x7f4e74c83668>})’: Error banning ************
    2022-09-25 16:35:21,794 fail2ban.actions [27540]: NOTICE [zimbra-smtp] Restore Ban ************
    2022-09-25 16:35:21,798 fail2ban.utils [27540]: ERROR 7f4e745089e0 — exec: ip route add unreachable ************
    2022-09-25 16:35:21,798 fail2ban.utils [27540]: ERROR 7f4e745089e0 — stderr: ‘/bin/sh: ip: command not found’
    2022-09-25 16:35:21,798 fail2ban.utils [27540]: ERROR 7f4e745089e0 — returned 127
    2022-09-25 16:35:21,799 fail2ban.utils [27540]: INFO HINT on 127: “Command not found”. Make sure that all commands in ‘ip route add unreachable ************’ are in the PATH of fail2ban-server process (grep -a PATH= /proc/`pidof -x fail2ban-server`/environ). You may want to start “fail2ban-server -f” separately, initiate it with “fail2ban-client reload” in another shell session and observe if additional informative error messages appear in the terminals.
    2022-09-25 16:35:21,799 fail2ban.actions [27540]: ERROR Failed to execute ban jail ‘zimbra-smtp’ action ‘route’ info ‘ActionInfo({‘ip’: ‘************’, ‘fid’: <function at 0x7f4e74c830c8>, ‘family’: ‘inet4’, ‘raw-ticket’: <function at 0x7f4e74c83668>})’: Error banning ************
    [root@mail fail2ban]# ip r
    default via ************ dev eth0 proto static metric 100
    unreachable ************
    unreachable ************
    ************ dev eth0 proto kernel scope link src ************ metric 100
    [root@mail fail2ban]# ip route add unreachable ************
    RTNETLINK answers: File exists
    [root@mail fail2ban]#
    [root@mail fail2ban]# grep -a PATH= /proc/`pidof -x fail2ban-server`/environ
    LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binPYTHONNOUSERSITE=1
    [root@mail fail2ban]# whereis ip
    ip: /usr/sbin/ip /usr/share/man/man8/ip.8.gz
    [root@mail fail2ban]#

    • Barry de Graaff September 25, 2022 at 11:56 PM #

      What Zimbra and OS version are you running?

Leave a Reply

Copyright © 2022 Zimbra, Inc. All rights reserved.

All information contained in this blog is intended for informational purposes only. Synacor, Inc. is not responsible or liable in any manner for the use or misuse of any technical content provided herein. No specific or implied warranty is provided in association with the information or application of the information provided herein, including, but not limited to, use, misuse or distribution of such information by any user. The user assumes any and all risk pertaining to the use or distribution in any form of any subject matter contained in this blog.

Legal Information | Privacy Policy | Do Not Sell My Personal Information | CCPA Disclosures