Zimbra HSM Plus – Secondary Volumes on Amazon S3

zimbra-amazon-s3-banner-enIn previous blog posts, we’ve seen how to download and install Zimbra Suite Plus and a quick overview of each feature. In this blog post, I want to introduce you to this new, cool feature included in Zimbra HSM Plus: secondary volumes on Amazon S3. This includes products that use the Amazon S3 API, but officially we only support Amazon S3 at the moment. Please let us know about your integrations with Zimbra HSM Plus and your storage solutions in the comments.

How Does Zimbra HSM Plus Work?

Zimbra-HSM-Plus-Image-2Zimbra HSM Plus includes several new features that I will cover in future blog posts, but the basic functionality of Zimbra HSM Plus is to move old items to secondary volumes based on a policy or policies. This allows you to save space on the primary storage solution, which can be SAS15K or SSD, and move old items to NFS, SATA, NL-SAS, etc.

This is a quick overview of how Zimbra HSM Plus works with Amazon S3 secondary volumes.

Zimbra HSM Plus Amazon S3

Create an Amazon S3 Bucket

This blog post is not a deep look into the Amazon S3 buckets, security, etc. I want to show you how to quickly create a bucket on Amazon S3.

On the Amazon Web Services Console, click S3.

zimbra-amazon-s3-003

Click Create Bucket.

zimbra-amazon-s3-004Select a name for the bucket and the most desirable region for the project.

zimbra-amazon-s3-005

Once the bucket is created, you need an Access and Secret Key with access to this bucket. Please refer to Amazon Help if you have issues with this.

Activate and Configure Zimbra HSM Plus with Amazon S3

The Zimbra HSM Plus Amazon S3 functionality needs a local cache folder inside Zimbra Collaboration. This folder can be on the primary storage, as you can limit the size on the Admin Console. If you fail to create this folder, Zimbra HSM Plus will create errors on the users when they try to access their old items. Click here for more info about it.
As root user:

mkdir /opt/zimbra/cache
chown zimbra:zimbra /opt/zimbra/cache

Once logged into the Zimbra Admin Console, navigate to Zimbra Suite Plus > HSM > and under Secondary Volumes > Add

zimbra-amazon-s3-001

Select the second option S3 Bucket.

zimbra-amazon-s3-016

Now you can select the name you want for the volume, a prefix if needed, and you will be able to select a S3 profile if you have one already, or if not, you will be able to create a new S3 profile with your credentials, etc.

zimbra-amazon-s3-002

If you didn’t have a S3 profile, this step allows you to create a new one. Simply introduce the bucket name created on Amazon S3 AWS, an Access and a Secret key with write-access to that bucket.

Mind the option Amazon S3 or Custom. As I said, Zimbra Suite Plus should work with other storage vendors who uses the Amazon S3 API, but they are not supported yet.

Select the Region where you’ve created the bucket in the previous steps, and click Test S3 bucket.

zimbra-amazon-s3-006

If you’ve introduced all the correct information, the test will show a message saying the bucket is valid.

zimbra-amazon-s3-007

If you return to the Amazon AWS S3 Console and navigate through the bucket, you will see the test connection files. You can safely remove these files.

zimbra-amazon-s3-012The wizard will not return to the previous window, but in this case you will have already selected the S3 profile. Click Next.

zimbra-amazon-s3-008

Check the Set as current checkbox to start saving files once the HSM policy is configured.

zimbra-amazon-s3-009

The Amazon S3 profiles can be edited, removed or you can add more under the menu Home – Configure – Global Settings – S3 Buckets.

zimbra-amazon-s3-010

Configure the Zimbra HSM Plus Policy

To move the old items to the Amazon S3 secondary volume, you need to create an HSM Policy. In this example, I’ve selected all items, and only the ones older than 30 days will be moved to the HSM secondary volume, in this case an Amazon S3 bucket.

zimbra-amazon-s3-011

At the same time, I want to schedule the task to be automatically executed at 02am every night.

zimbra-amazon-s3-014

For example, I have a demo mailbox with items older than 30 days. Per our previous policy, all the items will be automatically moved to the Amazon S3 bucket. This operation will be invisible to the end-user: it will just take a few more seconds to load old items on the web client once he or she needs them. The items on the primary volume will be accessible faster, as usual.

zimbra-amazon-s3-013

Once the HSM process completes the task of moving all the old items matching the policy to the secondary volume or volumes, you can navigate through the Amazon AWS S3 Console bucket and to see the directory with the regular Zimbra folder structure with the user ID, etc.

zimbra-amazon-s3-015If you navigate inside a userID folder, you will see the regular .msg files, size of the msg, etc., all inside Amazon S3.

zimbra-amazon-s3-017

And that’s it for this Blog post. We now have all the old items protected and secured on Amazon S3, and we have freed up the local storage which should be super fast. Now, with Zimbra HSM Plus, years and years of old items are moved from an expensive storage solution to the Amazon S3 Cloud.

Remember that you can download and try it free for 30-days: Zimbra Suite Plus here.

, , , , , , , , , ,

5 Responses to Zimbra HSM Plus – Secondary Volumes on Amazon S3

  1. Florent October 19, 2016 at 12:58 PM #

    Hi Jorge,

    What is the behavior regarding Zimbra Backups for medium/large deployments ?

    I whitch way it is compatible/supported with Zimbra backup ?

    Best regards,

    • Jorge de la Cruz
      Jorge de la Cruz October 21, 2016 at 10:43 PM #

      Hi Florent,
      The Amazon S3 feature is only for HSM Plus at the moment, we are working internally to allow Backups to Amazon S3 too, but for now, it’s not supported, I wrote more about it long time ago:

      Best regards

    • Florent October 22, 2016 at 10:54 AM #

      Jorge,

      In Zimbra NE and HSM, HSM is included in backups.

      As it is possible to use HSM plus with Zimbra NE, I was asking if object in S3 are included in Zimbra backups and what is the impact for large deployments (as s3 is know to be slow).

      Best regards,

  2. Sergio Bergamini November 21, 2016 at 4:43 PM #

    Hi Jorge, I have a little question. Is possible to deny user to delete some email from his mailboxes? Actualy I need a solution for audit mail. I have a postfix with “always_bcc = audit@”, but this email is growing a lot on my default zimbra volume, I want to store all incoming/outgoing mail from users.

    Regards,

    Sergio Bergamini