Today’s messaging and collaboration systems must be highly reliable: end users need access to email and calendaring 24 hours a day, 7 days a week. Designing a reliable messaging and collaboration environment entails choosing the proper storage, server and operating system. Many of us employ storage area networks (SANs) to provide that storage reliability, but SANs require specialized expertise to manage.

Today, VMware Zimbra is releasing a new whitepaper detailing our testing of Zimbra with support for NetApp Network File-based Storage (NFS) and VMware vSphere, a significant competitive advantage over other email and collaboration systems. Zimbra 7 takes a new, innovative approach towards messaging and collaboration.

File-based storage gives customers additional flexibility, scalability and cost savings when designing and deploying a messaging system. Listed below are just some of the benefits of using file-based storage for your enterprise email deployments:

Flexibility – At least 80% of enterprise customers are running a heterogeneous storage infrastructure that includes NFS as part of that storage infrastructure. Customers want the choice of Network Attached Storage. The VMware Zimbra Collaboration Server supports all of the available options for storage including NFS, SAS/SATA, Fibre Channel, and iSCSI. Other messaging and collaboration products such as Microsoft Exchange, Notes and GroupWise don’t support or have withdrawn their support for file-based storage or NFS. However, when deployed on vSphere, ZCS also supports Network Attached Storage (NAS) for all of its components. We are the only messaging product that supports the use of NAS.   See the new features in Zimbra 7.

Scalability – With file-based storage you can dynamically grow or shrink vSphere datastores up to 16TB. Deploying a virtualized Zimbra Collaboration Server means that you can just create an additional virtual disk file, add it to the virtual mailbox server and grow the logical volume and you have increased the mailbox store. For service providers and hosting services moving their services into the cloud, file-based storage provides significant operational, cost and management reduction. See how Zimbra 7 integrates with cloud computing.

Cost Savings – In all scenarios file-based storage represents cost savings across the board in management, upfront acquisition costs and maintenance. Forrester conducted a study of 7 large customers using or considering the use of file-based storage and found that the cost savings over three years was greater 50 percent. In medium to large organizations, fibre channel directors, edge switches, host bus adapters (HBAs) and cabling can cost in the hundreds of thousands of dollars not to mention the cost of maintaining these infrastructures. File-based storage uses the existing networking infrastructure that organizations are most familiar with. No need to deploy a separate storage infrastructure to support the organizations applications.

The VMware Zimbra team partnered with NetApp to create an enterprise test environment consisting of vSphere, Zimbra and NetApp storage that easily scales to 100,000 users. The test results are very encouraging: NAS performed on par with Fibre Channel. View the NetApp and VMware vSphere Best Practices TR-3749 at http://media.netapp.com/documents/tr-3749.pdf

The combination of VMware vSphere and NetApp file-based storage means:

• Full support for virtualization and all of vSphere’s advanced features (vMotion, Storage vMotion, DRS)
• Simplified high availability through the use of VMware HA
• Server consolidation (reduced datacenter footprint, reduced power and cooling and a dynamically balanced workload )

Zimbra Collaboration Server is designed to scale and perform reliably in enterprise as well as off-premise deployments. When deployed using the VMware vSphere platform one can design a highly reliable and scalable messaging and collaboration environment that can take advantage of vSphere’s benefits of DRS, vMotion, storage vMotion and the ability to simplify management and availability. For more information on VMware vSphere see http://www.vmware.com/solutions/datacenter/

Get more information by registering for the Webinar on April 27th at 10:00 am PST to have NetApp, VMware and Zimbra experts further explain the details and answer any of your questions. Click here to register.

You may also download the white paper now.

Zimlets and the ability to extend the Zimbra Web Client is a pretty widely known capability. But did you know that Zimbra also has a framework that allows developers to extend Zimbra server-side functionality?

Zimbra Server Extensions provide a mechanism to add functionality to the server in lieu of modifying web.xml and other web server configuration files. By implementing a Server Extension, you can inject or in some cases, intercept, server-side functionality. Some examples include:

• Handling authentication requests against a user store different than the built-in Zimbra LDAP user store. Server Extensions provide a way to “plug-in” your custom authentication mechanism.
• Creating custom SOAP requests to augment the current Zimbra SOAP API.
• Registering custom mime handlers for the processing of message mime components (i.e. body, message and multipart nodes).
• Performing registration or “bootstrapping” of server-side components, such as the Microsoft Exchange Free/Busy resolver.

Getting Started

Writing an extension starts by implementing the com.zimbra.cs.extension.ZimbraExtension interface:

public interface ZimbraExtension {

    /**
     * Defines a name for the extension. It must be an identifier.
     * @return the extension name
     */
    public String getName();

    /**
     * Initializes the extension. Called when the extension is loaded.
     *
     * @throws ServiceException
     */
    public void init() throws ServiceException;

    /**
     * Terminates the extension. Called when the server is shut down.
     *
     */
    public void destroy();
}

To create your Extension:

1. Write a MyZimbraExtension class that implements the ZimbraExtension interface.
2. Create a JAR file (for example: myext.jar) and include the following attribute in the JAR manifest file:
   • Zimbra-Extension-Class: com.example.MyZimbraExtension
3. Place the myext.jar into the {zimbra_install-dir}/lib/ext/{my-ext-dir} directory.

Below are a few examples, though not an exhaustive list, of how the Server Extension framework can be used:

Custom HTTP Handlers

A Server Extension can process HTTP GET/POST/OPTIONS requests by extending the com.zimbra.cs.extension.ExtensionHttpHandler abstract class and registering the subclass in the Extension init() method.

public class MyHttpHandler implements ExtensionHttpHandler {

    // the path under which MyHandler is registered
    public String getPath() {
        return "/myext/myhandler";
    }

    // override doGet/doPost/ doOptions as needed
    ...
}

and in MyZimbraExtension.init(), do the following:

com.zimbra.cs.extension.ExtensionDispatcherServlet.register(this, new MyHttpHandler());

This would result in processing of requests at http://{my-zimbra-server-url}/service/extension/myext/myhandler being delegated to the MyHttpHandler class.

Custom SOAP Requests

If an Extension needs to support extra SOAP requests, it can do so by implementing the com.zimbra.soap.DocumentService interface and registering custom SOAP requests & operations handlers from it.

public class MyDocumentService implements DocumentService {

    /**
     * Registers <code>DocumentHandler<code> instance with document dispatcher.
     */
    public void registerHandlers(DocumentDispatcher dispatcher) {
        dispatcher.registerHandler(HelloWorldOp.REQUEST_QNAME, new HelloWorldOp());
    }
}

and in MyZimbraExtension.init(), do the following:

com.zimbra.soap.SoapServlet.addService("SoapServlet", new MyDocumentService());

Custom Authentication

If your Zimbra deployment needs to authenticate user passwords from, for example, a “home grown” authentication system and not from LDAP (which is the out-of-the-box Zimbra solution), the Extension can do so by extending the com.zimbra.cs.account.auth.ZimbraCustomAuth abstract class.  The subclass would have to implement an authenticate() method to which the plain-text password is passed as one of the arguments.

Learn More

A detailed implementation document is available at ZimbraServer/docs/extensions.txt. Now that we’ve introduced the topic, stay tuned for more information on specific Server Extension implementation examples. 

For years, the Zimbra Collaboration Suite (ZCS) has exposed mailbox data via REST. And lately, a few customers have asked about the Zimbra REST API (e.g. How is the API used? What methods are available? How does the Zimbra REST API compare to the Zimbra SOAP API?). So I thought it would be good to revisit the topic.

Some background: REST (REpresentational State Transfer) is an approach for building application services that make application resources available via a URL. There is more than ample information about REST out there on the web so I’ll just point you to the REST Wikipedia article for simplicity.

The Zimbra REST API is a great mechanism for reading user mailbox data and outputting that data in different formats (everything from XML-and-JSON to RSS-and-Atom to iCal-and-VCF). In addition to reading data, we’ve exposed methods to import items — like contacts and appointments — into a mailbox. This capability is very useful when performing a migration (i.e. “how do I import contacts from one system into Zimbra?”)

Here is an example of using the REST API to read mailbox data: if you want an output of messages from the “john doe” inbox folder, you call the Zimbra REST API like this:

http://localhost:7070/home/john.doe/inbox?fmt=xml

You can try this method against a Zimbra server via a browser but more commonly customers use the REST API methods from perl scripts or PHP applications…or even just from the command line using curl.

Here is an example of importing contacts into the “john doe” account: perform a POST of a CSV file to the “contacts” folder. The following command shows importing contacts using curl:

curl -{username}:{password} --upload-file /tmp/mycontacts.csv http://localhost:7070/home/john.doe/contacts?fmt=csv

That’s just a couple examples and as you can see, the REST API is a simple yet powerful mechanism for accessing mailbox data. For more information on the Zimbra REST API (i.e. information on authentication, output formats or just a list & syntax for the REST API methods), here is a link to Zimbra REST API Reference document:

http://wiki.zimbra.com/index.php?title=ZCS_6.0:Zimbra_REST_API_Reference

One more thing: A very common question is: “why does the SOAP API support so many more functions than the REST API?” Well, the Zimbra REST API is different (in form and function) than the Zimbra SOAP API. At a high level, it’s really an apples-to-oranges comparison. The REST API is for accessing user mailbox data, and the SOAP API is for executing functionality on the server (which can access user mailbox data and also perform account and administrative functions).

The Zimbra SOAP API is the foundational platform service that the Zimbra Web Client uses to communicate between the browser and the Zimbra server. As you can imagine, to create the rich interface experience the Zimbra Web Client delivers, the SOAP API is very complete with advanced and complex functionality, much more than the REST API. To re-create all of this complex functionality in REST API would be a challenge as well as take significant time. So as the alternative, we expose the SOAP API for direct use by our customers & partners.

But that’s another story…I’ll save the Zimbra SOAP API discussion for a different day…until then, enjoy the REST API and happy coding!

It may sound odd offering more Zimbra installation advice since there is a lot on the subject in other blogs, our documents, wiki and Forums. In fact, some quick research surfaced over 1.4 million hits for Zimbra server install on the web and 36,000 on the Zimbra site alone.

But we are also fortunate to have more new Zimbra users than ever, and after helping some trial customers recently, it was a good reminder a few simple tips can help cut through some noise and avoid time-consuming snags once you start the install process. So without further ado here are the top 6 common pre-requisites to consider when preparing for your Zimbra installation:

1. Firewall
Servers have firewalls configured once the operating systems are installed for security purposes. Our recommendation is to temporarily disable the firewall on the system during a single and multi-server Zimbra installation. An alternative would be to refer to our installation guide to get a list of ports (see Table 1) used by the application and make sure the ports are open prior to installation.

2. DNS setup
All Zimbra configurations store hostnames. We do not have save any IP address information in our configuration. The advantage is this allows an administrator to change IP address (more likely) on the Zimbra system without having to perform any application changes.

This scenario means that all the hostnames to be used in a Zimbra installation have to be defined in DNS. Both A and Mx records for the hostnames and email domains need to be defined and verified prior to beginning your installation.

One other thing to consider is split DNS configuration if you are dealing with servers separated by a firewall.

3. Use of Fully Qualified Hostnames (FQDN)
It is crucial to use a Fully Qualified hostname during the Zimbra configuration. For example, you should enter server1.domain.com instead of server1. This avoids incorrect DNS address lookups and ensures that the client would be connecting to the right application.

4. Port Conflicts
Standard server configuration comes with support for numerous services like POP, IMAP and HTTP (see Table 1). These services are also installed with the Zimbra Network Edition. Therefore, you want to make sure you disable all these services prior to installation. The Zimbra installation scripts will check for any of these port conflicts and notify you to turn these services off before continuing.

5. Libraries and additional packages
Zimbra’s rich feature sets are dependent on additional packages being installed on the system. These packages vary between Linux and Mac Operating system. The Zimbra installation script does perform checks to verify all the dependencies have been met, but going through the System Requirements documentation (available on the Zimbra website) before will save you some time.

STORAGE CALCULATION EXAMPLE
(Based on ‘Mailbox Usage of 200 MB’ and 500 users)

+ User Data: 500 users with 200 MB = 100 GB user data
+ MySQL data: 5% of 100 GB (User Data): 5 GB
+ Zimbra binaries: 10 GB
+ Zimbra logs: 20 GB
+ Zimbra indexes: 25% of 100GB (User Data) = 25 GB

SUBTOTAL: 100 + 5 + 10 + 20 + 25 = 160 GB
Backups: 160 % of Subtotal: 160 * 160% = 256 GB for backups
TOTAL: 160 + 256 = 416 GB

6. Sizing
Storage sizing is important for an excellent performing Zimbra application (see example). If you are doing a Network Edition trial you should contact the Zimbra technical team for sizing information for storage including number of disks, which Raid level to use, and the size of the drives to use. Configuration of the Zimbra store volume is important in satisfying the application IO requirements.

Remember, it’s also a good idea to review the Zimbra Quick Installation Guide where you can find this information and many more good tips.

Do you have a good tip to share? Feel free to add a comment!

Someone in the forums recently asked about ways to migrate individual accounts from one ZCS instance to another, so thought I’d share the enlightenment with all. Whether you are going from an on-premise install to a hosting provider, want to create handy archives of old employee accounts, or just need to duplicate mailbox contents of a user; the syntax in this article proves remarkably useful, and applies to all editions.

There are a multitude of comparable RFE’s on addressing this need via different approaches. (Bugzilla entries 19630, 29573, 28443 & 30163 to name a few.) Some want graphical tools to browse data and selectively migrate certain things, while others would be happy with a cross LDAP zmmailboxmove.

Depending on your situation, several backup tools can take care of a large portion of your daily needs; and there are ways to do Zimbra-to-Zimbra migrations using the Network Edition’s backup and restore capabilities – however they require admin abilities on both systems. Meanwhile, most of the frequently used open source backup solutions are simply an “all accounts at-once” approach. So what to do when you need to move from your personal setup to a hosting provider? Or if you’re a hosting provider, move a tiny handful of accounts to a separate infrastructure? Before diving into the wiki on user migration for info on Imapsync, REST exports, CURL imports, etc; there’s a handy way to avoid the “one item type at-a-time” transfer methods.

In ZCS 5.0.9+ you can export an entire mailbox with:
/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz

Next transfer via rsync, scp, sftp, etc. You’ll also need to create the account on the 2nd server if the desired account doesn’t exist at your destination server yet.

Then import with:
/opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL “//?fmt=tgz&resolve=reset” /tmp/account.tgz

The resolve= paramater has several options:

• “skip” ignores duplicates of old items, it’s also the default conflict-resolution.
• “modify” changes old items.
• “reset” will delete the old subfolder (or entire mailbox if /).
• “replace” will delete and re-enter them.

‘Reset’ will be a bit faster on an empty destination mailbox because it skips most dupe checks.

Note: There were some duplication fixes and additional issues (mainly sync related) corrected with the tar formatter in 5.0.10.

Not a Zimbra Admin? Users can get the same zip/tar formatter on REST URL’s by visiting:
http:// server.domain.com/home/user?fmt=zip&query=is:anywhere

The zip format has been around for a long time, but doesn’t contain account & item metadata like the tar formatter automatically does:
http:// server.domain.com/home/user/?fmt=tgz

Infact, this same technique is currently used in Zimbra Desktop’s alpha backup solution.

If this approach doesn’t scale performance wise for your situation, or you simply don’t want to have everyone hit a REST URL for 30GB mailboxes all at the same time, here’s a collection of helpful scripts and other ways to systematically migrate:

Mysqldump & rsync with an interesting blob management technique: Zimbra2Zimbra

Imapsync for mail + postRestURL for contacts, calendar & filters: ZimbraMigrate (Expand the concept for tasks, documents, and briefcase items.)

Another method that could be extrapolated upon for migrations: Per User Mailbox Backup (OE Version – Zimbra :: Wiki)

Most of these solutions aren’t going to respect share permissions, but when pulling an account out of an environment that’s to be expected.

Zimlet spin-offs:
- Mail backup options for end users (.eml)
- Zimlet to save email in a txt file (.txt or html)

The above Zimlets are aimed at making quick self-copies & not for restores, but there are many methods for putting messages back into Zimbra, including tools like zmmailbox addMessge, zmlmtpinject, CURL, etc; for more info checkout these threads: Recover data from store folders & Moving Folders between users

If moving your entire server, I’m a huge fan of the install.sh -s trick when using NE backups to do so isn’t an option.

Part 1 covered Network Edition backup features, today’s snips apply to all editions.

First among the lesser known additions: We recently provided the possibility for a nice performance boost to some environments by adding the ability to turn on batched indexing in ZCS 5.0.3 (you can even fine tune it at the localconfig, COS, and account level). We’re not talking about when you re-index an entire account here, this is a change to the index-as-received model; now new items can sit in a ‘queue’ (really a ‘indexing deferred’ flag on the mail_items table of the pertaining mboxgroup database in MySQL) to run all at once when it reaches the zimbraBatchedIndexingSize threshold, saving you from all the tiny disk thrashing. It might not be immediately apparent that this works better, but you can mention it in the forums and we’ll show you the evidence to the contrary – it proves expecially useful for POP heavy or ZAD archive accounts.


 New & Enhanced Admin Tools

 

Your /opt/zimbra/bin & /opt/zimbra/libexec directories hold a wealth of tools to make your job easier.

zmdumpenv has been around for a long time, but underutilized – it grabs the basics that you should probably provide with every issue to help others understand where you’re coming from.

When you need to send ad-hoc SOAP commands to the server, the powerful zmsoap takes care of authenticating, generating the envelope, sending the request, and writing the response to stdout.

If your server freezes or is busy, running zmdialog can give that ‘my server hung’ support ticket a purpose. With JDK 1.5 it won’t collect a heap dump so you might also run /opt/zimbra/java/bin/jmap -heap:format=b [/opt/zimbra/log/zmmailboxd_java.pid] however zmdiaglog collects a core dump, from which it should be theoretically possible to get a heap dump. Thread dumps when you kill -QUIT/3 [pid] are helpful too. Info on ways to take them (like /opt/zimbra/libexec/zmmailboxdmgr threaddump) plus a handy script, are here.

There’s the MySQL metadata DB, the Lucene index, and the actual blob files on disk in /opt/zimbra/store. If you can click on a folder in the web UI and get results, it means MySQL is up and happy. If you open a message and get the infamous “missing blob for id” error, it means that the message files on disk aren’t where they’re supposed to be (ie: disk crash, not mounted/permissions, or you’ve recently pointed your volumes incorrectly). There’s all sorts of things you can do in that situation – here’s a more detailed list. Even if you don’t have a good grasp of the mailbox database structure yet, you can appreciate the need to easily determine if those blobs are still available (after you’ve moved whatever you can salvage of the store from corrupted drives to a better location). Enter the zmblobchk utility which can determine what files are missing – there’s also plans for a repair mode, for when you finally realize the blobs are gone and want to get rid of those UI error messages. Of course you’ll next consult the Network Edition’s restoreToTime feature or other backup solution you may have.

When you’re absolutely out of room on the disks housing your DB (seriously put in that purchase order for more storage – it’s cheap these days) in a pinch the optimizeMboxgroups.pl script (available in the public cache and added to the upcoming 5.0.10) can help you recover wasted space in your mail_item, appointment, imap_folder, imap_message, open_conversation, pop3_message, revision, and tombstone tables on each mboxgroup. Just note that it temporarily locks each table, and could use considerable IO while they’re being rebuilt. You can certainly use it pro-actively during a maintenance window to reclaim space as well.

There’s also a new wiki page on statistic collection so you can generate nice charts that help you figure out what you might need to tweak.

We could go on and on about the utilities in those folders, so throw up a test environment and experiment sometime – it may just make life easier when you have hundreds of users breathing down your neck. And if you should ever be ’stumbling around in the dark’ you can always enable additional debug logging to shed more light on the situation.

 

Continuing our earlier advice to take backups frequently, and secure them offsite – thought we’d highlight a few recent administrator related things added to ZCS that you might not have noticed.


 Network Edition Backup Enhancements

 

Speaking of backups, there are some new ways to take them in ZCS 5.0.x. With ever larger quota usage, full backups can often take a while to run, and even incrementals which process the redologs may still be one heck of a job when you’re talking thousands or millions of accounts. Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions – you’re completely backing up a target number of accounts daily rather than running incremental sessions. As a plus it automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. This allows you to do a point in time restore for any account. Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)

To save space, and therefore store older backups longer or run them more frequently, you can also auto-compress them with the –zip argument. This isn’t new, but it got improved handling of shared blobs in 5.0.5 as well as a -zipStore mode for speed. You can also adjust the buffer & queue capacity of the backup process, as well as additional options like the level of compression, or the number of archives per person via the backup_zip_copier_private_blob_zips localconfig attribute. Of course you lose the hard linking optimization (speed and space) for blobs that are in an earlier full backup already when working from the same disk – so it’s more advantageous for those off-site single-copies (you do make one often right?). However, there are legitimate uses for running it on your normal backups: Fewer files make it easier to copy or rsync later, and prevents you from running out of inodes. You can also easily delete individual backups rather than running zmbackup -del, and therefore keep just a few really old backups around for whatever compliance reasons you may have.

By-the-way, ZCS 5.0.6 added the ability to easily replay redologs from an arbitrary point in time with zmplayredo should you be in a unique situation that needs it. (Say you’ve been taking snapshot backups and but then need to restore, and you’ve also saved all the redologs since the snapshot. Or you take a snapshot, then manually copy redologs from the live system to bring the snapshot copy up to current. This allows you to force replay of them all and not just the uncommitted transactions.)

We’ve always recommended that: “After upgrade, you should run a full backup immediately as changes in this release invalidate all old backups.” It’s still good advice to have a fresh full, but there’s always the infrequent need to get data from an older minor version backup without throwing up a temporary machine. Well, with 5.0.x we aimed to make restores compatible across patch releases (i.e. an older 5.0.x backup, not a prior 4.5.x backup – major version restore is this RFE). There was a bug about zmrestore not handling database schema changes, but that’s fixed in 5.0.5 and later – so backwards compatibility for restore is now theoretically possible. And we’re also looking to put icing on the cake by adding a conversion tool to upgrade backups themselves to allow restore on later ZCS versions.

This may be the most important post you will ever read. If you’re a Zimbra Administrator, please read, and pass this onto your colleagues who use Zimbra. If you’re a CTO or CEO, take time to ask your Zimbra Admin about the subject of this post. This blog post is about backups.

Whether your an Open Source User, Zimbra Desktop User, or Network Edition Customer, you can do backups of your data. There is nothing worse than getting a call from a customer, or a Private Message from a Forum User that says, “I need help. My HD has crashed, and all my backups were on that drive.”

Let me Digress for a moment, and share with you my personal experience with backups. It’s sort of a Legend here at Zimbra. It all begins in 2005, Zimbra was the new kid on the block and I was an inexperienced System Linux admin for Tombstone Unified School District in Arizona. This was a small district with limited funds. When I interviewed for the post, the Superintendent handed me his card, and it had a Hotmail address on it. Right then, I knew this would be quite a difficult job.

One of the first things I did, was investigate E-mail Server platforms, and TCO (Total Cost of Ownership). A quick search on the interwebs (which is a series of tubes, not to be confused with the you tubes), yielded this new thing called a Zimbra. Much like our millions of Users, I downloaded it and tried it. It worked great! We used the Zimbra Open Source Edition (then it was in Beta).

I was so proud of my cool new setup, and our staff were so excited to finally have good e-mail and calendaring. In knew the importance of keeping backups, and archiving. All of you System Admins at Hospitals and Government institutions know what I mean. You get well acquainted with the Laws and Requirements. So, I would stay up until about midnight, and stop Zimbra, and rsync /opt/zimbra into /opt/zimbra/backups. Keep in mind that this was in the early days of Zimbra, and I wasn’t even an employee yet. As a matter of fact, myself and several others, pioneered the Open Source backup procedure.

Everything was great until one night I noticed I was running low on Disk Space on my /opt partition. So, I thought, “If I just remove all my backups, and make a fresh one, that will save me a bunch of space”. So I ran the following command as root:
rm -rf /opt/zimbra backups
Now, back then, we mounted a clamav partition ramdisk for quarantine purposes. The only indication that I had that something was wrong was when I got an error saying that it couldn’t unmount the partition because it was in use. Everything else in /opt/zimbra was gone…including my backups.

As most of you admins know, preserving data is important. We were involved in several litigation matters, and I would later be cited for obstruction of justice for destroying evidence.

When I noticed what had happened, I immediately called Zimbra and talked with MarcMac here at Zimbra. He tried to recover the inodes using Midnight Commander, but it was a total loss.

Lesson Learned. So, from one admin to another, please take time to make sure your backups are not located on the same machine that Zimbra is on. Please! We never want to hear about data loss. Whether an opensource user or network user, I hope you will take a few minutes to consider your backup strategy, and fix any single points in failure.

Learn from my experience.
-John (jholder)

We’ve had some great responses to my blog post yesterday about changing the default mailto links using Firefox. There’s a couple of problems with my method: 1) It’s not really scalable for an admin [ie no auto install] 2) Not really applicable for Safari and Internet Explorer.

(more…)