When your security scanner alerts for jspawnhelper in Zimbra

Many administrators rely on third-party security scanners such as Crowdstrike Falcon to analyze their Zimbra server for malicious activity. One process that is frequently reported is jspawnhelper.

What is jspawnhelper?

While this is not very well documented, jspawnhelper can be run as the result of a call to ProcessBuilder.start() or Runtime.exec() in a java application. jspawnhelper is provided by OpenJDK and Java itself decides when it will or will not use jspawnhelper.

When is jspawnhelper used?

Zimbra has many command line applications that allow you to configure, control and provisioning you email server. One example of legitimate use of jspawnhelper is by zmprov, the command line utility to provision accounts on Zimbra.

For example if you would like to delete a number of accounts, you could run:

zmprov da test1@example.com
zmprov da test2@example.com
zmprov da test3@example.com

If you run these commands it can take up to 20 seconds for each of these commands to complete. Not a big issue if you only need to remove 3 accounts, but if you need to remove 100 or 10000 accounts it will be very slow. One solution for this is to only call zmprov once and pipe the arguments into zmprov. You can do this by creating a file in /tmp/listofaccounts with the following content:

da test1@example.com
da test2@example.com
da test3@example.com

You can then execute zmprov only once using:

/opt/zimbra/bin/zmprov < /tmp/listofaccounts

Removing 3 accounts takes about the same amount of time as removing just one account. And while you did not notice it, in the background zmprov may have called jspawnhelper.

Security scanner alerts jspawnhelper, is this good or bad?

If your security scanner alerts you of malicious use of jspawnhelper in Zimbra it is important to look at the arguments that are used with jspawnhelper. An example of legitimate use:

sh -c zmcontrol status > /dev/null

Some examples of suspicious use:

/usr/sbin/unix_chkpwd zimbra chkexpiry
/usr/sbin/unix_chkpwd root chkexpiry
/bin/sh -c /usr/lib64/sa/sa1 1 1

If you do a Google search for unix_chkpwd (or unix_chkpwd privilege escalation) you will find that the arguments in the above example do not make sense, as unix_chkpwd can only check the password expiration of the current account, adding zimbra or root as arguments does not make sense. The sa1 command is used to log system accounting statistics. Zimbra will not run the sa1 command out of the box, nor does Zimbra install the package that provides sa1.

Preventative measures

When you deploy Zimbra it is recommended to set-up Centralized Logging, preferably with Elastic Stack. This will allow you to gather base-line logging of you system. This way you can compare current log with historic logs and analyse the use of jspawnhelper. See: https://github.com/Zimbra/elastic-stack. In addition take a look at https://wiki.zimbra.com/wiki/Secopstips.

Remedial measures

If you are not sure if the use of jspawnhelper in your installation is malicious or not, you could (at your own risk) change the permissions of jspawnhelper. The default permissions look like this:

ls -hal /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper
-rwxr-xr-x 1 root root 15K Feb 17 2022 /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper

To remove all permissions you can run:

chmod 000  /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper

Now every time jspawnhelper is called, it will fail. You can then try and see what functionality is affected and check your logs for errors. An example of failure reported on the command line:

java.io.IOException: Cannot run program "sh": error=0, Failed to exec spawn helper: pid: 14746, exit value: 127
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
    at java.base/java.lang.Runtime.exec(Runtime.java:594)
    at java.base/java.lang.Runtime.exec(Runtime.java:453)
    at jline.UnixTerminal.exec(UnixTerminal.java:290)
    at jline.UnixTerminal.exec(UnixTerminal.java:275)
    at jline.UnixTerminal.stty(UnixTerminal.java:266)
    at jline.UnixTerminal.initializeTerminal(UnixTerminal.java:77)
    at jline.Terminal.setupTerminal(Terminal.java:75)
    at jline.Terminal.getTerminal(Terminal.java:26)
    at jline.ConsoleReader.<init>(ConsoleReader.java:188)
    at jline.ConsoleReader.<init>(ConsoleReader.java:183)
    at jline.ConsoleReader.<init>(ConsoleReader.java:173)
    at com.zimbra.common.util.CliUtil.enableCommandLineEditing(CliUtil.java:115)
    at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:4130)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 14746, exit value: 127

To restore the permission you can run:

chmod 755  /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper


No comments yet.

Leave a Reply

Copyright © 2022 Zimbra, Inc. All rights reserved.

All information contained in this blog is intended for informational purposes only. Synacor, Inc. is not responsible or liable in any manner for the use or misuse of any technical content provided herein. No specific or implied warranty is provided in association with the information or application of the information provided herein, including, but not limited to, use, misuse or distribution of such information by any user. The user assumes any and all risk pertaining to the use or distribution in any form of any subject matter contained in this blog.

Legal Information | Privacy Policy | Do Not Sell My Personal Information | CCPA Disclosures