Many administrators rely on third-party security scanners such as Crowdstrike Falcon to analyze their Zimbra server for malicious activity. One process that is frequently reported is jspawnhelper.
What is jspawnhelper?
While this is not very well documented, jspawnhelper can be run as the result of a call to ProcessBuilder.start() or Runtime.exec() in a java application. jspawnhelper is provided by OpenJDK and Java itself decides when it will or will not use jspawnhelper.
When is jspawnhelper used?
Zimbra has many command line applications that allow you to configure, control and provisioning you email server. One example of legitimate use of jspawnhelper is by zmprov, the command line utility to provision accounts on Zimbra.
For example if you would like to delete a number of accounts, you could run:
zmprov da test1@example.com zmprov da test2@example.com zmprov da test3@example.com
If you run these commands it can take up to 20 seconds for each of these commands to complete. Not a big issue if you only need to remove 3 accounts, but if you need to remove 100 or 10000 accounts it will be very slow. One solution for this is to only call zmprov once and pipe the arguments into zmprov. You can do this by creating a file in /tmp/listofaccounts with the following content:
da test1@example.com da test2@example.com da test3@example.com
You can then execute zmprov only once using:
/opt/zimbra/bin/zmprov < /tmp/listofaccounts
Removing 3 accounts takes about the same amount of time as removing just one account. And while you did not notice it, in the background zmprov may have called jspawnhelper.
Security scanner alerts jspawnhelper, is this good or bad?
If your security scanner alerts you of malicious use of jspawnhelper in Zimbra it is important to look at the arguments that are used with jspawnhelper. An example of legitimate use:
sh -c zmcontrol status > /dev/null
Some examples of suspicious use:
/usr/sbin/unix_chkpwd zimbra chkexpiry /usr/sbin/unix_chkpwd root chkexpiry /bin/sh -c /usr/lib64/sa/sa1 1 1
If you do a Google search for unix_chkpwd (or unix_chkpwd privilege escalation) you will find that the arguments in the above example do not make sense, as unix_chkpwd can only check the password expiration of the current account, adding zimbra or root as arguments does not make sense. The sa1 command is used to log system accounting statistics. Zimbra will not run the sa1 command out of the box, nor does Zimbra install the package that provides sa1.
Preventative measures
When you deploy Zimbra it is recommended to set-up Centralized Logging, preferably with Elastic Stack. This will allow you to gather base-line logging of you system. This way you can compare current log with historic logs and analyse the use of jspawnhelper. See: https://github.com/Zimbra/elastic-stack. In addition take a look at https://wiki.zimbra.com/wiki/Secopstips.
Remedial measures
If you are not sure if the use of jspawnhelper in your installation is malicious or not, you could (at your own risk) change the permissions of jspawnhelper. The default permissions look like this:
ls -hal /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper -rwxr-xr-x 1 root root 15K Feb 17 2022 /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper
To remove all permissions you can run:
chmod 000 /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper
Now every time jspawnhelper is called, it will fail. You can then try and see what functionality is affected and check your logs for errors. An example of failure reported on the command line:
java.io.IOException: Cannot run program "sh": error=0, Failed to exec spawn helper: pid: 14746, exit value: 127
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at java.base/java.lang.Runtime.exec(Runtime.java:594)
at java.base/java.lang.Runtime.exec(Runtime.java:453)
at jline.UnixTerminal.exec(UnixTerminal.java:290)
at jline.UnixTerminal.exec(UnixTerminal.java:275)
at jline.UnixTerminal.stty(UnixTerminal.java:266)
at jline.UnixTerminal.initializeTerminal(UnixTerminal.java:77)
at jline.Terminal.setupTerminal(Terminal.java:75)
at jline.Terminal.getTerminal(Terminal.java:26)
at jline.ConsoleReader.<init>(ConsoleReader.java:188)
at jline.ConsoleReader.<init>(ConsoleReader.java:183)
at jline.ConsoleReader.<init>(ConsoleReader.java:173)
at com.zimbra.common.util.CliUtil.enableCommandLineEditing(CliUtil.java:115)
at com.zimbra.cs.account.ProvUtil.main(ProvUtil.java:4130)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 14746, exit value: 127
...
To restore the permission you can run:
chmod 755 /opt/zimbra/common/lib/jvm/openjdk-17.0.2-zimbra/lib/jspawnhelper
Comments are closed.