Hello Zimbra partners, customers, community and friends,
I would first like to apologize for the issues triggered by the recent patch release. I know that many people rely on the services offered by our products and when we do not execute well, the impact is felt by customers. While we take efforts to prevent problems, there are times when those efforts are not enough.
What Happened:
Upon releasing Zimbra 8.8.15 patch 32 and 9.0.0 patch 25, we recognized there was a defect which would affect a number of users. Our attempts remove the availability of the patches broke everyone’s ability to install the product or update to another patch. This happened to coincide with the release of information on a security issue covered in the previous patch release (8.8.15 patch 31 and 9.0.0 patch 24). Panic over the security issue drove a number of people to attempt to update, only to find out they could not.
Where things stand:
The patches for Zimbra 8.8.15 and 9.0.0 has been made available once again with the blocking issue resolved. The package repository issues have been corrected, fixing the issues with installations and upgrades. Everyone should now be able to install and upgrade without issue.
What we have learned:
Our quality assurance release gates failed to uncover the issues present in the patch release. While we have identified the specific gaps related to the patch release, we will be undertaking a full evaluation of our processes to ensure a better outcome in the future.
Our release process did not properly account for pulling back releases due to issues. This has been corrected and we are expanding our resources who have knowledge of the process for redundancy.
Moving forward:
I want to personally thank our partners, customers, community members and friends who worked closely with us during this incident. With your help, we were able to quickly identify the major issues and eventually resolve them. Without your support and interaction, things could have been much worse.We will be taking the lessons learned from this incident and implementing adjustments to our processes to ensure we do not repeat the mistakes made here.
We have also separated out our security and patch announcements from our email newsletter. We felt this would be a better way of highlighting security issues which could arise and bringing attention to them without the distraction of other content. We believe this will help you make better decisions around the content of patches and the availability of security releases. You can sign up for the announcements here.
Thank you.
John Eastman
John, I have to disagree with you that the broken packages have been fixed. I spent long hours last night with a collegue trying to fix mailboxd that would still not start. Rebooting the server yesterday yum on CentOS updated zimbra and after reboot, zimbra was broken. I found a link on a discussion forum about downgrading packages including zimbra-core-components, etc (4 packages in total), and only after downgrading to the previous version, did Zimbra start back up again.
Please remove the current highest version out there. There is still a jetty/java issue.
Thanks for your feedback, we are investigating to determine the issue.
Dear Jim,
Please share a link on a discussion forum about downgrading packages including zimbra-core-components
Thanks
Hi Jim, how can I downgrade those packages?? Thanks
Please fix java jetty problem as soon as possible, which waste CPU resources high percentage.
momentally exceeds 100% and calm down, this cycle repeats.
Something wrong compared with Patch23.
As the same as Jim, I downgraded 4 pakages including core/jetty/ldap/JDK) and as for me, zmmailboxd can run, but java’s PID suddenly active momentally(I confirmed ‘top’ command)
Temporary you should Patch23’s era perfectly, please.
As a temporary patch that will allow zimbra to deliver messages, with the cost of “soft” restarts, you could add a line like this in crontab (mind you, do not run it more than 6-8 times per hour):
/sbin/lsof -nPi | grep ‘:7025-.*CLOSE_WAIT’ | awk ‘{print $2, NR}’ | tail -1 | while read a b; do if [ $b -gt 18 ]; then kill -HUP $a; fi; sleep 20; done
Please continue the discussion on the forums: https://forums.zimbra.org/viewtopic.php?f=15&t=70865
How can I fix error message “OUT OF RPC CACHE” My product is 8.815_GA4173 Build 20211022125231 Thank you.
This error is shown on Classic UI if too many or recursive requests are made. Try and disable all Zimlets and remove all shares from/to the account.