CBL blacklists my Datacenter IPs every day for almost 3 months.

There are many times in this job, that I managed not to cry….There are several others that I said….I quit, I can’t stand it no more….but it’s the IT virus inside us that makes us keep on doing this job with a smile of self-fulfillment on every success.

This particular problem started early December 2010 and we were over it till the 21^st of February 2011. We have a quite complicated set of Datacenters, one for inhouse operations and another one stated in another territory outside our headquarters just for services. The public IPs on our services datacenter started being blacklisted in CBL (http://cbl.abuseat.org/) on December 15^th 2010.

Thanks to great Blacklist check tool http://www.mxtoolbox.com/ we were able to monitor changes made to listings and walk us through our checking procedures.

Based on our application services that are: email hosting, DNS hosting, Apache Web Hosting, IIS Web Hosting, VOIP, all behind an array of ISA2004 servers, we started checking what is wrong.

We checked all customers served in-front and servers/clients behind the Firewall set for malware, virus and rootkit like apps that sent out spam. No results capable of producing this listing were found. I should state that every time we saw ourselves listed we went in the process of looking at another issue and delisted our selves accordingly in CBL. We got listed and requested delisting more than 50 times during this period, and as you may know there was the threat of inability to re-delist after so many times. We forced user password changes, stopped services that we didn’t need to, disabled ftps, disabled contact forms on all our websites and waited for google to expire its cache on them, checked robots for all sites –one by one, double-checked our DNS spf records on all domains hosted, checked EVERYTHING!!! Some of the servers where rebuilt in just a few hours, routers where changed, hell knows what we did not test.

It took us that long cause we had to make changes, check logs and Mxtoolbox and wait. We finally found out that every one or two days after the delisting we were relisted in CBL AGAIN! Just image how frustrating this was.

Actually if RTFM (Read the f@@@@ing manual) worked better, as we ITs tend to not read, we would have faster understood that all this was JUST a matter of a proxy. CBL clearly states this in its FAQ, however I believe they should somehow let you contact them and ask for a log stating the particular incidents occurring and resulting the listing, just in order to understand about timings between your checks and compare them with your own logs…No contact link is still available on their website, and every mail submitting I tried did not reach anyone (My mails where from other IPs, not the ones blasklisted, therefore should be accepted).

In order to make a long story short. The actual problem was quite similar to another guy’s post in the following url

This God sent guy refers to a continuous CBL listing due to lunix server he had. This gave me the idea to start looking for an open proxy in our Elastix servers. Bingo!!! There was one elastix PBX server left as a virtual machine on our HYPERV with 2 nics (one on a standalone intranet and one on the extranet). Since we had not done any modifications to it, and no ports where addressed to it through firewall and routers and it was just for testing purposes we never installed a firewall on it! HOW STUPID of mine. Note that all the rest production servers were fully protected during this period, but not THIS ONE! Apparently even without any port publishing on him the little bastard managed to open a web proxy….and that was just the start. So we thought that some clever spammers started using him for their purposes resulting the CBL blacklisting. So we left him as it was for a couple of days while we checked apache logs and my mail logs (mail.err, mail.info, mail.log). Nothing AGAIN, but still relisting!!!

I was frustrated once again, but somehow convinced that all this came from this.

Took the vhd to labs and started checking again, and again. During another installation, i forgot to give a name to the server, so the hostname was:  myprovider.com instead of the correct one.

So I Edited /etc/postfix/main.cf, and add/edit this line:
relayhost = my.server.comThen restart the postfix service:
# /etc/init.d/postfix restart

After stopping the Virtual PC –voila…not blacklisted in CBL!!!

What does not make sense to me is that CBL says it does not count over phishing. According to them, they use their own spam traps to hunt down spam sending IPs. So if it was not that, what was it? I still have the vhd in my labs and try to understand what is wrong with it by using the netstat -tap command. I guess I will know soon, if this small testing environment was hacked, but if it was hacked –how???? We never published it. It’s only connection to the web was to get updates through the FreePbx environment, nothing else!!! I will not stop this virtual pc unless I find out why….

The good outcome of this is that it made us study really hard about processes, procedures and email spam checks that in another case we would have not….Knowledge is power! So next time I will read more carefully what FAQs say, the problem is that you always need time and it is always against you.

Special Thanks to Peter Antakis.