Email issues – an education in reverse DNS

Email yesterday was acting very strange. We could send to most domains, but others would not be delivered. It never gave us a return message for non-delivery or otherwise. For all standard purposes we were sending and assuming it was arriving. However, this was not true… some domains, particularly cardinalhealth.com and novellaclinical.com would not get messages sent from us… No error, no nothing… just no delivery… I ended up calling support with CardinalHealth and had them do some tests on their end to make sure nothing was being blocked, filtered, etc from our domain. The guy was actually very helpful considering he could have just told me to go away…

Once I was satisfied that it wasn’t just a setting on the recipient side, I began testing for spam blacklists and other issues with the outside DNS. This is what I would have bet was the issue as some people received mail from us fine yet others did not. Normally this means we are on a blacklist that some peopIe subscribe to but others do not. However, we were on none. MXToolbox.com has some nice utilities and I ran a spam test from them. Nothing glaring was reported. Time to look at our own stuff then. In the interim, I began sending all of our mail through the smart host with HTC. This basically makes it so that all our mail, no matter what the address, appears to go through sccoast.net, which we can safely assume is not on any blacklists and has their domain in correct, working order. Normally, this will fix any outgoing email oddities. HTC doesn’t like to have us use this for any extended period as then they become the target of hackers and spammers instead of our servers. For now, though, it works and we can send to all the domains we need to.

I removed the Sonicwall Email Security appliance from the outgoing path and checked the queues when sending to one of the domains in question. I watched the server happily send to gmail.com, aol.com, and many others, but my email test to cardinalhealth.com would fail and sit on retry. The error was “SMTP protocol error occurred”. While generic, this actually points to a connectivity issue between servers. Basically, what our server is telling their server does not match what their server requires for a proper, secure connection. This can happen for many reasons, but it usually points to an open relay (sending through another server without authentication) or an improperly set DNS or rDNS (reverse DNS) entry. The SMTP test on MXToolbox showed that our SMTP server header did not match the reverse DNS records. It matched on the numbers, but not on the names.

The problem with our email setup is that we have many domains coming into a single server (except CUC) and no defined default email address for outgoing. Some people use gsuro.com, some use grandstrandurology.com, and others still use the atlanticurologyclinics.com addresses… Each of these domains have A and MX records set on GoDaddy to point them back to our external IP through HTC. No matter what address is used to send, it all goes out and back in to the same IP at the head of our network. Normally, this would be exactly what one would expect. However, our internal DNS also is the same as an external DNS entry (gsuro.com).

The problem with this lies in reverse DNS. Remote servers will often check to make sure that the DNS entry for the sending server’s domain matches the IP that the ISP is providing using reverse DNS. This uses the SMTP header which is the only thing that cannot be spoofed in mail. The IP address is tagged in the message header when it is sent and cannot be changed as the protocol itself is what makes the tag. This IP address must match the domain that is returned with a reverse DNS call. In other words, when we send as user@grandstrandurology.com to user@outsidedomain.com the server at outsidedomain.com will then check the IP address in the message header and make sure it translates back to grandstrandurology.com … If it comes back to atlanticurologyclinics.com, you probably have an open relay or incorrectly configured DNS / Exchange server. Pretty easy to fix but it also blocks a lot of spammers and spoofers with this technique. It has become the industry standard to have a matching rDNS entry for any email domain.

Now, I checked our reverse DNS and it is pointing to grandstrandurology.com . This is fine even if the sender is using an atlanticurologyclinics.com address as long as the server reports back grandstrandurology.com in its SMTP banner. I ran the MXToolbox SMTP Test and it mentioned a warning (not an error) that the Reverse DNS did not match the SMTP banner. The transcript of the connection showed that the server was announcing itself as es.gsuro.com. However, the IP address set for reverse DNS with HTC was returning grandstrandurology.com … This is a mismatch, but it normally it will work because the server will still accept messages sent to this domain. In a normal, single domain site this really would never enter into the equation as there will never be any other domain used anywhere, so everything will match up by default. The SMTP banner did not match because it was going straight to a server on our internal domain (gsuro.com) which also is one of our external domains… this is why standard practice dictates that the internal windows domain should never contain a .com, .net, or .org extension. It should be setup as a .local unless you are actually setting up a server in a domain that will actually be visible to the outside. In our case, we only have email visible on the outside so there is no need to have the domain named gsuro.com. However, as any admin will tell you, changing the domain name once it is fully in place is easily one of the worst ideas you can have. It is not easy and you might as well plan on rebuilding everything. So, we work around the poorly named domain until we have to do otherwise. Thankfully this is not the time for that. In a nutshell, the email is sending out as grandstrandurology.com, reverse DNS reports that the domain should be grandstrandurology.com but the sending server is actually saying that it is on the gsuro.com domain.

Enter in the other variable: Sonicwall Email Security. You use it like you would use any smart host in Exchange. All mail gets filtered and routed by exchange to the next hop in the list, which should be the ES device. The ES device then checks for spam, virus, etc and forwards to the next server in the list (cardinalhealth.com in this example) if everything checks out. It was at this point that the message should have just been sent and never looked at again. The ES was sending it, but the remote side was not accepting it. However, there was no NDR, so we didn’t know where the message was falling. The monitoring and reporting on that device is pretty bad. It is very robust but very hard to get what you want out of it… I cannot track a single message, as far as I know… once the ES sends it out, it goes into the pile and can’t really be extracted for research…. I guess they figure if you want to do message testing you can just have the mail go straight to your server and do it from there. Pretty lame, I think… Regardless, the ES device is the first item that mail sees when it is sent to us. A simple telnet test shows the SMTP banner is announcing as es.gsuro.com (the local hostname). This is, as far as I can tell, the root of the problem. Our ISP, HTC, which basically vouches for us using reverse DNS, reports that we are grandstrandurology.com but we are turning around and saying that we are gsuro.com with our email server. On Exchange, this is an easy fix. However, the ES device does not have a way to masquerade the external banner as anything other than the hardcoded hostname. In this case, because it was an internal device, we used the internal structure, which did not match what the world was seeing.

There are a couple of ways to fix this. I could call HTC and change the rDNS to be gsuro.com instead of grandstrandurology.com … In reality, it should be set to atlanticurologyclinics.com and the device should say the same. However, this is a lot more trouble. Instead, I changed the hostname of the ES device to es.grandstrandurology.com and it now reports the SMTP banner matches the reverse DNS entries. We are currently sending directly out of the ES instead of sccoast.net. We are sending/receiving fine and I have verified that we can still send to cardinalhealth.com and novellaclinical.com with no problems.

The underlying difficulty with an issue like this is that nothing appears to be wrong.  The remote servers were not sending an NDR so I assume it goes fine.  The end user has to tell me that they aren’t receiving messages.  Hopefully this change fixes everything and it won’t get us put on some blacklist somewhere that will further complicate things….

Advertisements

~ by oldirtdog on December 1, 2011.

2 Responses to “Email issues – an education in reverse DNS”

  1. Thanks for using our Tools and Forums to help diagnose your rDNS issue. We are psyched about our tools and would greatly appreciate any feedback you might have.

    • I used to use DNSStuff but they started charging and I don’t have a subscription anymore… I really like how your tools link to the next test you may need: domain to IP then to Spam or SMTP test… Also, the fact that the results stack up and are visible in chronological order is really nice to look at things or create reports at a glance… If you would like a review, I would be happy to write a quick one if you give me a link…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: