Ok, this site is going dark… Well, it will still be here, but I’m not going to update this one any longer… all of my further OlDirtDog blog posts will be going to L337Inc.net. Please go there and enjoy. Thanks for any patronage you may have given me in the past… All three of you are very appreciated…
I recently found out that GoDaddy gives you free hosting when you purchase a domain through them. It’s very simplistic and has a pretty low limit, but I’m going to make use of it if I get it for free… I’m going to migrate this blog over to the one at L337Inc.net . It will still be the same content and the same backend. However, I have had that domain forever and just never did anything with it. Time to make use of it. I doubt anyone will be hurt by this move since nobody really looks here anyway. But just in case, please update your links….
Email yesterday was acting very strange. We could send to most domains, but others would not be delivered. It never gave us a return message for non-delivery or otherwise. For all standard purposes we were sending and assuming it was arriving. However, this was not true… some domains, particularly cardinalhealth.com and novellaclinical.com would not get messages sent from us… No error, no nothing… just no delivery… I ended up calling support with CardinalHealth and had them do some tests on their end to make sure nothing was being blocked, filtered, etc from our domain. The guy was actually very helpful considering he could have just told me to go away…
Once I was satisfied that it wasn’t just a setting on the recipient side, I began testing for spam blacklists and other issues with the outside DNS. This is what I would have bet was the issue as some people received mail from us fine yet others did not. Normally this means we are on a blacklist that some peopIe subscribe to but others do not. However, we were on none. MXToolbox.com has some nice utilities and I ran a spam test from them. Nothing glaring was reported. Time to look at our own stuff then. In the interim, I began sending all of our mail through the smart host with HTC. This basically makes it so that all our mail, no matter what the address, appears to go through sccoast.net, which we can safely assume is not on any blacklists and has their domain in correct, working order. Normally, this will fix any outgoing email oddities. HTC doesn’t like to have us use this for any extended period as then they become the target of hackers and spammers instead of our servers. For now, though, it works and we can send to all the domains we need to.
I removed the Sonicwall Email Security appliance from the outgoing path and checked the queues when sending to one of the domains in question. I watched the server happily send to gmail.com, aol.com, and many others, but my email test to cardinalhealth.com would fail and sit on retry. The error was “SMTP protocol error occurred”. While generic, this actually points to a connectivity issue between servers. Basically, what our server is telling their server does not match what their server requires for a proper, secure connection. This can happen for many reasons, but it usually points to an open relay (sending through another server without authentication) or an improperly set DNS or rDNS (reverse DNS) entry. The SMTP test on MXToolbox showed that our SMTP server header did not match the reverse DNS records. It matched on the numbers, but not on the names.
The problem with our email setup is that we have many domains coming into a single server (except CUC) and no defined default email address for outgoing. Some people use gsuro.com, some use grandstrandurology.com, and others still use the atlanticurologyclinics.com addresses… Each of these domains have A and MX records set on GoDaddy to point them back to our external IP through HTC. No matter what address is used to send, it all goes out and back in to the same IP at the head of our network. Normally, this would be exactly what one would expect. However, our internal DNS also is the same as an external DNS entry (gsuro.com).
The problem with this lies in reverse DNS. Remote servers will often check to make sure that the DNS entry for the sending server’s domain matches the IP that the ISP is providing using reverse DNS. This uses the SMTP header which is the only thing that cannot be spoofed in mail. The IP address is tagged in the message header when it is sent and cannot be changed as the protocol itself is what makes the tag. This IP address must match the domain that is returned with a reverse DNS call. In other words, when we send as email@example.com to firstname.lastname@example.org the server at outsidedomain.com will then check the IP address in the message header and make sure it translates back to grandstrandurology.com … If it comes back to atlanticurologyclinics.com, you probably have an open relay or incorrectly configured DNS / Exchange server. Pretty easy to fix but it also blocks a lot of spammers and spoofers with this technique. It has become the industry standard to have a matching rDNS entry for any email domain.
Now, I checked our reverse DNS and it is pointing to grandstrandurology.com . This is fine even if the sender is using an atlanticurologyclinics.com address as long as the server reports back grandstrandurology.com in its SMTP banner. I ran the MXToolbox SMTP Test and it mentioned a warning (not an error) that the Reverse DNS did not match the SMTP banner. The transcript of the connection showed that the server was announcing itself as es.gsuro.com. However, the IP address set for reverse DNS with HTC was returning grandstrandurology.com … This is a mismatch, but it normally it will work because the server will still accept messages sent to this domain. In a normal, single domain site this really would never enter into the equation as there will never be any other domain used anywhere, so everything will match up by default. The SMTP banner did not match because it was going straight to a server on our internal domain (gsuro.com) which also is one of our external domains… this is why standard practice dictates that the internal windows domain should never contain a .com, .net, or .org extension. It should be setup as a .local unless you are actually setting up a server in a domain that will actually be visible to the outside. In our case, we only have email visible on the outside so there is no need to have the domain named gsuro.com. However, as any admin will tell you, changing the domain name once it is fully in place is easily one of the worst ideas you can have. It is not easy and you might as well plan on rebuilding everything. So, we work around the poorly named domain until we have to do otherwise. Thankfully this is not the time for that. In a nutshell, the email is sending out as grandstrandurology.com, reverse DNS reports that the domain should be grandstrandurology.com but the sending server is actually saying that it is on the gsuro.com domain.
Enter in the other variable: Sonicwall Email Security. You use it like you would use any smart host in Exchange. All mail gets filtered and routed by exchange to the next hop in the list, which should be the ES device. The ES device then checks for spam, virus, etc and forwards to the next server in the list (cardinalhealth.com in this example) if everything checks out. It was at this point that the message should have just been sent and never looked at again. The ES was sending it, but the remote side was not accepting it. However, there was no NDR, so we didn’t know where the message was falling. The monitoring and reporting on that device is pretty bad. It is very robust but very hard to get what you want out of it… I cannot track a single message, as far as I know… once the ES sends it out, it goes into the pile and can’t really be extracted for research…. I guess they figure if you want to do message testing you can just have the mail go straight to your server and do it from there. Pretty lame, I think… Regardless, the ES device is the first item that mail sees when it is sent to us. A simple telnet test shows the SMTP banner is announcing as es.gsuro.com (the local hostname). This is, as far as I can tell, the root of the problem. Our ISP, HTC, which basically vouches for us using reverse DNS, reports that we are grandstrandurology.com but we are turning around and saying that we are gsuro.com with our email server. On Exchange, this is an easy fix. However, the ES device does not have a way to masquerade the external banner as anything other than the hardcoded hostname. In this case, because it was an internal device, we used the internal structure, which did not match what the world was seeing.
There are a couple of ways to fix this. I could call HTC and change the rDNS to be gsuro.com instead of grandstrandurology.com … In reality, it should be set to atlanticurologyclinics.com and the device should say the same. However, this is a lot more trouble. Instead, I changed the hostname of the ES device to es.grandstrandurology.com and it now reports the SMTP banner matches the reverse DNS entries. We are currently sending directly out of the ES instead of sccoast.net. We are sending/receiving fine and I have verified that we can still send to cardinalhealth.com and novellaclinical.com with no problems.
The underlying difficulty with an issue like this is that nothing appears to be wrong. The remote servers were not sending an NDR so I assume it goes fine. The end user has to tell me that they aren’t receiving messages. Hopefully this change fixes everything and it won’t get us put on some blacklist somewhere that will further complicate things….
Have you ever had one of those moments where you know you CAN’T have lost what you may have just, in fact, lost? Something so important that the chance that it may be lost forces your brain into instant damage control mode and begins thinking about excuses and ways to fix the situation immediately. Well, I know the feeling… I’ve been there a few times, as I’m sure most of us have… In this case, it was the company credit card…
I do not have my own credit card (with the company paying) but I have to make purchases sometimes. I end up borrowing one of the other admins’ cards. Pretty standard fare as far as I am concerned. Keeps me from being liable for any crazy purchases or having it stolen along with my wallet. After making said purchases, the card is normally returned the next day. In theory, this works, but sometimes our paths don’t cross immediately after and I’ve never been one to just leave something like a credit card unattended on someone’s desk, inbox, or even under their door. I would just hate for the last thing I say to be “I promise I put it in your desk!” Not me…
In this particular situation, I had gone to Lowe’s to purchase some items and install them the next day. It was actually two days later when I opened the bag from Lowe’s and stumbled upon the card. Uh-oh. I’ve already forgotten I even had it at this point. So, now, my mind doesn’t know where the card belongs to make sure I don’t forget it again. It didn’t exist up to 5 minutes ago! Well, I smartly put it in my pocket. Surely I wouldn’t lose my pants, right? I have to go in the pockets many, many times a day for keys if nothing else. I’ll remember. Fast forward through the day and I get in my car and leave…. The next morning, I remember the card. I just remembered I needed to check on it before I headed back to the office because I would certainly deliver it then. Hand in pocket… nothing. NOTHING! Mind racing, I realize there actually is something in my pocket as I check again. A single, but sizable hole. Oh no! It dropped out of my pocket, but where?!? I would easily have missed something like that sliding out, but it could have literally been anywhere inside the last office and my home… from surfside to 82nd… oh man…
I decide to get started by driving to 82nd, where it most likely was considering I had my pants on the whole time I was working there… Also, I couldn’t remember another place that I had seen it… Long drive ahead considering I needed to be in Conway already and I was on the way to the far end of MB… Going to be a long day… On the way, I decided to check in my car… You know, in case I put it somewhere else that I wouldn’t forget… Well, I reached beside my seats and found nothing… nothing in the trunk nor glove compartment… ugh, but it’s gotta be at 82nd… Ok, nothing has changed… calm down… On a whim, I reached my hand behind the seat into the rear area and my hand hit something… I already knew what it was…
I had a moment that I wish I could recreate a thousand times a day… That same racing mind that was filling up with dread and excuses earlier filled just as quickly with the feeling of elation that you gain from breaking down a wall or accomplishing a big goal. Then, a rush of blood to the head that forces a big smile even though you are the only one that will see it. I had found the card and rendered useless all the terror I was looking forward to experiencing. I changed course and the day continued on as it should. I came to work and handed the card over as if nothing had happened.
“Oh, I forgot you had that!”
Enjoyed yesterday. Had a surprise lunch at work of pizza. The ladies even
came and surprised me! Installed some wireless in Conway and worked through
other things. Had to work on my Bday, which is never ideal, but it wasn’t
bad. I know if I was home I would just be helping with the girls, so it
wouldn’t be any different than any other day off. We ate at Nacho Hippo
with the Willises and topped it off with a visit to Plyler Park to see TTU.
We ended up getting home a little late, so it was pretty much time for bed
as soon as we got home…
Larn is coming to town this weekend so maybe we’ll continue the festivities
Really? Does that actually work? Is there actually anyone out there that sees these things and doesn’t feel loathing toward whoever put them out? My initial reaction is to just throw them away… not come see Jesus…