Unbound…A Truer Word Has Never Described Software

Very occasionally a tool or software package stands-out to me on the pure virtue of being the right solution at the right time. If it happens to do exactly what you need RIGHT NOW, and other, perhaps more traditional, mature or well-known tools cannot, the best fit becomes clear. Unbound is exactly this kind of fit. Self-described as “a validating, recursive, and caching DNS resolver”, it’s utility boiled down to one particular line in the config (and the comments that describe it):

# Enable or disable whether the upstream queries use TCP only
# for transport. Default is no. Useful in tunneling scenarios.
tcp-upstream: yes

The reason why that particular ability…being able to force upstream DNS queries to go over TCP (instead of the default of going over UDP)…was important to me today is dumb, but the whole story is a useful example of network fun under the thumb of an ISP that can sometimes do really stoopid things, so I’ll go further and explain.

I should probably really begin by saying that I have not yet fully explored all the coolness wrapped-up in Unbound. Nor can I comment on its long-term stability or usefulness in an enterprise computing environment. When I have time, I may explore such things…but that was not my immediate focus for the tool.

You see, last night my Internet connection dropped. Well, it didn’t drop…I could still ping IP addresses, but DNS resolution wasn’t working…so for all intents and purposes, my connection was down (without name resolution or knowing all of your services by their IP addresses, a functioning link is pretty useless). I shook my head, went to bed and just knew the ISP would find the issue…and of course, it would be resolved by the time I got up. No worries, right? Wrong. Exact same issue this morning. Link: yes. DNS: no. Troubling.

I could ping 8.8.8.8 (the IP address of google-public-dns-a.google.com), but using dig to query DNS at that server resulted in an error…no response. Hmmmm…since I could ping, I started thinking about what I knew about DNS. DNS sends queries over UDP, a connectionless protocol, instead of TCP by default. UDP can be problematic, so lots of firewalls and even ISPs do some “magic” to limit the scope of damages UDP flood attacks can cause by limiting or even blocking some UDP traffic. With that knowledge, I decided to try a DNS query over TCP:

dig @8.8.8.8 google.com +tcp

Got a proper response with this query. So, whatever happened to my connection last night, it seems the primary effect was to “break” UDP traffic necessary for DNS upstream queries from my network. Sigh.

I spent quite a while looking through docs to see if my firewall could force (or proxy) upstream DNS to prefer TCP instead of UDP. Some pursuits down blind alleys gave me hope, but alas…nada. The only solution seemed to be to bring up a DNS server on my private network, and configure it to only use TCP for upstream. OK. Easy enough…or so I thought. I’m old skool, so immediately went looking at bind. The version of bind available (9.82) for CentOS 6.5 did not seem to have a configuration option suitable. Everything I read said bind could use TCP for queries, but would do so only if the server rejected the packets over UDP (or gave some other signal). Then, and only then could you query over TCP. UDP first, and then TCP as supported. As part of my issue appeared to be a DROP vs. REJECT scenario somewhere in the ISP config, the described fallback would never work as described for resolution.

My disappointment with bind led to an exhaustive Internet search (using my cell phone as a 4G access point). I finally happened upon Unbound, and found it could do exactly what I needed. I downloaded the source, compiled it in my CentOS 6.5 virtual machine (running via VMware Workstation on my laptop), worked through the config and finally, started and tested the service. Changed my VM from a NAT to Bridged connection so it got an address on my normal private network (allowing everything on the network to see it without having to contort routing tables), updated my private network DHCP settings to use the new DNS Server address for clients and suddenly, as if by some dark magic, I had by-passed the issue introduced by the change upstream at the ISP. Suddenly, every name had a number again. Yay!

Thanks to some useful tried-and-trues and Unbound, mischief managed. The fact I had to find a solution like this in response to a change that was neither requested nor communicated is almost beside the point…the fact that I could is a testament to the scratch-an-itch functionality found in lots of really cool Open Source technologies (yes, Unbound is BSD licensed). I may not have found Unbound without having this problem, but now that it has solved one problem for me, I’m inclined to look to it for other DNS related solutions down the road. Ain’t it cool?

One last thing I feel compelled to say….while I will not say what ISP on which I encountered my issues (I’m sure those of you that have seen similar issues may know), I will say that it WAS NOT Sonic.net. I’ve been a Sonic.net customer several times over the years, and have even recommended them to those that have asked. I have never had any such difficulties using their services…and issues I have had have been resolved with quick, easy phone calls. This particular network I’m on does not (yet) use Sonic.net…so for now, I’m a prisoner of another provider.