It was DNS, It’s Always DNS!

What is DNS and how I broke it.

  Last week.  Catracing.org suffered a 24 hour downtime.  This made my WordPress (blog), the phpBB (Forum site, and the Commodore 64 Bulletin Board system running in emulation unable to their users on the internet.  I am fortunate to be able to maintain 98.5% uptime on my self hosted services!  So to me it is a bit deal when something breaks, especially when looking back I am the one who caused the issue! 

  So what happened?  In a nutshell, I broke DNS!.  Through a well intentioned but poorly planned migration from one DNS service provide to another, a simple but catastrophic misunderstanding of a key piece of the top level of DNS, and nor remembering a key password put me at the mercy of waiting on customer support. 

  The ability of being able to come up with a workable plan B prevented me from experiencing an even lengthier downtime.

  So let’s take a look at a few things, what is DNS, how did I break it, how did I fix it, and what can I do to prevent or mitigate a similar failure in the future.

How did I break it?

  Upon returning from Korea and becoming an ATT internet customer, SMTP (Port 25) has been blocked both incoming and outgoing.  This is commonly done by internet service providers to combat spam, due to improperly configured email servers.  To get around this and continue running my email server had to subscribe to an SMTP replay service.  One of the other services this company provides is dynamic hostname registration, and providing zone level DNS services, which I have previously been getting from dyndns.org.  I had the bright idea, that if my SMTP relay server will also do dynamic IP and zone level DNS, I can cancel the dyndns services and only pay one company for all 3 services!  Great idea, right?  It would have been, if I had remember all the key processes of DNS name resolution and all of the steps I needed to do to migrate.   

  To compound the problem, I forgot the password to go in and make the changes that would set everything right, but more on that in a second.

What is DNS?

  Let’s take a step back to give a very quick and dirty rundown on what is DNS, you can better understand where my problem lay.  DNS or Domain Name Services is a hierarchical group (A very very large group) of servers that match domain names (google.com, yahoo.com, catracing.org) to their respective ip addresses.  While you and I think in terms of names, computers thing in terms of numbers. A computer really has no idea what google.com is, however it does know it by its Internet Protocol “IP” address 74.125.136.104.   Think of it as a White Pages of the internet.   If you wanted to call Baxter D. Bat, unless you call him all the time, you might not remember that his phone # is (619)555-1212.  So, you took out the phone book, looked up Bat, then found Baxter D, and then had the number you could then call him on the phone.  Keep this analogy of comparing an IP address and a phone number in your head, it will come in handy. 

  For instance, just like a phone number has multiple parts (Area Code)Local Prefix – Subscriber Number.  NOTE:  I am omitting country code for simplicity.  So does a domain name.  Let’s take www.catracing.org for example.  I am going to work this backwards from right to left, because it better feeds in to what I broke, so work with me here.  .org (TLD or Top Level Domain), .catracing (domain), www. (subdomain).

  The Top Level Domain or TLD is the groups IANA breaks down the taxonomy of various internet services.  For instance.

.org = Non Profit Organization
.com = Commercial Organization
.net = Network service provider
.gov = Government Organization.

  In a nutshell, here is what happens when you type www.catracing.org into your web browser and hit return.

  Your IP stack will look to be what the lowest level DNS service is, if you are on a corporate network, and have you own DNS server it will first query this server to ask if it knows who www.catracing.org is.   Most likely it will not, so it will ask the DNS server if it has a forwarder configured.  Let’s say it does, and it is configured for 8.8.8.8 (Googles Name Server).  It goes out and asks the Google Name Server.  Google says, NOPE, so it sends it out to its forwarder, and let’s say it is actually IANA.  It looks in it’s .org TLD, and say’s yup I see catracing.org, and in the domain registration, which was done by name.com, it shows the name server it is registered under is  ns1161.dns.dyn.com.  So the next DNS query is to ns1661.dns.dyn.com.  Do you know who www.catracing.org is?   “Why yes I do, its IP address is 104.57.170.70.  Now the computer knows the ip address, your computer can go there and retrieve the web page.

So,  What happened?

  Now that I have gone over how DNS works, and put the pieces in place that would cause catracing.org to become unavailable for 24 hours, lets go over how it happened.

I painstakingly recreated my DNS zone records on my new service, and used nslookup directed to their servers to make sure everything was resolving correctly.  Which it did.  Calling it good I canceled the service on dyndns.org.

  Several hours later when I was already asleep, catracing.org completely disappeared from the internet!  I did not catch the notification that my blog was unreachable until later in the day.  At first, I thought that the script I wrote to update my ip address did not work, and my IP address had changed.  However while pinging it, I noticed that it wasn’t just the pings to my IP address were coming back unreachable, it was www.catracing.org could not be resolved..   I had forgotten that the top level query, if it needed to do so, would tell whoever was searching for my site, to look on the dyndns.org servers, and I had completely removed my domain records. 

  Okay, no problem, I just need to go update my domain registration with the new DNS servers.  Oh no!  I have not done anything with that account since they spun out of DYNDNS.ORG in 2019!  I don’t remember my password. 

  I know what you are asking yourself, Just ask for a password retrieval..   Simple..  Except…. If you can’t find .catracing.org on the internet.  How will you send the email with the recovery link.  There, right there lay the real crux of the matter!

  I contacted their support, and they could change my registered email address, provided I can prove who I am, and that I am a living person.  No problem!  Sent what they asked me, however.  The address associated with my domain registration was from 2019, and did not match my address that I had moved from twice and was not on my current ID.  So with the site already being down 24 hours, the easiest way would be to re-open the account with DYNDNS.ORG (Which turned out to only be $5 for a month).  Recreate enough of the zone records (The MX record is really the only one I needed), wait potentially 24 – 48 hours worst case (DYNDNS.ORG) had me available within an hour.  Retrieve the password recovery link, then update the DNS information in my domain registration information.  Which is what I ultimately decided to do.

  It took less than an hour to get at least my domain and MX record reestablished, and searchable again on the internet.  I now have until the middle of September to try migrating my outside DNS.  At least this time I have a plan!  Have access to all of the necessary accounts and have time to have a fall back in case it does not work!  All things I should have thought about and had available to me the first time!

  Had this been at work and a critical piece of the business I supported, it would have been embarrassing at best, a resume updating event at worse, but this is one of the reasons I choose to self host!  So I can learn these kinds of lessons at home, and not in an environment where I am playing “Bet Your Job”!

  I hope you enjoyed this week’s blog, I hope you come back next week for more exciting content!  If you wish to be notified when new content is posted, please consider registering by clicking HERE!