May 31

Business Continuity

On the way to my carpool drop off this morning my phone began to buzz uncontrollably. I knew it couldn’t be good but since I was driving and PA law is now in effect about texting and driving I didn’t dare look. Once I arrived I pulled out my phone to find the internet at the office is down. I’m in for it today, I thought. Then the confirmation came–I was 3 minutes from getting to the office and I get a text from my boss asking how long till I got to the office. Yup, it’s going to be a “Fun” morning.

Once I got to the office we were already forty to forty five minutes into the outage and it was all hands on deck. The ISP was already engaged by our Network Engineer and they were working on the issue but didn’t know how long. No one in the office has internet or external e-mail communication. We do have a second internet connection but no action plan to use it if our primary internet would go out, problem number one.

Problem number two all critical services for communication to our clients run out of this building. I don’t have a script to work from so i start with getting a small group of our customer support team on our secondary internet so they can support our customers and maybe get lucky that the outage is small and don’t have to execute a plan on getting external e-mail up.

Getting them connected was easy however they need access to our production site. I enlisted a couple peers to go and install the VPN client and the appropriate profile and get them connected to our production site so they can fully support our customers. at this point we are an hour forty five in and the ISP still didn’t know whats wrong. At this point I need to start executing on a plan to get external e-mail up.

Problem number three, we don’t have a spam filter for our DR leg, and problem four, our firewall in the DR site is not configured to allow mail traffic to my DR mail servers. My morning just gets better. So I begin discussing with our Network Engineer the plan to take the new replacement spam unit to our DR site and getting the firewall configured to allow mail traffic through.

We then get the Director of IT to sign off on the plan and start executing. I changed the IP address of the Spam appliance and then gave it to my coworker who is responsible for the data center it was going to. It should only take about an hour to get to the site and since the unit was already configure and tested and ready to be transitioned to the active unit on Thursday evening. The plan at this point was to add a secondary MX record to our External DNS for the DR site and then configure the firewall with the settings. It was 11:20 and I now just had to wait for everyone else to get their pieces in place.

Then at 11:30 everything came back up. Mail started flowing again and everyone in the office had internet again. Total time of outage 3:13.

This “little” outage shined a bright light on many areas that were lacking cogency plans. It was a project that is on my Summer to-do list and had started giving it a lot of thought. I had even started doing more in-depth research. We are a SaaS organization so most of our efforts for DR are focused on our other data centers. We have DR plans for when something were to happen to them but not to our corporate office. It is unfortunate that I didn’t have a Business Continuity Plan to execute in this situation and come out of this incident looking like a superstar however it may help in executive buy-in to put more resources and focus on how important the corporate office is to the day-to-day operations of the business. Starting tomorrow the focus is on getting a working BCP in place by the end of June and monthly test to ensure my team is ready when an event like this or worse happens, because it will happen again.


  1. Farah Ramser

    I discovered your weblog web site on google and check a few of your early posts. Continue to maintain up the superb operate. I just extra up your RSS feed to my MSN Information Reader. Searching for forward to reading more from you later on!…

  2. click here

    Hello, I just hopped over to your webpage thru StumbleUpon. Not somthing I would typically read, but I enjoyed your thoughts none the less. Thank you for making something worthy of reading through.

Leave a Reply