Transition Nov 2007

Transition Nov 2007

This is the 'transcript' of the messages that went out during the dark days of Nov 4 - 7, 2007. Because the data center where we kept our servers was bought out by another company some very unexpected things happened. We had never in over 5 years of hosting websites had experienced this kind of downtime, ever. And we never will again.

November 7, 2007
8:00 pm

Now that things are going smoothly again, I want to personally apologize for the incredibly long downtime. I've never seen anything like it. Over 100,000 websites were affected, all clients of about 100 or so web hosting companies. What a nightmare that was. My sincere apologies for the huge disruption to your business and other communication.

I have tried to return every phone call, voice mail and email, but please excuse me if I missed you. I didn't know the phone could ring so many times in a day!

I also want to thank the many people who sent messages of kindness during this very difficult time. To them, my sincerest gratitude. You really helped me keep my spirits up (or at least not completely down) during this 4 1/2 day fiasco and reminded me that things would get better.

And to those who happened to receive invoices or overdue notices during this time, please don't take it as a sign of callousness. Our invoices are automatically generated and sent.

Once I can think clearly again I will come up with some way to make up for your trouble. Suggestions are welcome!

Sincerely,

Paula M. Pierce

November 7, 2007
1:00 pm

Looks like we're back up!

For those of you who chose to move your mail to the new server, please change your email settings in Outlook to the ones below, in the previous post and you should start seeing email tonight.

Thanks for hanging in there during this very trying time!

November 7, 2007
10:30 am

Update from the data center:

Requests for ARP have now been resolved, and the name server issues are now under control. We have been sequentially bringing customers back up one at a time to validate both Quality Assurance (QA) and optimum performance levels.

All of those servers that we have selectively validated and brought back on line are operating efficiently. For those that we have brought back on line, we are getting feedback that their overall performance levels have improved significantly as our teams continuously are bringing an increasing number of customers back on line.

We will continue to provide these updates on our progress throughout the day. As always, the toll-free phone lines (listed below) are open to answer your questions.

November 7, 2007
10:00 am

As of 10:00 am Wednesday, the data center techs keep reporting many sites coming online, but most of you are still down.

I set up a new server unrelated and completely unconnected with the previous servers. I am now offering to re-setup your account on the new server.

Two Very Important Things:
1. If we setup your account on the new server, and make a change to your domain record to point there, it will take 24 - 48 hours to be available on the entire internet. That's a minimum of 24 hours from the time we setup your account, not from the time you email your request.
2. You'll need to FTP your website files up to your new account to get your website live. If you do not have a copy of your website, and we designed or maintain your site, we may have a copy here and will upload it for you. It may be an older version. (Full backups, including databases, from Friday are located at the data center and are as inaccessible as the originals right now.)

If you choose to have us setup your account on the new server, please email paulawalla@yahoo.com with "Move Account" in the subject and including the following info:
1. Your domain name
2. Your domain name login (if not held here) including Registrar name, user id and password
3. A list of your email accounts and passwords
4. If you have a copy of your website or not

Pop Email settings for the new server are:
Mail Server Username: username+yourdomainname.com
Password: the password you choose
Incoming Mail Server: mail.yourdomainname.com
Outgoing Mail Server: mail.yourdomainname.com (server requires authentication)

I personally am choosing NOT to move the HostingCT.com website or email.

Thank you very much for your patience. My sincerest apologies for this incredible delay.

Sincerely,

Paula M Pierce
Tel: 860-683-4201
Fax: 860-298-9617
Offsite email: paulawalla@yahoo.com
Alt website: www.HostingCT.net

November 6, 2007
5:13 pm

I just got off a conference call with Navisite (they've been having conference calls every 2 hours to keep hosts informed) and they remain reasonably confident that tomorrow morning they'll have the last of the networking issues worked out, meaning we should see sites and email come up by then.

I do believe they are doing everything possible to get every last website back up and running as soon as possible.

I'm so sorry for this lengthy downtime.

PS - we're trying to keep up with all the phone calls, voice mails, and emails. I apologize if I have not been able to contact you personally yet.

November 6, 2007
1:00 pm

Update from the Data Center:

Our two primary focuses currently are to bring all of the name servers back online as well as collaborating with Cisco to correct high levels of ARPs on the network. As more hosts were being brought on line, a significant number of ARP requests were overloading the network, so we were tasked with resolving stability issues.

Overnight, we were troubleshooting and resolved name server issues many more hosts were being brought on line. This required a network configuration change in the midst of resolving specific DNS issues.

We have continued to hand off completed environments to clients at a much faster pace. We are doing this across multiple environments so that we are satisfied that standards for Quality Assurance (QA) have been met. Routing issues were resolved by assigning dedicated engineers on a client-by-client basis to support the process.

A full complement of engineering resources have been committed to completing the overall migration and have been working around the clock to resolve overall service issues and to resolve specific problems. We did have some improved service-level commitments in mind

November 6, 2007
10:00 am

The data center techs are reporting many sites coming online, but I haven't seen that yet.

I am now in the process of setting up a new server unrelated and completely unconnected with the previous servers. I am going to start moving email accounts there.

If you would like to have your email accounts moved, please email us the following information:

1. Your domain name
2. Your domain name login (if not held here)
3. A list of your email accounts and passwords
4. Your off-site email address

Please note:
- changes to your domain name record take 24 - 48 to fully propagate although you could see some email start flowing earlier.

Thank you very much for your patience. My sincerest apologies for this continued delay.

November 6, 2007
8:00 am

Update from the Data Center:

As we were bringing more and more hosts on last night, we were creating a significant number of ARP requests on the network. The number of ARP requests is primarily due to the fact that we were still resolving name servers issues in part of the environment. The number of ARP requests ultimately overloaded the network which is why you were seeing instability.

In the process of troubleshooting the over utilization issue with Cisco, a routing error was made which has caused most of the environment to be inaccessible. We are working to reverse this change out with Cisco and will then work both the name server and ARP'ng issue.

November 6, 2007
7:00 am

As they continue to work to resolve the problem, there seems to be little information forthcoming. I rejoined the conference call with Navisite again at 6:30 am and will stay on until I can get some kind of update. The other webhosts on the line are in the same boat and each one has a minimum of 300 clients. I apologize for this extended downtime. I've never seen anything like it.

November 5, 2007
10:45 pm

Update from the Data Center:

We are making continued progress in troubleshooting specific issues and bringing up individual hosts - as we fix them. The root cause of the DNS issue that we have been experiencing - and which has prevented us from bringing up the additional hosts - has been identified.

We will be making a network configuration change at 11 p.m. EST that will take approximately 15-30 minutes to resolve the issue. We expect this will solve both email and name resolution issues that have been identified, and we will continue our troubleshooting efforts

Notes from Paula:

- I'm sorry these updates have not been all that helpful. It's the best info we're getting.

- When your site starts coming up and email starts flowing, please note that if you have email quotas set, you'll likely get 'overquota' notices pretty quickly and will want to delete mail quotas till you can download your mail.

November 5, 2007
6:00 pm

There has been some additional progress on the server migration. Official word from the data center appears below:

"Thank you for your patience as we work diligently to complete the migration. We now have over 90% of the server operating systems up and running. The network core is stable and we are now primarily focused on two key areas:

- Performing extensive quality assurance testing throughout the servers to validate the configurations.
- Continuing to work on the name servers to enable the availability of the individual hosts."

November 5, 2007
2:40 pm

I have heard that some clients are receiving emails sporadically. Please note, we are trying to return all phone calls and emails as quickly as possible but may not be able to respond until the rush is over. Please keep an eye on this site for updates. If you have an urgent need, please email paulawalla@yahoo.com. Thanks. - Paula

---

November 5, 2007

As of 2:00 pm Monday, our servers are still in the process of coming back online. The data center techs are reporting many servers coming up but that DNS issues are causing delays (ie. the sites/email are not accessible on the internet yet). I don't have a firm estimate at this time, but I'm expecting sites to come up later today.

For more details on the server migration, please go to the Navisite website. (Navisite purchased the company we lease our servers from.)

As soon as I get any new information, I will post it here and send the info by email.

Thank you very much for your patience. My sincerest apologies for this continued delay.

November 5, 2007

As of 6:45 am Monday, we are still down. We're still hoping to be up and running at some point this morning.

I apologize for the inconvenience I know this is causing. This server migration has affected not only Hosting CT and our customers, but also hundreds of other web hosts and over a hundred thousand websites.

The latest update from the data center is that the network traffic congestion issues have been resolved which has allowed a great number of sites to come up and be accessible via the internet. Some sites/domains are still not available due to DNS issues. Engineers are troubleshooting this issue currently.

As soon as I get a new update, I will post it here and send the info to your off-site email addresses. If you have an off-site email address (yahoo, hotmail, cox, etc) and did not get my email on Sunday, please send an email to paulawalla@yahoo.com with your off-site email address and I'll add you to the list.

Thank you very much for your patience. My apologies for this continued delay.

November 4, 2007
10:40 am

Status Update:

As you may have noticed, our servers are currently experiencing some down time.

The data center migration we mentioned a few weeks ago began on Friday night, November 2nd at 10 pm. That move did not go as smoothly as we'd expected and we started experiencing some longer than anticipated down time yesterday morning. We are doing everything possible to get all servers back online by Monday morning.

I apologize for any inconvenience this may be causing. As new information becomes available, we will keep this site updated and send email to your off-site email addresses.

Contact Info
Contact: Paula M Pierce
Tel: 860-683-4201
Fax: 860-298-9617
Offsite email: paulawalla@yahoo.com