Saturday, February 6, 2010

Google Calendar Server Error and Manual Dashboard –- Chronicles of February 5, 2010

About a year ago when I had a problem with my GMail, I stumbled upon the Google Dashboard which kept the world updated on statuses of Google Apps.  At that time, GMail’s status indicated that Google knew about the problem and was working on it.  I couldn’t check email, but I was comforted to know that some of the smartest people in America knew about the problem and were working on it.

February 5, 2010 - 6:00am

Today began like a normal Friday for me.  Alarm clock goes off at 6:00am.  Turn it off.  Wake up in a panic at 6:40am realizing I’ve fallen back asleep.  Shower.  Run off to a 7:00am breakfast with past co-workers.  8:10am, roll in a little late to work.

8:10am

Then something new happened.  I got an email about a house party next weekend that I wanted to add to my calendar.  I opened my Gmail browser tab and clicked GoogleCalendarError-Feb-5-2010on the “Calendar” link… SERVER ERROR in a dozen languages.

Odd.  This is Google.  I assume this is probably a temporary glitch while Google’s automated disk farms fix themselves.  I imagined hundreds of PC disk drives spinning furiously, reloading/indexing my calendar data on it.  Assuming that my data may’ve moved around in the clusters, I figured I probably need to follow the instructions about cleaning my browser cache to get updated IDs to my data.  I try again.  No luck; same worldly error.

9:00am

In between getting some work done, I was searching Twitter for possible details about the problem.  Was it only affecting me?  Should I start backing up my GMail now because Google was being attacked?  There were plenty of Tweets saying that their co-workers had access to the Calendar just fine.  So this wasn’t a global issue.  But at least I found confirmation that others were having problems.

9:43am

After 30 more minutes of looking for confirmation of others with this problem on Twitter and not finding a solution, I decided to post to the Google “Calendar not loading” Help forums.  When I entered my problem in the subject area, an automated search presented me with a list of similar topics.  They all were from a couple of months ago.  It appears that I may be the first person reporting on the forum, so I posted my issue.  Minutes later, my comment was updated by another Google user who indicated that another thread was already collecting posts from everyone who is having a service outage today.  I wish the automated search results that I got while posting my issue had included this more recently relevant post.  Regardless, now I found a thread of users that I could collectively share experiences with.

11:08am

In between more work, I did some more hunting around through the Google Help links; I wanted to find a phone number or an email address.  Most of the time, I ended up in a loop and would end back up where I started.  But finally, I found a link that allowed me to report my issue.

11:08am – 5:25pm

I kept on monitoring Google’s Dashboard and the Calendar Forum thread for updates on this issue.  The forum continued to get updates from people complaining about losing all of their calendar entries on their mobile devices.  However there was also a mixture of users who said their coworkers can still get in just fine.

As the hours went by, it was frustrating to watch Google’s Dashboard sit unchanged at “No Issues”.

5:25pm

Google finally updates their Dashboard indicating a service disruption.  Did they really just now detect the problem?

CroppedGoogleCalendarDisruptionDetails-Feb-5-2010 

9:14pm

I get into my Google Calendar for the first time today. 

 

 

So What Happened

The details on the Dashboard indicate that this Service Disruption affected 0.03% users.  Wiki says GMail has about 146 million users.  0.03% * 146,000,000 =  43,800 users affected.  And I’m sure an even smaller percentage of those are daily Google Calendar users.

I don’t know when the problem was first identified.  I do know that my calendar worked Thursday night.  Here are my own calculated stats of the day.

  • ~10+ hours before Google says they detected a problem
  • ~3.5 hours to fix the problem
  • ~43,800 people late for or missed appointments today

 

Takeaways from this experience

- Google’s Dashboard is not fully automated.  I assumed that Google App service issues would be self-reporting and would trigger alarms when there were problems.  It appears that a human has to trigger status changes on the Google Dashboard.

So is the Google Dashboard a misnomer?  A dashboard is supposed to be an automatic reporting system.  I sense that Google’s Dashboard really needs to be re-named Google Statusboard.   

This was disappointing to find out.  I felt helpless for almost 10 hours since there was no indication at Google that there was a problem.  I also had to wonder what I would do next if a day went by and I still couldn’t get into my Calendar.  Calendar is less critical for me; what if this had been my GMail!?

 

- How does someone report a problem about a Google service?  It took many different approaches in the Google Help system to actually find a link that had a webpage to report this problem.  But even after submitting that issue, Google’s Dashboard did not updater recognizing the problem until 6.5 hours after I submitted my problem.

Considering that there is no advertised way to get one-on-one support from Google, the only approach to reporting issues is a shotgun approach. 

  1. Post to Google forums.  But first check in the forum’s recent entries to see if you can reply to an existing thread rather than start your own.
  2. Search and post to Twitter.  At least search Twitter.  Verify that there are others with the same experience.  That will be good supporting evidence to ensure that the issue isn’t a local firewall or routing issue or just an issue with your web browser.
  3. Submit your issue using the Report an Issue form in Google’s help labyrinth.  Once you find the page to report the problem, make sure to reference the Google Forum thread and Twitter search URL.  That will support your issue and provide proof that you have done research into the problem.

No comments:

Post a Comment