Service interruption 2008.07.14 (UPDATED 2008.07.15 09:31PST)

UPDATED 2008.07.15 09:31PST

Our voice servies including certain inbound services depending on Allstream voice traffic seem to be restored. Please report any continuing problems immediately.

We will continue to follow up with all related providers in order to ascertain what happened and what will be done to avoid a recurrance.

As you know yesterday Bitblock customers experienced a loss of service throughout most of the Pacific Time business day.

This was a direct result of a catastrophic fire at a BC Hydro facility in downtown Vancouver. Links to related articles are posted on our site and we will continue to post updates to this story and our resulting actions and plans.

http://www.bitblock.com/news/network/32-service-interruption-2008-07-14

Bitblock houses it’s primary infrastructure at Western Canada’s main communications hub at Harbour Centre, and has taken several precautions against interuption of services which include:

• Making sure our equipment is served from three separate city power grids

• Multiple backup generator power

• Multiple backup battery system (UPS).

A rough chronology of the events is as follows - note this account is based on various official and unoffical information sources and may be updated as more accurate information becomes available.

• A fire and/or explosion in a power substation below stree level at Pender and Richards knocks out power

• Harbour center switches to generator more - the systems run on battery while the generators start up

  • Note the generators themselves are water cooled

• Fire crews arrive, and hydro de-energizes other power to enable the firefighters to work on the fire

• In combatting the fire, water is diverted to pump trucks disabling buildings city water systems

• Generators overheat and have to be shut down

• Essential systems and work arounds were started to repair services affected in other cities as far away as Toronto resulting from the outage

• During the UPS run time, systems were shut down as fast as possible to preserve data integrity

• Repairs to generator systems were effected and generator power was restored (approx 1630)

• BC Hydro restores partial service to harbour center (approx 1800) due to it's top priority

• Repairs continue, much of vancouver's downtown core remains without power, however all our systems and networks appear to be restored to full service (approx 1645).

Disruptions noted during this event affected many users beyond businesses located in the core itself. Hosting companies as far away as Toronto reported service issues, and many ISP's (Cable, and ADSL) seemed to experience disruptions at this time.

In the aftermath of this event we are carefully analyzing our options to prevent a re-occurrence of yesterday’s outage, and will be elevating the priority of our existing plan of action to add more fail-safe’s to our system including a second service location in Eastern Canada.

I would like to take this opportunity on behalf of Bitblock Systems and myself to apologize for yesterday’s service interruption and any problems it may have caused. Rest assured our team is working to prevent a repeat occurrence of yesterday’s events.

Thank You for your patience. More updates and the history are available under the "read more" link below.

UPDATED 2008.07.14

After one false start, power, cooling and connectivity services have been restored and we are working on testing systems integrity and notifying customers. Please report any continuing problems immediately.

We will continue to follow up with all related providers in order to ascertain what happened and what will be done to avoid a recurrance.

Thank You for your patience.

More information / news is available at the following sites:

Canada.com

Canada.com / MontrealGazette

CBC - Link Unavailable