MTS Allstream Failure in Western Canada 2008.04.09 13:47PST (UPDATED 2008.04.10 00:49PST)

UPDATED 2008.04.10 00:49 PST

Our voice services including certain inbound services depending on Allstream voice traffic seem to be restored. Please report any continuing problems immediately.

We will continue to follow up with Allstream in order to ascertain why a single fiber cut had such far reaching effects and what will be done to avoid a recurrence.

Today our main carrier for inbound services, MTS Allstream suffered a fiber optic failure which resulted in much of its national network suffering a service loss. This outage affected thousands of customers across Western Canada and seriously impacted their nationwide network. As a result of this failure inbound phone service on the BitBlock network was severely affected from 12:30 pm to 6:20 pm. Unfortunately for inbound numbers we must rely on a national Canadian carrier for termination, and are therefore affected by major outages such as this one.

We are happy to report however that service has been completely restored and tested. Another positive note is that as a VoIP service we where able to reroute your outbound calling away from the trouble to other providers, which can not be said for thousands of traditional telecom customers with traditional land lines.

Thank You for your patience. More updates and the history are available under the "read more" link below.

UPDATED 2008.04.09 23:00 PST

Allstream has updated their website, and the CBC has a brief an somewhat inaccurate story posted (read more below) on their website. We have attempted to make them aware of the errors in the story, if we receive response we will add it or reference it in this story.

The MTS announcement on mts.ca states:

Earlier today, our network was impacted by a fibre-cut in the Winnipeg area. The outage, which began at approximately 3:30 PM eastern time/2:30 pm Central time has been resolved and voice traffic has been restored as of 9:20 pm Eastern time/8:20 pm Central time.

Some local voice services remain impacted within the Winnipeg area. Work continues on restoring these services. and it is expected they will be repaired overnight.

We found this link which contains an admission of local access effects in Western Canada:

http://www.newswire.ca/en/releases/archive/April2008/09/c4249.html

WINNIPEG, April 9 /CNW/ - The MTS Allstream network has been restored as of 8:20 pm central time. The outage, which began at approximately 2:30 pm central time, was caused by a contractor fibre cut in the Winnipeg area affecting a portion of our national fibre network. This had been primarily impacting our local voice services in greater Vancouver, Calgary and Edmonton areas and some long distance voice traffic on the MTS Allstream network across Canada. These services have been restored and calls are now moving as normal across the network. Some local network components remain impacted within the Winnipeg area. These are expected to be repaired overnight.

The CBC has an article on the website which contains inaccuracies in our opinion - we posted a comment / feedback to their site, but received a server failure during the process - we repeated the attempt and as we received a "Thank you" page we can only hope the comment was received. The original article is located at:

http://www.cbc.ca/cp/technology/080409/z040917A.html - link unavailable

Here is our post:

The Allstream article posted at http://www.cbc.ca/cp/technology/080409/z040917A.html - link unavailable contains several inaccuracies, and doesn't seem to allow comments.

It [seems to be] essentially a quote of information released by Allstream but appears to have been neither fact checked or researched. I'd appreciate a response to this comment as I'm becoming a little frustrated with mainstream medias inaccurate handling of technical matters. The inaccuracy will be noted in our blogs along with any response we receive in fairness to CBC and it's sources.

1) A cable cut should NEVER cause a large scale outage like this - telecom systems generally run loop based or otherwise fault tolerant technologies to allow a single failure to be tolerated without significant disruption.

2) The scope of the failure is wrong - it also appears to have affected customers with Allstream local service i.e. phone lines and fax lines in the Vancouver and other areas as well as 1800 service and other services. When these customers could not dial OUT it is reasonable to presume they could not dial 911 either.

We could ourselves see and confirm that people in Vancouver could not reach other people in Vancouver due to this outage in the simplest of business systems - a standard T1 line which is a normal multi-line business communications product.

Currently repairs seem to have been completed - at least enough that services we can test have been restored, although we never received an update or any information indicating progress from Allstream.

We also found it noteworthy that while there was no definite explanation given on the support lines, the information displayed on the mts.ca home page was formatted in a way possibly intended to PREVENT it from being picked up by search engines - the text of the message was displayed in a graphical form.

For your reference, our company was one of the first if not the first to notice and report the Rim / blackberry outages which CKNW seemed to take an interest in reporting and researching in detail.

Thanks. I hope this helps.

UPDATED 2008.04.09 19:34 PST

NOTE: For the duration of the Allstream phone service outage, BitBlock can be reached at 604.484.9374. In keeping with BitBlock's high-availability philosophy, we will be looking into service options which will facilitate alternate carrier inbound numbers for our clients in the unlikely event of a recurrence of this type of event.

(UPDATED 2008.04.09 16:36 PST)

As of 2008.04.09 16:36 PST, a notice on the mts.ca home page states:

NETWORK OUTAGE: The MTS Allstream network has been impacted by a fibre cut in the Winnipeg area, which has impacted certain cross-Canada voice traffic. Specifically, long distance voice traffic our of Toronto, Vancouver, Calgary, Edmonton, and Manitoba may be blocked by this incident. Work is being done to repair the problem as soon as possible. Further updates will be provided as soon as they are available.

As a comment on this post we find the following points interesting to note and are hopeful more information will be forthcoming...

1. The information was posted in a format which will be difficult or impossible for many search engines to detect - making it unlikely to be widely distributed in our opinion
2. There has been no official press release we can find
3. A single fiber cut should not disable any modern loop or other highly available system
4. The service interruption is affecting local traffic as well as long distance at least in Vancouver

Original Post 2008.04.09 13:47 PST

MTS Allstream is reporting an outage affecting Western Canada (BC and Alberta at least) with some reports indicating the problem may be further reaching than that.

Phone service to analog customers appears (from reports) to be affected both outgoing and incoming.

Our customers will note that outgoing calls have been routed around the disruption, allowing calls to be placed as normal, however incoming calls may terminate with either a fast busy, silence or endless ringing.

Allstream ticket number GQ093

More updates will be posted as information is available.