2012.09.24 Peering issue / packet loss

2012.09.29 1805 Issue resolved.

The issue was confirmed to stem from a failing switch port within the Vancouver peering center. The port was issuing intermittent errors which was causing traffic using that link to momentarily be rejected while the error resolved. Once the link was rebuilt the issue has not been seen to recurr.

2012.09.25 1830 No loss observed for over 3 hours.

At this point we are cautiously optomistic that the problem has been resolved. Clients with open tickets will be advised of this change of status. We will continue to monitor and will post a detailed explanation as soon as we receive one. Thank you to all who posted detailed service reports - details such as the time of interruption were essential in determining the scope of the issue.

2012.09.25 1505 Emergency maintenance COMPLETE.

Normal route has been restored. We will continue to monitor the situation throughout the night.

2012.09.25 1425 Emergency maintenance starts.

Traffic has been routed away from the affected link to allow repair of the issue. Latency is increased, however packet loss seems to be eliminated.

2012.09.25 1300 Vancouver peering operation center confirms problem.

After raising the issue with multiple carriers / operation centers we were able to assist in determining the location of the problem. A this time we do not have a confirmed explanation of the actual cause, but it seems that a route in BGP is being recalculated / re-evaluated resulting in momentary loss of connectivity as traffic is re-routed around the perceived failure - the failure is NOT real however - which results in another re-route back. Emergency maintenance will be performed this afternoon.

2012.09.24 Peering issue / packet loss confirmed.

A routing issue has been detected originating in the Vancouver PIX. Symptoms of this may include occasional disconnections from Email or RDP services and intermittent audio loss (both directions) on VoIP calls. We have started tickets with the various service providers and interrconnects, and are following up to push for a resolution or work around.

This issue does not involve our equipment or network components under our control but seems to affect a wide variety of services.

We will post updates as they are available.

Thank you for your patience.