"Terrifying" 911 failure exposes deeper flaws

On 9 April 2014, a software error prevented emergency calls from getting through to eighty-one 911 call centres across seven US states. More than 11 million people lost 911 service for six hours.

The error happened in a 911 call-routing facility in Englewood, Colorado. The 81 affected call centres - formally called public safety answering points, or PSAPs - were spread across California, Florida, Minnesota, North Carolina, Pennsylvania, South Carolina and Washington.

During the six hours of the outage, more than 6600 emergency calls failed to get through.

Fortunately, there was no loss of life as a result of the problem.

During a briefing streamed online to coincide with the release of the report into the failure, Federal Communications Commission (FCC) Chairman Tom Wheeler described the situation as “terrifying”.

“We are used to thinking about 911 outages as a result of acts of nature - a hurricane, a tornado, vast flooding - that damage local equipment with localised impact,” said Wheeler in a statement accompanying the report.

“But there is a new threat. The emerging Next Generation 911 system is more complex than the legacy 911 system and relies more extensively on infrastructure, resources and relationships that are multistate or national in scope. It is supported by a larger number of service providers, including new entrants that are offering new, niche functionalities.

“Innovation is good, and we want NG 911 to support new forms of emergency communications,” added Wheeler. “But the creation of new, complex technologies with a broader universe of operational  relationships also has introduced new potential vulnerabilities that need to be addressed if we are going to  have a reliable and resilient 911 system.

“The outage was not caused by a storm or disaster. Nor was there a failure of the local telecommunications provider’s network. The outage was traced to a preventable technical problem in a third-party vendor’s equipment used to route 911 calls on behalf of the primary telecommunications provider. We have seen similar problems in other instances.”

Not an isolated event

“What is most troubling is that this is not an isolated incident or an act of nature. So-called ‘sunny day’ outages are on the rise,” the report says. “That’s because, as 911 has evolved into a system that is more technologically advanced, the interaction of new and old systems is introducing fragility into the communications system that is more important in times of dire need.”

The report reveals that “the outage was caused by a software coding error that prevented 911 calls from being processed timely and directed to the appropriate PSAP. It could have been prevented. But it was not.

“The causes of this outage highlight vulnerabilities of networks as they transition from the long-familiar methods of reaching 911 to IP-supported technologies. In particular, the technical and operational failures that caused and prolonged the outage suggest the need for a close examination of the transition to IP-supported 911 services.”

The report goes on to say that Next Generation 911 (NG911) networks, “which rely on IP-supported architecture rather than traditional circuit-switched time division multiplexing (TDM) architecture, introduce promising new capabilities, such as more flexible call routing and the ability to provide PSAPs with a greater range of information (such as video). At the same time, however, they can also introduce new vulnerabilities and challenges.”

The report says that call control in legacy 911 networks was primarily performed in a central office switch that was close to the customers it served, whereas IP-supported networks increasing rely on geographically remote servers and software-based components to support key 911 functions, such as 911 call routing, across multiple states and jurisdictions. Consequently, a 911 outage in an IP-supported network has the potential to affect a much greater number of PSAPs and people, across multiple states, as demonstrated by the multistate effects of the April outage.

“This outage also highlights the ongoing trend among communications providers and PSAPs to ‘contract out’ 911 service functions to third-party vendors,” says the report’s executive summary. “This has concentrated critical functions in fewer locations that are more distant from the PSAP and the end user, and created a corresponding need to ensure such contractual arrangements do not compromise situational awareness and accountability for the end-to-end 911 call-to-completion process. Redundancy and responsibility are both endangered.

“The introduction of NG911 and IP-based technologies will require industry as well as state, local, tribal and territorial governments and commissions to move aggressively to ensure that technology enabled optimisation does not introduce unacceptable risks that threaten imperilling 911 reliability and resiliency. Everyone has a role in ensuring that 911 works as it should, when it is most needed.”