City 911 disabled, backup also fails

Lawrence County also out for several hours

Monday, April 29, 2002

By Steve Twedt, Post-Gazette Staff Writer

An apparent computer problem at a Downtown switch connection, plus a failure in the backup system, disabled 911 service in Pittsburgh and Lawrence County for several hours yesterday afternoon.

Residents in those areas making emergency calls between 2:30 and 6 p.m. got a dead phone line, while officials scrambled to get word out through local news media and police stations that they should use an alternate 10-digit phone number. The alternate number could not provide 911 operators with the location and identification information they would normally have.

While both 911 centers typically receive a steady influx of calls during the day, emergency officials were not immediately aware of any missed calls of a life-threatening nature during the failure. The Lawrence County center did receive two calls about house fires from cell phone callers.

"Machines break, and that's what happened," said Frank Jannetti, Lawrence County's 911 director.

"The whole issue is, if the system does break down, what do we do? We've been assured by Verizon that we have a backup system in place. We've been assured that several times. It's just frustrating, that's all."

A similar breakdown two years ago prompted Verizon, which maintains the system, to change the way it rerouted emergency calls. Now it's trying to determine why it happened again.

"I'm not sure what failed today. I'm still trying to get information from the technicians," said Lee Gierczynski, Verizon spokesman.

The breakdown occurred just as a major storm passed through the region, but the high winds and electrical outages did not cause the 911 failure, Gierczynski said.

"Luckily, the city wasn't hit that hard [by the storm]," said Hank Caparelli, emergency operations coordinator for Pittsburgh, who noted that the city's emergency center typically gets its lowest volume of calls on Sundays.

Allegheny County's 911 system was unaffected, said Bob Full, director of emergency services. The main center in the Lexington Technology Park, North Point Breeze, and the regional centers were all operating normally, and, in fact, many Pittsburgh 911 calls are automatically switched to the county in the event the city system fails, he said.

For example, Wilkinsburg calls go to the county, and calls from nearby Homewood that go through the same call center would also go to the county, he said.

"We believe we were handling about 60 percent of their calls," Full said.

Pittsburgh Public Works Director Guy Costa said the problems at the city's 911 center had some impact on his crews' abilities to respond to weather-related problems.

"It took them awhile to notify us. We were a little slow to react."

Jannetti said yesterday's severe storm forecast had prompted the center to double staffing. Luckily, the worst parts of the storms missed Lawrence County while the system was out of commission.

"We fully expect a machine will go down eventually, but we expect the backup system will work, and this backup system has never worked," he said.

Incoming emergency calls for Pittsburgh and Lawrence County are routed through a computer system located on Seventh Avenue, Downtown, using what's called a Rockwell switch. The system is more than 15 years old and Rockwell no longer makes them. Lawrence County, in fact, is already in the process of replacing that system and plans to have a new one in about three months.

Gierczynski said, "You can't attribute [the failure] to the fact that it was old. It's fundamentally fine as a 911 system."

But he did acknowledge the failure of the backup system is a major worry. That failure was in system software that should have taken over switching the calls when the Rockwell switch failed, he said.

"We're very concerned that it didn't work the way that it should have," he said. He added that the company will bring its technicians together in a conference call today to figure out what happened.

Post-Gazette

-- Anonymous, April 29, 2002

Answers

911 system failure tied to computer programming error

Human error caused the backup 911 system to fail Sunday in Pittsburgh and Lawrence County, Verizon officials said yesterday.

An undetected programming error kept the backup system from switching calls into 911 centers after the main system crashed, leaving the city of Pittsburgh and Lawrence County without 911 service for more than 3 1/2 hours Sunday afternoon.

The primary routing of 911 calls is done by a computer called a Rockwell switch, located Downtown. A power card on that computer blew out Sunday, causing the main system to crash, said Eric Rabe, vice president of media relations for Verizon. The city and Lawrence County contract with Verizon for 911 switching services.

The backup system is part of Verizon's overall switching for all phone calls, and a programming error that had gone undetected caused that backup system to be unable to recognize where 911 calls were coming from.

Because it lacked that data, the backup system treated those calls as errors and did not switch them to the 911 call centers, Rabe said. That was the cause of the failure that made it impossible for many callers to get through to 911 and left officials struggling to alert people to call a 10-digit emergency number. Officials do not believe that they missed any crucial calls during the period the system was down.

The programming problem has been corrected, as has the hardware problem that caused the Rockwell switch to fail, Rabe said. Both systems have been tested.

Verizon is continuing its investigation and still does not know who made the programming error, or when it happened.

"What we do know is that some changes should have been programmed in as part of upgrades and changes in system," said Rabe. "We're constantly upgrading, and there should have been some changes made to the programming of that backup switch that were not made."

The error could have been made any time over last year or so, he said.

To make sure it doesn't happen again, "we have been in touch with the city and Lawrence County to advise them of what we found here and we've proposed that we begin routine regular testing of both the primary and the backup system."

He said such testing is disruptive to the 911 centers.

"We had not felt it was necessary, but believe me, now we do."

"We are frankly surprised this could have happened," said Rabe. "We are doing an investigation now of not only what happened in Pittsburgh, but we are looking at systems in other parts of the country" to make sure that there aren't similar problems there.

"The failure to make programming changes was obviously a failure on our part. We realize 911 is a vital emergency service, and we take that seriously."

In the past, the city and Allegheny County have entertained the idea of merging their respective 911 call operations. The county maintains its own 911 system in Point Breeze and has an off-site backup system.

However, a spokesman said that Pittsburgh Mayor Tom Murphy is focused on solving Sunday's problems, which he blamed on Verizon, and is not interested currently in a 911 center merger or co-location.

"The mayor is focused on determining what caused Verizon's system to fail, and his communications staff is discussing the issue," spokesman Craig Kwiecinski said.

The city and county have moved their emergency operations centers together at the county's location in North Point Breeze, partially in response to September's terrorist attacks, Kwiecinski said, but there are no talks currently planned on other co-location plans.

Kwiecinski said that the 911 centers are "not the issue right now. We are focusing on why this happened and preventing it in the future."

Yahoo!

-- Anonymous, May 02, 2002