PG&E Blasted By CA Public Utilities Commission Over Errors That Caused San Francisco Blackout --140 Mistakes Prolonged Blackout By Seven Hours

More shades of Y2K, truth or consequences?

Could it be that Silicon Valley IS in Y2K "trouble" too? With PG&E's blackout track record, what do you think?

*Big Sigh*

Diane

PUC Blasts PG&E Over Errors That Cut S.F. Power
State probe says 140 mistakes prolonged blackout by seven hours
Rebecca Smith, Chronicle Staff Writer
Thursday, April 1, 1999
)1999 San Francisco Chronicle

http://www.sfgate.com/cgi-bin/article.cgi?file=/chronicle/archive/1999/04/01/MN73911.DTL

In a blistering report on the blackout that crippled San Francisco in December, state investigators yesterday accused PG&E of committing 140 blunders that prolonged the power failure by several hours.

They also found evidence of a coverup by workers to deflect blame.

Substandard work practices and equipment breakdowns at Pacific Gas and Electric Co. combined to create a utility with an ``error prone work culture,'' says the report issued late yesterday by the state Public Utilities Commission.

In addition, information received by The Chronicle indicates that PG&E executives tried to get the PUC to back down from its investigation, challenging the agency's authority to conduct an inquiry.

The PUC investigation, which took three months, is far more ambitious in scope than two other reports on the subject. Its goal: to get to the bottom of what happened on December 8 and see what changes are needed in work procedures or system design to prevent future disturbances.

PG&E executives dismissed the investigative team's major findings as ``flawed.'' In a letter sent yesterday to the president of the PUC, PG&E chief executive Gordon Smith said the agency should reject the ``unsupported, broad conclusions'' of the report and work more closely with a quasi-public agency, the grid-operating Independent System Operator in Folsom, to help PG&E sort out which ``if any'' recommendations should be implemented.

Major points of the report:

-- Scores of workers committed mistakes that not only caused the power failure but delayed service restoration. Some mistakes were committed years earlier but did not show up until the system was under stress. Errors included misdiagnosis of problems, improper maintenance of equipment and unauthorized shortcuts in procedures.

-- A critical log in the control room of the San Mateo substation where the problem surfaced may have been falsified to make it appear action was taken to protect the system from collapse.

-- The blackout reflects systemwide problems at PG&E that have resulted in a 400 percent increase in switching errors since 1997.

-- The blackout, which lasted seven hours and 39 minutes, cost the local economy as much as $400 million. But making repairs to prevent similar power failures could cost as little as $10 million.

-- Another major disaster is inevitable if PG&E does not overhaul its work practices.

Until now, only two major mistakes had been publicly identified. PG&E said a construction crew failed to remove grounding rods at the San Mateo substation after installing a new transformer.

That mistake, which caused a huge electric fault, was compounded by an error in the substation control room when an operator failed to reactivate equipment that is designed to confine faults to the substation.

As a result, when the switch was thrown, electricity bypassed four 115,000 volt lines that supply power to San Francisco and instead coursed down the unremoved grounding rods and went into the mud. Only one line stayed energized. Two million people lost power in San Francisco and parts of the Peninsula.

Investigators said they found a host of errors. ``In my opinion, they were all significant,'' said Chong Chiu, an independent consultant and world-renowned expert in disaster analysis who was hired by the PUC to lead the investigative team.

For example, critical pieces of equipment were not properly installed or maintained. More than half the breakers inspected by Chiu's team showed signs of aging, such as thickened grease. Twenty- one of 700 breakers failed outright.

One breaker, the No. 612 breaker at the San Mateo substation, had been wired incorrectly in 1991. A crew labored 45 minutes to get it reset because, without it, they could not restore power on a 115,000-volt line.

Chiu's team also found that PG&E employees ignored written operating procedures, which created confusion and slowed recovery. A 1990 management directive instructs employees to ``repeat back'' orders and important communications.

``Throughout seven hours of the outage, nobody repeated back any information that I could find,'' said Chiu.

PUC enforcement staff are not seeking fines or penalties. Bill Schulte, head of the Consumer Services Division that commissioned the independent inquiry, said the next step is to ``sit down and see what needs to be done to implement improvements and prevent another outage.''

PG&E executives said they think investigators took a few isolated problems and exaggerated the impact.

``San Francisco reliability is extremely high,'' said Jim Macias, senior vice president of transmission for the utility. ``One event doesn't indicate there are systemic problems.''

The report also said there are design flaws that must be corrected. The PG&E transmission system serving San Francisco is designed to protect the supply of electricity to the downtown business district in the event of a power failure to the rest of the city.

But the system is currently set up primarily to guard against frequency fluctuations and not the voltage fluctuations that caused the December 8 blackout. ``That means San Francisco is wearing a suit of armor but there's a hole over its heart,'' said Chiu.

PG&E and the System Operator disagree with Chiu. They say his recommendation could lead to more minor power failures.

One obstacle, the report found, is that PG&E has a culture ``that is not self-critical and forthcoming with problems.''

As investigators identified problems, they met resistance from PG&E employees, team members said.

PG&E executives disagree. ``We bent over backwards to help everyone who wanted information,'' said PG&E's Macias.

There is evidence, though, that the utility attempted to use another regulatory agency, the grid-operating System Operator, to intervene in PG&E's behalf, assert authority over the case and force the PUC to back down.

Private e-mail exchanges between PG&E and a manager at the grid-operating System Operator, obtained by The Chronicle, show that the utility urged the System Operator to assert its jurisdiction.

In fact, both the PUC and the System Operator share responsibility for reliability of the state's electric system.

E-mail messages show PG&E asked the System Operator whether it needed to answer Chiu's questions. An System Operator employee, who used to work for PG&E, said the System Operator ``does not require PG&E to respond to transmission-related questions from Dr. Chiu.''

EXCERPT FROM PUC REPORT

The log entry allegation

A switching log at the San Mateo substation shows the steps that were taken when operators attempted to switch on some equipment at a portion of the substation where problems started that cascaded into a huge blackout December 8.

The log shows a critical step -- rearming protective devices designed to contain problems -- was added by hand.

Investigators believe the log here was falsified when an entry was made after the power failure to make it appear that the protective system was working just prior to the power failure.

But PG&E says it examined the log and interviewed workers and feels the evidence is inconclusive. It's currently in the process of disciplining 13 workers.

-- Diane J. Squire (sacredspaces@yahoo.com), April 01, 1999

Answers

Believe me .... I WISH this all was a joke!

Unfortunately, PG&E is our local April Fool's joke.

*Another Big Sigh*

Diane

Cant find the report up yet at ...

California Public Utilities Commission

http://www.cpuc.ca.gov/

See also thread (for a historical perspective) ...

Power OUT In San Francisco

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id= 000IIh

-- Diane J. Squire (sacredspaces@yahoo.com), April 01, 1999.

And they don't even mention the dude selling his homemade bombs to the other employees from his locker .... guess that's too 'normal' to fault.

-- Leska (allaha@earthlink.net), April 01, 1999.

.....and yet, I keep reading those three lil words: FIX ON FAILURE!

God help us all

-- Sheila (sross@bconnex.net), April 01, 1999.

FUCKED on failure is more like it.

-- fof (fof@wont.work), April 01, 1999.

This is why I expect....irregular outages over irregular areas affecting unknown levels of services and supplies due to unknown causes that result in unknown amounts of failures in unknown numbers of other systems that will affect ....

Things will recover - I just don't know when, nor which areas will be affected by what failures for how long. I can't predict what individual reactions to the failures will be, what the group reaction will be, and what society's reaction (at large) will be. Probably different in each different region in any case.

And I hope I'm smart enough to realize I don't have the exact answer, but am wise enough to prepare adequately for the uncertainity.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 01, 1999.

So you see, Bob, the polly's have it partly right. Paraphrasing, they're predicting

"...irregular successes over irregular areas affecting unknown levels of services and supplies due to unknown causes that result in unknown amounts of successes in unknown numbers of other systems that will affect .... "

My problem is, where will I be when all this patchwork happens...???

Don't take me too seriously, folks, it's late at night.

-- Tom Carey (tomcarey@mindspring.com), April 02, 1999.

Black start training, Robert?

How about job training.

When I get a lead on the actual report, I'll post it.

Phoned the PUC offices yesterday and talked with the clueless, who referred me on to the PUC webmaster, who wasn't in.

Since it's a holiday today, I expect to hear something by Monday. Plan to be a death-grip terrier on this one. Afterall, I live inside this potential mess!

At least the story was on the front page of yesterdays Chronicle. Somebody in this Valley may put two and two together!

*Sigh*

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), April 02, 1999.