The following is a classic example of why our inability to anticipate everythingrequires us to prepare for the unexpected. ------------------------------------ Peach Bottom Unit 2: Loss of Plant Monitoring System Computers During Y2K Testing On February 8, 1999, while performing testing for a Y2K remediation modification to the Unit 2 rod worth minimizer (RWM) system, operators experienced a lock-up of both the primary and backup plant monitoring system (PMS) computers. As a result, operators also lost the following PMS-supported systems for about seven hours: safety parameter display system (SPDS), emergency response data system (ERDS), and 3D Monicore thermal limit monitoring system. Engineers had taken the backup PMS computer off-line and had advanced the PMS clock to a year 2000 Date. This led to a lockup of the backup PMS, and the system transferred to the primary, on-line PMS computer. The engineers did not recognize that the system had transferred and, believing that the original command was not accepted, again advanced the system clock, causing the primary PMS to lock up also. Several initial attempts to restore the PMS computers were unsuccessful, and operators determined that this constituted a major loss of emergency assessment capability. The PMS computers are not Y2K compliant, but the engineers believed that this would not impact the testing. Operators did not expect the testing would affect the on-line PMS computer. However, before the testing began, operators took contingency actions to lower Unit 2 power slightly to ensure shift average power levels were not exceeded. The licensee plans to perform a full root cause analysis of this event.

http://www.nrc.gov/NRC/NEWS/WIR/week3.html#_1_10

---------------------------- A line-by-line analysis is not necessary to understand the lesson from this incident. In a few months, and I believe 'a few months' is the correct frame of mind, incredibly complicated systems will behave in unbelievably baffling ways. Intelligent, competent people will still be people - unable to anticipate everything and capable of making errors. After all, people designed the systems. There will be no barriers or borders. In a few months, y2k will truely be an equal opportunity event.

-- PNG (png@gol.com), March 03, 1999

Answers

PNG: Thanks for bringing this over. I read about this in comp.software.year-2000, and posted the following.

Thank you very much Mr. Cowles:

Perhaps a "bunker mentality" is not such a stretch.

Out of all the things related to Y2k and systems failure, human error - scares me the most.

I've mentioned in previous posting's on csy2k, my personal involvement in remediation of the consequences of the Chernobyl catastrophe, as well as the Exxon Valdez. Both of these events were brought about as a result of human error.

In the next ten to twelve months, throughout Y2k remediation and testing, we may see more of the same as Peach Bottom Unit 2. It's possible that during the efforts to meet deadlines; death marches, fatigue, and stress, will impact judgment and safety. Hopefully we'll get by without another Chernobyl, or worse.

Please everyone - let's be careful out there.

-end-

Best Regards, Tom McDowell

-- Tom McDowell (bullriver@montana.com), March 03, 1999.

This story is at Rick Cowles site, euy2k, Electric Utilities and the Year 2000, with a bunch of other info about same. <:)=

a href="http://www.euy2k.com/newsroom.htm">euy2k.com - The Newsroom

-- Sysman (y2kboard@yahoo.com), March 03, 1999.

Sorry, fat fingers and a late nite! euy2k.com - The Newsroom


-- Sysman (y2kboard@yahoo.com), March 03, 1999.

Sysman - human error in action? ;-)

-- Tricia the Canuck (jayles@telusplanet.net), March 03, 1999.

You got it Tricia. One would think that after spending 31 years of programming at a keyboard, I would have it down by now! Only human I guess. <:)=

-- Sysman (y2kboard@yahoo.com), March 03, 1999.

If I recall, Homer Simpson works in a nuclear power-plant...

-- Anonymous99 (Anonymous99@anonymous.com), March 03, 1999.

Last Friday, I attended a presentation by Rick Cowles entitled "Will the Lights Stay on in the Year 2000?". He talked about the Peach Bottom incident, and I reported (on another thread) this summary of what he said:

The recent event at the Penn "Peach Bottom" Unit nuclear power plant,
where both secondary and primary safety monitoring systems locked up
during Y2K testing (read all about it at www.euy2k.com), should serve
as a big Wake Up Call. Heretofore, the Nuclear industry has assured
everyone that Y2K was not an issue in terms of their safety shutdown
procedures (not to be confused with day-to-day power generation, of
course), yet we seemingly have a very sobering counter-example to
this claim. Further, according to Rick, only a couple of days
afterward, the Nuclear Industry was still going on record as
claiming that Y2K was not a safety shutdown issue, as if the
Peachbottom Incident had never happened! (Think: Three Mile Island
Lessons Learned.)

-- Jack (jsprat@eld.net), March 03, 1999.

Thanks for the info Jack. Sorry for the redundant post.

-- PNG (png@gol.com), March 03, 1999.

No problem, PNG, probably the more visibility the Peach Bottom incident gets, the better! Anyway, for what it is worth, here is a link to the earlier thread:

notes on R.Cowles' "Will the Lights Stay On in the Year 2000?" presentation on 2/26/1999

-- Jack (jsprat@eld.net), March 03, 1999.

OOPS, try this:

notes on R.Cowles' "Will the Lights Stay On in the Year 2000?" presentation on 2/26/1999

-- Jack (jsprat@eld.net), March 03, 1999.

Please critique my opinion about how this specific test is a reason why this shows why nuclear plants are MORE likely to be safely running next Jan - Feb than many conventional plants:

http://greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000XqU

Add comments here, or there, as you (the reader) sees fit.

-- Robert A. Cook, P.E. (Kennesaw, GA) (cook.r@csaatl.com), March 03, 1999.