Robert, you have claimed in several of your postings recently that the nuclear power plant where you work will be ready for Y2K. How is your plant dealing with the issue of embedded systems? I read in an article recently in PC Magazine that it will be impossible for utilities to locate and replace all embedded systems in the time remaining before January 1, 2000. Therefore, utilities will have to fix-on-failure. How can you be so sure that your plant won't have serious problems?

-- Joe Williams (wondering@dontknow.com), March 26, 1999

Answers

Joe: I'm not Robert, but I can tell you that the whole embedded systems thing for the power industry has been a big bust, so don't worry. We've tested and tested to find embedded systems that would cause devices to fail their basic functionality, but cannot find much of anything. The 100 or so power devices with embedded systems I have tested (protective relays, RTU's, controllers, etc.) have all been found to be Y2K ready (by ready, I mean that those devices will continue to perform their basic functions into the year 2000 and beyond). If you quoted PC magazine correctly, they are wrong in their assessment, because there are very, very few embedded systems that need replacing anyway.

The bottom line is that no one can guarantee anything, but based on all testing thus far, power delivery to customers will not be interrupted to any significant degree come the year 2000.

Dan.

-- Dan (dgman19938@aol.com), March 26, 1999.

Dan,

Do you speak--ith with fork-tongue or what!

ROTFL!!! Ha! Ha! Ha! Ha! Ha!

-- Twaddle (drivel@large.com), March 26, 1999.

I can't wait for Robert's comments on Dan's post!

In the mean time Joe, you may like the following thread if you haven't seen it yet. This was a test at a hydro plant. Nuclear plants are just a little more complex I think. <:)=

did you all notice this y2k power plant examp...

-- Sysman (y2kboard@yahoo.com), March 26, 1999.

dan,

every so often i hear this from people in the power industry- no problems with embedded systems. and i think that in their experience, it's probably true. the problem comes when one tries to project that experience on all power plants nationwide. are they all so problem- free? certainly not from what i can tell. let's hope they are, but let's not necessarily assume so- unless you have solid evidence to the contrary. do you?

-- Drew Parkhill/CBN News (y2k@cbn.org), March 26, 1999.

Besides which, how did you roll the clock on the embeddeds forward????

CR

-- Chuck, a night driver (reinzoo@en.com), March 27, 1999.

Well, I see I've generated a few interesting responses, even though I'm not the one being asked the question.

Twaddle: No, I don't speak with forked tongue. Can you name a single embedded device used in the transmission and distribution system, that when the clock is rolled into 2000, will fail functionality? What have I said that is untruthful?

Sysman: I've heard this Brazil story--still trying to get the details, so I can't say much for sure. See the NERC reference below.

Mr. Parkhill: I think the "solid evidence" comes in the whole nature of how power systems and their controls are designed. 99.9% of a power system is totally date-immune: Turbines, generators, cable, wire, transformers. Further, power is a commodity that is expected at all times, every day, so the equipment is not designed to care what day or time it is. The few devices that do have date awareness use it only for non-essential purposes like data logging. Those devices use their real-time-clock for stamping purposes. So the real challenge is trying to find the problems and fix them, not to "prove" there are no problems.

Now, admittedly, the control system of a power plant is more complex than simple discrete devices like those used in the transmission and distribution system. In many cases the DCS or distributed control system is being upgraded at power plants (many were planned long before Y2K issues came up), and the vendors certify their compliance. But many companies aren't just taking the word of their vendors, we are forwarding the clocks to see for ourselves. Some of us have forwarded the clock with the plant on line and putting energy up on the grid. The NERC report summarizes that dozens of power plants have been tested this way with success.

I encourage you not to use the "until all devices are tested we can't be sure" frame of reference. If you are looking for Y2K problems out there, but aren't sure if the problems exist, how can you know when you are done looking? We have looked and looked and looked. If we found a device that failed, everyone in the industry would find out one way or another, and eventually folks outside the industry would know.

Are we certain there are no problems? We can't be, because we're not sure there is a problem as it relates to the power industry. But we'll keep testing and looking for anything that might be of concern.

-- Dan (dgman19938@aol.com), March 27, 1999.

Dan, thanks for the information you gave me. I truly hope that you are right about there being no significant interruptions in power in the year 2000, but I wonder how PC magazine could have gotten the story so wrong about the embedded systems problem. A lot of what I have read about the Y2K remediation efforts of utilities indicates that they are behind other industries and most troubling of all, that about 1,000 small rural utilities are not doing any Y2K fixing at all. An article I read recently indicated that when NERC requested information about the status of Y2K repairs from the nation's top 200 bulk power producers in September 1998, that 25% of them didn't even bother to respond. This does not give me a "warm and fuzzy feeling" about the power industry as a whole, in spite of your sunny outlook. Robert Cook, are you out there? What do you think about the embedded systems issue as it pertains to the utilities industry, and in particular, the nuclear power plants?

-- Joe Williams (wondering@dontknow.com), March 27, 1999.

Hi Dan. First, thanks for your information. There's nothing like the experience from one that is "in the trenches". I'm not looking for an argument here, and I'm not qualified in this area. However this issue is a major concern for all of us. The following is a snip from the US Senate report on Y2K. I've posted this several times (sorry regulars):

At the time of the hearing, there was a lack of industry-wide survey data of the electric power industry. As a result, the Committee staff surveyed five large electric and five large gas and oil companies to obtain cursory readiness information. Figure 3 below displays the result of the survey. Based on the survey results, the Committee concluded that the utilities were proceeding in the right direction, but the pace of remedial efforts was too slow and there was so much remaining to be done that there was significant cause for concern. Only two of the eight firms reported completion of assessment, making assertions of Y2K compliance by December 1999 highly suspect. Committee concern was heightened because the most difficult tasksrenovation and testingwere yet to come. The utilities lack of information regarding Y2K compliance of their major suppliers, vendors, and service providers created additional concerns about the utilities assertions of readiness. The survey results raise significant levels of concern given that the firms surveyed were among the largest utilities and were dedicating many resources to Y2K (collectively over $400 million). Smaller firms with fewer resources are presumably further behind in their Y2K remediation efforts.

I'ld like your comments on this Dan. Also, do you think the 400 million figure is for only their business systems? Thanks for your time and effort. <:)=

-- Sysman (y2kboard@yahoo.com), March 27, 1999.

dan,

i understand your points, and have heard them from others. however, my understanding is that some plants have more problems than others, which goes back to my original point that results from one or even several plants cannot necessarily be projected onto the grid as a whole.

i also keep running into stories of systems passing a test, then completely unexpectedly failing the next one, quite to the surprise of the previously-confident test team. (in the case i'm thinking of, though, they were able to correct the problem fairly quickly, as i recall). i don't know if that was an embedded failure or what it was. but i do know such things happen, requiring multiple tests. hence the need for the "test all devices" model, at least in some cases.

ps: "mr. parkhill"??? i ain't *that* old! :)

-- Drew Parkhill/CBN News (y2k@cbn.org), March 27, 1999.

More later - finished a long interview with EPRI covering power and gas distribution on late Friday. Dan's summary is partly true in his details - but also incorrect in its more sweeping conclusion by trying to assume that just that one part (embedded chips) he is familair with is representative of the whole process.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), March 27, 1999.

Also - to clarify an implied point - I have worked "at" (underneath, inside of, on top of, in the middle of, and through the middle of) many nuclear power plants of many different "classes", designs, and styles. Yes, I've built them, studied them, designed them, repaired them, fueled them, started them, ran them, stopped them, defueled them, and rebuilt them. So, in most areas of any conventional fossil, unconventional fossil and nuclear power plants, I feel I'm qualified to offer usually valid general comments and criticism, or know enough to be able to get hold of experts better than I to cover the more detailed questions I can't address.

Right now, I do not work "at" one, but do 3D CAD design, testing of programs and designs and analysis of several different plants for several different clients at many different sites, both here in the US and abroad.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), March 27, 1999.

My responses to the more recent inquiries:

Joe Williams: You referred to the "small rural utilities" as possibly being behind schedule. The organization overseeing that activity is the National Rural Electric Cooperative Association, or NRECA. Their web site is www.nreca.org/y2k/. In the "news" section of the NERC web site, NRECA is quoted as stating that they are "right on schedule". If you are served electrically from a co-op, I suggest going to NRECA's site and seeing if the company serving you is listed. In general, you have less to worry about with co-ops, because in general they have not added a lot of digital equipment, and it they have any, it is the same models that us larger utilities have on our distribution systems (protective relays, load tap changers, disturbance recorders, meters). This equipment has been tested and found to be Y2K ready.

Regarding NERC getting a 75% reponse back in September, that is old news. Presently NERC is receiving greater than 98% response. If you want to find out if your power company is reporting, go to NERC's site and you can find out.

Sysman: I understand your concern about power companies, because we all know how important electricity is to our society. I have two small children, and I'll have to be at work during the rollover, so I want to be sure they will be safe just as much as the next guy. Regarding the Senate report, I'll have to review that one again before I respond; as I recall there was some formal response from NERC on it, but I'll try to dig it out. Regarding the $400 million figure, that may include embedded systems work, but for the power companies I've talked with, the large majority of $$ spent is on business systems, around 75% of the total expenditure.

Mr. Parkhill...oops, I mean Drew: Most of the power plant stories are just that..."stories". I haven't yet seen good documentation of a power plant truly shutting down because of a properly run y2k test. There have been several stories (I'm sure you've heard about the plant in England, the Con Edison plant being down for 13 days, etc.), but when we tried to verify them the story dried up, kind of like an urban legend. I'll work on the Brazil hydro plant story...that one might have promise because the article does mention a specific power plant.

Mr. Cook...everyone seems anxious for your reply to Joe's questions...I was invited to this board by someone to discuss issues with you (evidently we don't share the same opinion on Y2K and power companies), so let the discussions begin....was the EPRI interview relating to Y2K, or something else?

That's all for now.... Dan.

-- Dan (dgman19938@aol.com), March 28, 1999.

Dan, There were a couple of threads this week about the embedded systems issue. The upshot seemed to be that embedded systems at various organizations are being found to have a failure rate of between 1.5% and 100%. It seems from company reports that failure rates of 6% and 13% percent are common.

Are you saying that your company has no such failures, or are you saying that your company's failure rate is significant, but the consequences of the failures is not significant? . . . or something else?

-- Puddintame (dit@dot.com), March 28, 1999.

Welcome back Dan, and thanks for your reply. Here's a link to the Senate report for your reference:

HTML Senate Reort on Y2K with bookmarks

I look forward to your discussion with Mr. Cook. I've only been a "regular" here for a couple of months, but I've developed a great respect for Robert and his insight into the industry. This thread is getting pretty far down the list, so if you don't hear from him in a day or so, you may wish to "ask a new question" to get his attention. Thanks again for your time. <:)=

-- Sysman (y2kboard@yahoo.com), March 28, 1999.

Hello Puddin: I am saying that we had zero devices in the embedded systems category fail their basic functionality during Y2K tests. Counting other power companies, the number is the same...I would estimate that more than 300 different device types used in power transmission and distribution systems have been tested representative of thousands of devices with zero failures.

If a device functions fine but uses 2 digit dates or mis-prints a non- essential report, are some people counting that as a failure? If that is the case, there are many devices that don't meet that standard, but again, this isn't important.

Dan.

-- Dan (dgman19938@aol.com), March 29, 1999.

dan,

i'll follow up on that power plant story. my guess is they don't want it discussed much beyond what i've told you. however, to the best of my knowledge it's true. i don't know the details, though.

i just don't have the time (trust me) to track down some more, beyond the old familiars- hawaii electric, et al. that said, i do know that the level of degree of worry inside the industry has always been much higher than discussed publicly. cowles & martin have said the same thing. in addition, there has also been the report from dick mills that federal regulators have discussed whether or not to ration power between localities should one have it and another not. thus, one would assume that federal regulators are taking such possibilities seriously.

-- Drew Parkhill/CBN News (y2k@cbn.org), March 29, 1999.

Dan, Thanks for the info. To me that is very good news. You mentioned transmission and distribution but not generation, but I assume that the same can be said (?).

Dan, Why don't the power companies widely publicize these types of results. If we could go to a website and see that XYZ Co. has 3,000 embedded systems and all have been fully tested in all relevant environments, and 4% functioned somewhat abnormally but such malfunctions were of a minor or trivial nature which did not interfere with any critical function, then I think that would be reassuring evidence. This would not even require a "guarantee" the thought of which seems to put management types into a cold sweat.

Why is it so important that industry of all types hold the details of remediation so close to the vest? I am much more interested in the data than in conclusory statements of "readiness." I don't expect you to have an answer for that one, just wondering.

-- Puddintame (dit@dot.com), March 29, 1999.

Hello again, Puddin. Why don't power companies advertise it more? Here's my perception on this. Most of us are still working hard to finish up (the last 10% of a project always seems to be the most difficult). I suggest that we wait a little while, because many Y2K project manager types are suggesting that a more proactive approach be taken with the media. Come July, there should be lots more information.

To Drew: Let me know if you find out anything on the Brazil story.

To Mr. Cook, how come you never answered Joe's question? Did this get answered in another thread?

Dan.

-- Dan (dgman19938@aol.com), April 01, 1999.

Welcome back Dan. I watch "New Answers" so I noticed this thread. Got your e, hope the link was useful. Are you going to be in town for a few days? <:)=

-- Sysman (y2kboard@yahoo.com), April 01, 1999.

Dan, there is a post in this forum on March 31 about CP&L spending $4 million on y2k from 1994 to now, while planning to spend $13 million between now and August when it will be "ready." Can you shed some light on what that's all about? Are they going to be doing three times as much work in the next 6 months as they've done in the past 6 years?

-- Puddintame (achillesg@hotmail.com), April 01, 1999.

Haven't had time yet - it's on my "too-due" list, but got superceded by my boss's "do-this-first" list.

Work first, hobbies later.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 01, 1999.

Robert, hope to hear from you soon

-- dick of the dale (rdale@coynet.com), April 01, 1999.