My apologies if this link has been posted already. You'll need an Acrobat reader in order to view it.

The press releases link is at:
http://www.nerc.com/y2k/newsitems.html (html format)

The April 9 press release is at:
ftp://ftp.nerc.com/pub/sys/all_updl/docs/pressrel/4-9-99-y2k-drill-press-release.pdf (Acrobat format)

-- Tim (pixmo@pixelquest.com), April 09, 1999

Answers

If you have RealMedia, you can hear
Mike Adams on CNET Re: NERC Tests & FAA
Mike Adams on CNET Note: Just a thought from a paranoid, hoarding, conspiracy nut

-- WebRNot (webrnot@ncap13k.com), April 09, 1999.

Of course it was a success. After all, wasn't that the intention of the drill? But they did throw out a few minor problems, so it wasn't TOO obvious

7 There were some incorrect phone numbers identified.
7 Geographic coverage of low and high frequency radios will require relocation of some antennas.
7 Personnel require additional training on the use of satellite voice systems; however, some of these systems have been only recently installed, leaving little time for training prior to the drill.
7 Some congestion of voice traffic on some of the backup systems indicates a need to establish and practice voice traffic priorities.
7 Some of the backup voice transmissions were noisy.
7 A small number of the backup systems did not work as planned and will require further checkout prior to the next drill.
7 There were severe lightning storms in the Midwest today associated with a storm front; this weather did temporarily interfere with some of the radio systems.


-- DQS (none@none..), April 09, 1999.

It helps, but I'm disappointed in the systems they chose to cover, and the limited effective systems analysis actaully checked.

They need this repeated each shift, with actual comm's set-up and integrated with the controllers - or they only THINK they know what will really happen. Nice PR, but it isn't a test.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 09, 1999.

Mr. Cook: I am becoming increasingly concerned with the negatism you portray about the utility industry's ability to prepare for the Year 2000.

I ask you for the third time: have you Y2K tested a single device or system used in a power plant or T&D system? I have personally tested over 100, and I've found that the systems are Y2K Ready. The few systems that need upgrades are widely known in the industry and are (or have already been) replaced.

You say about yesterday's test "It helps, but...." Geez, man, why not give us a little credit? I was in charge of my company's drill, and we actually found out that our operators know of some clever ways to get the information they need when primary communications go down.

You say tests need to be with the "comm's integrated with controllers". What the heck are you talking about? One of our tests was to simulate the loss of data as it appears on the energy management system screen for several substations. Do you know how this data is normally received? Here it is: Transducers translate the actual current and voltage magnitudes into analog quantities. Those quantities are then translated to the remote transmitting unit, which then sends the data over a fiber optic system (which has its own controls). The fiber system transmits the data to a computer program which displays the values on a screen. Now that is quite a system, and these are the types of SYSTEMS we have already Y2K tested.

Finally, you refer to "more testing" to be done. Did you read the NERC information? This was a DRILL, not a test. It was SIMULATED loss of communications and EMS functionality. What we were trying to find out was whether PEOPLE knew how to properly react when primary voice and data communications fail. We ran the drills just as the NERC guidelines stated, and found that our operators know their procedures well. Was the drill perfect? No, we found some procedures are slightly out of date, and some backup comm lines were not working. So the drill revealed some work we need to do.

I think it's time you and I made a gentleman's wager, Mr. Cook. You said in a previous thread that "The failures will be due to simultaneous failures you don't have experience in-from factors you aren't aware of yet-because they have not been tested yet."

Here's the wager: I say that there won't be any outages of significance. If more than one half of one percent of power customers end up out of power due to the y2k bug, I will publicly on this forum admit that I was wrong and greatly mis-characterized and misunderstood power systems and Y2K. If less than 1/2% go out of power, you admit the same. Do we have a deal?

My intention is not to flame, but to "draw the line" with you, since you seem to have some popularity on this forum. I was invited to this forum to discuss power and Y2K, and I think it's time to raise the stakes a bit.

-- Dan (dgman19938@aol.com), April 10, 1999.

Dan,

First, let me say I am happy to hear good news. I like having power. :-D Second I wanted to make a couple quick points.

You stated: "We ran the drills just as the NERC guidelines stated" (snip)

NERC Guidelines stated:

* Do not make the drill too complex. We want to have a successful and meaningful story for publication.

You stated: " Did you read the NERC information? This was a DRILL, not a test.

From the NERC information: "1. The April 9 drill is intended to install public confidence through success and at the same time be a real *test* of our ability to operate with limited communications capabilities." (emph. mine)

ftp://www.nerc.com/pub/sys/all_updl/docs/y2k/drill-preparation-strateg ies.pdf

BTW-I have not tested any of these systems. If I had to, we would be in deep trouble. I'm just wading through all this info trying to find truth.

-- Deborah (infowars@yahoo.com), April 10, 1999.

Hello Deborah. I have a few comments:

I'm glad you like having power. I like working for a company that provides it.

Yes, it is true that the NERC document advised us to not make the drill too complex. Some might read into that wording that the drill was "rigged". This is not true. I was at the NERC conference in Texas in February, when we utility folks were discussing drills. There were several people present who worked in the nuclear power industry and had much more drill experience than the rest of us. They warned the rest of us that if we tried getting fancy with our drills, that the drill could take too much time or confuse the drill participants. That's why NERC wanted everyone to keep it simple. Here's an insider's tidbit: I know of another utility that did try to get fancy, and ended up taking hours longer than the rest of us to finish! The drills were a success though.

Regarding the difference between "drill" and "test", you are correct that the NERC document used both words at times, and yes, they can become interchangeable. But we utility types were deliberately not using the word "test" because it might cause undue fear in the public. Sure enough, an e-mail started flying around the net the day before the test warning people to expect power "surges".

I don't have a "bone to pick" with you, Deborah, because you don't purport to have expertise on the subject of Y2K and power. But Mr. Cook does, he has an audience here, he is what I would call a "gloom and doomer", and I want to make sure that people know where he is coming from on this issue, and that the power industry is quite prepared for Y2K.

Dan.

-- Dan (dgman19938@aol.com), April 10, 1999.

Dan,

Thanks for your response. I can understand where you are coming from. You seem extremely confident the energy industry will be ready. I hope so (I really do like electricity!).

In my State (Illinois), their web page spoke of concern about smaller power co's, which seemed to be 'waking up' to the y2k issue at a really late date.

I realize most of the power in our Country is supplied by large power co's, however there are many small power co's.

Do you think the majority of these smaller co's will be ready? If not, do you think this could impact larger co's connected with these smaller co's to the 'grid'? Why/why not?

I have read as much as I can find about these issues, I really haven't had a 'conversation' with an actual person in the industry. I am not trying to argue, I really am just interested in your opinion, especially based upon the industry you are in.

Thanks for taking the time,

-- Deborah (infowars@yahoo.com), April 11, 1999.

Hello again Deborah. Yes, I do think that some of the smaller power companies might be lagging behind the larger ones with regards to Y2K. However, those companies are even less dependent on embedded systems with date functionality than us larger ones. Also, they use a lot of the same equipment that we do (we both use distribution equipment, it's just that the larger ones also have generation and transmission), so I can tell you that in general, distribution equipment (protective relays, line reclosers, meters), is almost all already Y2K ready. One caveat...there are a few meter types that need software upgrades, so if you have a meter that has date awareness you might want to get with your power company. The best web site for co-ops, is NRECA.com (or .gov).

Dan.

-- Dan (dgman19938@aol.com), April 11, 1999.

No Dan I understand your position - and your statistics (0 of 100) validate the national averages. (That is, they are within the margin expected for what's experienced nationally of 2-5% of embedded chips are affected.)

Where I am less optimistic than you is in your extrapolation of your single experieince in testing the embedded chips in a single case, to my experience in the "whole system" - the whole process of getting power from the coal embedded in a rock wall 1500 feet underground to the 120v outlets in 100,000,000 homes and businesses scattered all over the US and Canada. Your experience is correct, valid and good. Every success should be applauded - each will contribute (eventually) to the successful conversion of enough systems so the whole may stumble through.

But your experience is incomplete in that it doesn't cover the whole process. I've not specifically been overwhelmling concerned about "embedded chips" as a whole - several instrument and sensor companies indicate that the instruments themselves will (most likely) be unaffected - as you have pointed out. But the controller they report to, the logging and control processor that act on this data (and the PC or "embedded computer" behind it) remain very vunerable.

I've long held that, with extra care and due diligence by its operators, many plants can "get power to the fences" successfully - but will likely have trouble past that relatively small specific area they can control and directly support manually. Unfortunately, this level of "special effort" - as I've had to do many times suring startup and extended testing - can't be sustained indefinitely. Mistakes, out-of-spec conditions, and compounded errors become more likely over time - especially as people get more tired, less mentally alert and less careful.

You've got to understand that most observers have always expected as much as 80% of the nation's systems will (most likely) manage okay - maybe not as good or as carefully controlled as now, but "okay". Now, which "20%" may fail, and in what ways the remaining "20%" may fail, and what the effect of that "20%" failure may have, and even whether "20%" is even accurate - it may be 50% or it may be 5% of the world's systems - is open to legitimate debate. What is true in what you have personally seen does not extrapolate to other systems and processes - worldwide or across other industries.

Your observation gives me one point of reference - in a "graph" of several thousand observations, it (by itself) is not statistically relevent. It does confirm that the year 2000 troubles can be mitigated (we've always agreed with that), that they can be solved if enough time, money and effort are spent early enough by enough people who are empowered to analyse and solve enough problems. (For example, if Y2K analysis is limited to "asking the vender" - but not tested full-up by integrated systems testing across all processers and system controllers in the plant - then I expect - based on my experience in systems installtion and program testing, not on your experience in checking single chips - that many problems will remain hidden.

Until they "creep out of the woodwork" next January and February.

What we (generically speaking) cannot specifically predict - and why I'm always trying to emphasize this by couching things in terms of probabilities and approximations - is the cumulative effect of thousands (milliions ?) of such random failures in supposed "remediated" systems. Then the cumulative effect (on the economy overall, not just power plants - they are only a single small industry using a limited number of processers) of all of these failures, plus those 5-20-50% of non-remediated systems in the US and Canada, plus the cumulative effect of non-remediated "critical", poorly remediated "critical", and utterly neglected federal, state, and local "non-critical" systems, plus the compounded situation internationally must be considered.

As each single system, or single plant such as yours, gets remediated, the whole comes a small bit closer. But the whole remains too troubling too far away. By now, (mid-April)

30% of the states should be compliant. None are.

100% of the federal agencies should be compliant. One is.

40% of the counties should be compliant. Four or five have tested.

50% of the Fortune 500 should be done. None are publically saying so.

50% of the small businesses should be compliant. Only 50% "expect" to complete Y2K remediation at all.

That's not enough progress to be optimistic.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 12, 1999.

Robert,

Actually, Robert, not even SSA is truly, ie, totally compliant. It is internally compliant, but not end-to-end compliant, with respect to banks and the rest of the government system. I do expect SSA to function, but one can't actually call it completely compliant as of this point.

Also, keep in mind that SSA, unlike, say, the FAA, historically has a good tech track record. They have never changed their number of mission-critical systems at any time throughout the OMB process. They what they were doing. In addition, they got an early start, as opposed to the rest of the federal gov't. And finally, they didn't have as much to do as, say, the Pentagon (the Mother of All Federal Systems).

-- Drew Parkhill/CBN News (y2k@cbn.org), April 12, 1999.

Sir Dan - You've brought an amazing prediction to the floor - 1/2 of 1% national success rate is indeed much more likely than I expect - so please, kind sir, clarify your numbers and values, beginning/ending time frames, current year or only post-2000 oerations, incident (and co-incident reliability included or excluded? such as fires/floods, wind-storms affecting year 2000 reliability, or being secondary-trips affecting year 2000 problems?)

Your experience is valuable, and in general, I value the commmercially-paid EPRI results and shared database information more than NERC's "voluntary" stat's - so very heavily flavored by the federal administration. I firmly agree with you there.

Gloom-and-doomer? Hmmmmmn. Not really, not likely, though I'm more positive in expressing my opinions; and in couching those opinions with definite, specific, exacting, and highly subjective probabilities; because I've had ten years trying to keep the d**m system running during outages and repair periods, and ten year's exerience and testing and re-testing new and updated software. In other words, I've crawled through more piping, hydro'ed, flow tested, and hot tested, radiographed and welded, cut and isolated, repaired and repalced and flushed more systems than you may have. Inmore power palnts and services than you have. Or then again, maybe not. Certainly, it appears I've seen more failures in more different systems appearing more ways than you. (Else you would not be as certain as you appear.)

Now, I do not begrudge your experience - it's perfectly valid. Incomplete? Not really, just different than mine.

Pardon me and my experiences - but I've seen the results of software failures in many, many unexpected ways - and believe me, you haven't tested things yet. You (generically speaking) haven't fixed enough plants yet to begin to be optimistic.

You can only claim - even in the best of all possible worlds - that "What you specifically have tested, in the specific plants that have been tested under the specific conditions that they were tested under, with the exact test conditions as set before, during, and after the test, have passed the tests conducted."

Now - this is progress, and I will grant you progress at the tested plants. But you cannot claim any more meaningful results than stated above. After all, the Tacoma Bridge failed under light winds and the total load of one car - but it had been "successfully" built and tested under several months of service.

Until it was stressed by unexpected conditions.

Now .... I'm only willing to admit to the following absolute statement - can you deny that this following prediction is somehow wrong? I will not make other predictions, nor claim absolutes in predicting the future - you, I, and no one else can honestly do it either. A person can only analyze the past - if correct, accurate and full knowledge is available about the past, and if you assume that past experiences can be used to predict the future - and, afeter doing such an analysis, I reserve the right to come to different predictions than you.

The Brooklyn Bridge still stands - longer than any other built to that time, and more "technically risky" than any other built to that time - because it was built based on the considered experience of the designer who had analyzed previous failures, and who worked very hard to minimize the effect of previous failures, and who increased the safety factors according to judgement to account for things he didn't know and was wise enough to admit he couldn't accurately predict. He even tried to prevent failures that others didn't even anticipate - remember the sign warning workers to break step, not to jump or run on the catwalks spanning the river, to limit the number of people crossing to 25 maximum? He was anticipating trouble - and trying to prevent it from happening.

His conservative example, in turn, is what I'm trying to train people to emulate. To learn from similar events in the past, to wisely and conservatively extrapolate to the future, and be very cautious of assuming future (and hence unpredictable) events in the future. This may mean extra expense - after all - his caution meant it tooklonger for people to get to work in the moring, and to get home to their families at night. But they did get home at night -safely.

The builder of the Golden Gate bridge spent extra money on the safety net under his bridge - and his workers lived because of his decision - despite the wishes of those politicians in Washington who wanted to reduce costs and improve schedules. The engineers who cautioned against launching the Challenger were overridden by the NASA and company administrators "who didn't want to listen" to their cautions - despite the evidence of blown-by o-rings in previous "cold weather" launches - and NASA lost seven people and several years - because the politicians in Washington "wanted" a good public-relations display by launching a teacher on time and in front of the national TV audience.

Well, they launched her on TV all right - right into the ocean floor.

The Tacoma Bridge failed - after years of experience with longer, higher, and more technically challenging bridges - because its designers failed to account for these same things. It failed for the same reason that bridges in England built over 100 years previously failed (wind stress under dynamic conditions of low weighted load on low-resistance girders). So the lessons were available - just as year 2000 "lessons" are avialable now - but the engineers failed to heed those lessons.

And theorectically, under the conditions that they built and tested the bridge - it couldn't fail. Well, they were right, under the conditions that they built the bridge, it didn't fail. Under the conditions that they tested the bridge, it didn't fail.

Under the conditions that the bridge was placed in real life - it failed.

The lessons of the Challenger o-rings under cold weather launches were available - but did the administrators feel they weren't "catastrophic enough" to warrent caution? Just how much evidence do you want to justify a degree of caution?

Now, what will happen to the power system - under the conditions it may be placed under early next year? We don't know. I won't predict the 99.50% success (0.50% failure) you feel it will achieve. I also won't claim the 99.50% failure (0,50% success) that perhaps Gary North believes will happen. So I claim "title" to that 99% percent range in the middle of the two.

After all, one of the three of us is bound to be right. 8<) You can join me on the Kennesaw forum next January if you wish as we analyze the "world" and then eat lunch - depending on events, that might be hotdogs and beans, or it might be soup and peanut butter sandwiches, or it might be steak and eggs at the resturant across the tracks - I don't know yet.

----------------

My absolute, guaranteed-to-be-correct prediction of 99% service availabilities and outages (all systems, all countries, all utilities):

.... there will be many and varied failures of many and various systems affecting many and various people in many weird and various ways in irregular regions of the world that will last irregular periods of time, leading to fast and slow recoveries of various sorts back to normal over various timeframes; each failure of which will be caused by and affected by the different effects of the different problems in different computers and processes to different populations and cultures - and many of the worst effects will be in those systems and processes which will not be predicted to be affected ahead of time.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 12, 1999.

Dan...

I, like Deborah, have no expertise in electric power. I DO have some expertise in living near the #3 nuke (Chernobyl, St. Petersburg, Palo Verde Nuclear Generating Station) in the world.

Are you saying that there is no cause for concern?? This is not a safety issue. Are you so confidant that we will have that power in January 2000? We endured the cost overruns, etc. when it was built and we here in Arizona depend on it and would like to have it available.

-- K Stevens (kstevens@downwind_of_a_nuke.com), April 16, 1999.