(note: this was sent by my husband who is Operations Manager of a power plant, to several of his colleagues and friends. I am posting it for general info)

You know, I've been interested in this Y2K crap for a while now, as a kind of fantasy "Doomsday" scenario. Planet of the apes, Conan the Barbarian, make a new go of it, all the good stuff. Being the instigator that I am , I liked to throw it out there it and see what kind of reaction I would get. Good study of people.

The wife and I were preparing for a disruption in services, like the ice storm we had here last year. Folks around here didn't have power for a month. No water. The local comminutes ran shelters, fuel and food rationing happened, restricted travel, National Guard, that kind of thing.

What goes around comes around.

At work I got tasked from on high with fixing the "Y2K Problem". No sweat. The only thing that I figured would be goofy would be some software, mostly in accounting, other people interfaced stuff, and the older desktop computers. They needed to be replaced anyway. We and the World could muddle through.

The major portion of the plants controls, (remember? I work at a fairly new power plant) although computer operated, were a different breed of machine, and care nothing about the date, they merely control their specific machine and send info to the plants main control computer, which operates valves and various other support devices for the main processes. No big deal.

Investigation into the plant's main control computers also showed that it was no big deal, the only thing that (reportedly) would go wrong was some trending functions, which allow us to "look in the past" ( I am the master of the run-on sentence... On a dark and stormy night...)

Checking some 40 odd control systems and over 400 circuit cards nothing showed. So far, so good. No indication of Y2K failures. Oh, I've got 2 systems, besides the trending thing, to software de-bug, but they only look to see if a pump has been running for a week straight, if so, the program shifts to the standby pump. Once again, no big deal.

I am now down to the "stuff you never think about " level. Embedded chips. Supposedly the real crux of the matter. It seems to me that if the embedded chip doesn't have a people to machine interface, a read out of some sort, or operate on a calendar of some kind, such as : I haven't been serviced in 6 months so I will shut down function, then how could it know or even care, what today's date is ?

Silly boy.

I ran into a doosey of an "embedded chip". Three of them as a matter of fact. The only three I have looked at so far. This does not bode well...

Doosey #1: The plants phone system will not work. The manufacture of the chip says so. The rest of the system will, but if the brains of the systems don't, what part will ? The manufacturer doesn't support this system anymore even though it was built in 1992.

Doosey #2: The meter which reads the total power output of the plant will not work. Who would check a simple meter ? Here is the good part : This meter masquerades as a simple output meter, but in reality it sends a signal to the supplementary firing system which controls the Steam Turbine portion of the plant. 36 million watts worth. For $ 95.00 we can get the meter upgraded to Y2K OK status. Consider it done.

Doosey #3: The best one. The gas turbine's ( the main source of power & heat for this plant and most power plants built in the last 15 years...) fire suppression system central processing unit will not work. The card manufacturer says "We never built anything like that", but I'm looking at the card...and they are sending me a test program for the "card we never built". I went to the manufacturer of the CPUs web site and by gosh, it ain't going to work...Once again, if the brains of the system are not going to work, what part will ? That's not the best part. If the fire suppression system is down, the Gas Turbine will not start. If the Gas Turbine is running on Y2K date roll-over, the Gas Turbine will shut down. Violently. If the Gas Turbine is not running, the rest of the plant will not be running. NO POWER to the grid. We could send a false signal to the Gas Turbine to indicate that the fire suppression system is OK, but what would our insurance carrier have to say about that ?

It appears that I will have to have a whole new fire suppression system system installed, and that will take at least 6 months to do. On top of the pain in the ass factor of specifying the thing, I'll have to convince the Boss to spend the bucks to do it (at least $20,000 and 1~2 weeks that the plant will have to be shut down). Thats after it fails its Y2k test, like the manufacturer says it will. Crap.

This Power plant is fairly new, and one would expect things like this to happen. On older plants, we've got no problem you say ? Wrong-oh. All plants are constantly being repaired, replaced, upgraded. The manufacturer of choice for the utilities, (company name withheld), has major Y2K problems. Not counting the embedded chip thing.

I called the local municipal utility, who supplies our back up power, to inquire about their Y2K status. They don't believe in Y2K I was told... They are doing nothing. They won't have a problem. I was told this this by their Chief Engineer. He also said he was taking all his money out of the bank by December next year.

Kids, I think we are in the crapper.

Bobbi http://www.buzzbyte.com

-- Bobbi (bobbia@slic.com), January 03, 1999

Answers

Thanks Bobbi, the more you learn about this subject, the more amazed you become.

With one year to go, it seems to me that what it will boil down to in most cases is the integrity of the people individually in charge of entities, whether they will byte (ha!) the bullit and do the right thing or pass the buck and head for the hills at the last moment. Human nature does not suggest a positive outcome, alas.

Also what struck home with me was the responsibility placed on your husband. If he was a fuck-up professionally, who was to know? - he's in charge of his own empire after all. The odds on there being a lot of fuck-ups in positions of power are extremely high - one rises to one's level of incompetence, no? Just look at Bill Clinton.

We are in trouble.

-- Andy (2000EOD@prodigy.net), January 03, 1999.

Hi Bobbi

Just where are you located? Hopefully, not upstate New York.

Duane

-- Duane (Duane24062@aol.com), January 03, 1999.

LOL Duane! Yep...upstate New York. Actually northern New York state. Just what you wanted to hear, right?

-- Bobbi (bobbia@slic.com), January 03, 1999.

Hi Bobbi:

Thanks for your post. Reading it was a little like getting punched in the chest for me. I have thought for a long time that if the lights go out they will probably stay out for a very long time. How could your husband hope to fix what needs to be fixed without power. If your husband's scenario is duplicated by 100 or more plants, and the municipal power plant that backs his plant up is duplicated by another 100 or so, its all Kaput. Could you tell us what part of the country you live in?

Bill in South Carolina

-- Bill Solorzano (notaclue@webtv.net), January 03, 1999.

Bill wrote: "Could you tell us what part of the country you live in?"

See post above. Northern New York state, on the western edge next door to Canada, and for reasons which should be obvious, that's as much info as I can give for now. :-) Bobbi http://www.buzzbyte.com/

-- Bobbi (bobbia@slic.com), January 03, 1999.

Ugly story, and one sure to be repeated in alot of utilities around the country.

It doesn't matter where the poster is, nor what company it is. There are likely many of the same situations going on around the country. If the big utility in my area (but not the one I'm hooked to), goes down, it'll drag most of the region down too, and that'll chain react and take down still more. So, that NY state utility could take out my local guys, and the guys in NC state too.

We're in for a rough ride.

-- Bill (billclo@hotmail.com), January 03, 1999.

Bobbi: California is even in worse shape. The newest power plant that we have here in California is Diablo Canyon Nuclear Power Plant, built in the late 60's. Here in Northern California, PG&E has sold off (due to deregulation) their fossil fueled plants, they were built in the the 50's. They are holding onto Diablo Canyon and some hydro plants. They would rather be in the distriubtion business than the generating business, can you blame them? I worked at one plant for about a year as a contract employee, and I can honestly say that the left hand doesn't know what the right hand is doing.

-- bardou (bardou@baloney.com), January 03, 1999.

IF the DOE understands this, and IF they can convince whoever is President, then look for martial law about 12/26/99. Only emergency and basic use of utilites; with heavy fines for non-compliance.

-- curtis schalek (schale1@ibm.net), January 03, 1999.

If you can't name the plant how can we verify this? I'n not saying you are lying, but how can I be sure? Please do give us some details.

-- James McLacribittle (athanasiancreed@5star.net), January 03, 1999.

James, I emailed you privately with contact info for my husband. You can correspond directly with him on this if you wish. Bobbi

-- Bobbi (bobbia@slic.com), January 03, 1999.

Whose surprised? I'm expecting that we'll end up with "islanding". It seems the newer the plant the more at risk not the other way around though. The older plants seem to have more manual and analog systems still in place. Do they have crews that can run them that way?? Lets just pray that 99's y2k problems don't crash the fix on these plants. Some analysts like Gartner group and Ian Hugo's UK task force 2000 are predicting big trouble right ahead.

-- Glenn L. klotz (lee00@earthlink.net), January 03, 1999.

James: You raise a good point and one that you will continually have to deal with in Y2K issues. Since verification (in the event that the information is true) would mean asking this person to put their job and family's well being on the line, verification is unlikely.

If the information cannot be refuted from context alone (in the event that it could be easily proven incorrect without resorting to independent verification), then the information must be categorized as simply 'unverifiable anecdotal evidence'. No one single piece of such 'evidence' should ever be used to conclusively 'prove' any specific position.

In the last six months of researching Y2K, I have read many such statements. I try not let any single such post influenece me too much one way or another. On the other hand, dozens of such examples exist and pattern which emerges is cause for concern.

I imagine a few people with a great social/spiritual conscious will become true whistle blowers over the next few months. But becoming a whistle blower in our culture is heavily penalized. You'll lose your job. You'll have great difficulty getting another one. Your old company will probably sue you for violating your non-disclosure. Their lawyers will agressively seek legal remedies to prevent you from discussing the matter further and their PR folks will paint you as a 'nutcase' disgruntled employee with an axe to grind. Discredit and financial woes will mount. This is the machine at work - it's who we are.

Likewise, the same 'standards of verification' which we would like to apply to such examples of 'anecdotal evidence' should also be applied to statements made by corporations. In their official statements, will companies will always seek to minimize troubles, maximize good news and paint as rosy of a picture as possible. This is one reason why so many of the current 10-Q statement are absolutely worthless.

In the end, we have three real ways to measure the accuracy of statements made about about Y2K (from all sources) and all 3 are problematic:

Evaluating the statements within the context of what is generally known and accepted as fact. This is problematic because even the experts are in strong disagreement on many issues.

Independent Verification: This is problematic because independent verification and validation (IV&V), to be believed must be performed by a truly independent 3rd party - not a group of employees from the company seeking IV&V. It is expensive and time consuming. Also, IV&V in the case of anecdotal evidence requires a whistle blower willing to sacrifice a great deal to bring negative information to light.

The "proof of the eating is in the pudding" approach. This means waiting to see if the statements made are born out by the events that unfold. This is problematic because it requires us to wait until preparation is no longer feasible.

In issues as large and as serious as Y2K, you must not focus on any one peice of evidence as 'conclusive'. You must gather as much evidence as possible, weigh each in the context of who is saying it and what their likely motivations are to be, then try to at the 'larger picture' by focusing on the patterns of behavior which emerge.

It is not any one single peice of evidence that gives me concern but rather the overall pattern which emerges.

-- Arnie Rimmer (Arnie_Rimmer@usa.net), January 03, 1999.

Bobbi - I have posted this all over the place, now I am putting it here, and will post at the top level as well

TITLE SEEKING INDUSTRIAL PLC THAT REALLY FAILS AT ROLLOVER

Now please understand exactly what I am asking here before you respond. I am asking about INDUSTRIAL PLCs - not microwave controller chips, not the stuff in your kids toys - I can live without that crap. And I am talking about FAILURE - REFUSAL TO RUN ANY LONGER. SHUTTING DOWN THE SYSTEM TYPE FAILURE. I am not talking about - I am sending bad data to the billing dept. and the bills are screwed (oh woe). Billing is not essential to life. I want to hear about PLCs in current usage (NOT stuff retired years back) that shut things down if they are used during 1/1/00 - and I don't mean stuff that reboots to 1/1/90 if you cut the power - a thirty second fix - I mean stuff that has to be replaced.

Please include the manufacturer name, model #, serial #. The date of acquisition or of manufacture would be nice to have as well. This info will prevent false data from finding its way into my logs.

send reply to

davisp1953@yahoo.com

Keep hearing second hand stuff - no first hand reports yet with the requested data yet!

-- Paul Davis (davisp1953@yahoo.com), January 03, 1999.

Bobbi,

I posted a link to your story on the "Gary North Is An Idiot" web site to see the reactions the posters there would have to your story. You might want to visit that site and comment on their observations.

http://www.smu.edu/cgi-bin/Nova/get/gn/338.html

Robin Messing

-- Robin S. Messing (rsm7@cornell.edu), January 03, 1999.

What can be done to get the utilities to come clean? All we need is a statement on where they're at? And is there anyone out there who knows what's going on at PNM New Mexico? this is incredibly frustrating s

-- Sharon Schultz (shalom100@aol.com), January 03, 1999.

Earlier today I read information referring to the power supplies in Texas. It stated that outages in Mexico can result in the power in Texas to be brought down because of the grid connections...adios amigos.

Texas Terri

-- Texas Terri (DeepInTheHeart@Texas.com), January 03, 1999.

The unusually candid assessment of how Y2K could impact electric utilities in Texas, by their Public Utilities Commission, is at:

http://www.puc.state.tx.us/Y2K/2392/TEXTS/ELECIMPC.HTM

-- Kevin (mixesmusic@worldnet.att.net), January 04, 1999.

Noticed your little aside about "violently shutdown" when the controller/sensor/program card dies. Depending on when/what/how that "violent" shutdown happens - there may or may not be subsequent damage to the generator and gas turbine themselves - that's a lot tons of very close tolerance machinery spinning at very high RPM's to be shutdown. Also recovery is harder if the shutdown is prolonged, abnormal, or faced by subsequent damage - bowed rotors, loss of lube oil, etc. - thaqt requires slower warmup times or repairs before startup can resume.

The recent SF electric loss was a linked chain of failures from one transformer tripping. The weeks long gas and power failures in NZ and Australia were from one failure each time. The latest natural gas line failure yesterday cut out services to large areas of WA (?) and several smaller cities.

-- Robert A. Cook, P.E. (Kennesaw GA) (cook.r@csaatl.com), January 04, 1999.

I am the one tasked with the embedded project for a large utility. Reading the posting my reactions are:

1. The findings should not have been a big surprise. Each of the functions mentioned uses time so was a possible problem. The telephone problem was not specifically described but there are known problems with a few switches and other devices. The meter problem is somewhat unique in that I haven't heard of anybody using a meter to feedback into a turbine system but the newer meters do have more capability so it doesn't surprise me too much to see that somebody did. The fire suppression system problem is described as a cpu problem, these have been seen mostly as a BIOS-RTC problem that in extreme cases (few and far between) cause the cpu to lockup. I would look at doing a BIOS changeout first, then if that is not possible or the problem is different than I am assuming, changeout the system. Again, though, a fire system problem is common. I've seen several facility fire/hvac/security systems that have problems and this is an area that has been flagged as troublesome. What does surprise me is the statement that the gas turbine would shutdown violently. This is either a terrible design or an incorrect statement as loss of interlock signals such as this normally cause a system trip, which is not violent. Perhaps he is talking about the temperature transient which would be large but something the turbine is designed to do. I need more information before I would believe this.

2. The large utilities, those controlling the transmission and distribution systems, have been working on Y2K embedded problems for most of 1998. Problem areas and equipment are known. I would look at your DCS and CEMS as well as any recorders you may have. In your facility I would look at phones, radios, lans, security, and fire protection systems. A large body of data is being gathered and shared through EPRI. Get in contact with the larger utility you connect too and discuss what you are doing. I have a hard time believing a municipality would say they are doing nothing if they own generating equipment, if they are only doing transmission and distribution it is possible, I've heard of no problems that would cause the transmission or distribution systems to crash due to internal problems.

3. NERC, through the regional reliability councils is conducting a survey of Y2K readiness of IPPs that should be complete within a month. Larger utilities have been supplying readiness information since August, 1998. This means that there will be little surprise. Those plants not doing anything will be planned for. My utility is already planning how to handle plants that have done nothing and ensuring we have enough generating capacity that is Y2K ready.

4. I would appreciate more information on these problems and would be willing to reciprocate in kind.

-- Murray E. Jennex, Ph.D., P.E. (jennexme@sce.com), January 04, 1999.

I'm assuming the nuclear power plant being referred to in New York State, is the one in Oswego? This plant is known to have a cracked [taurus?] not sure of the name of the part. Would problems with the crack, in conjunction with Y2K problems, force the shutdown of the plant? I HOPE SO! Max (40 miles downwind from the Oswego plant]

-- Max (etendu@yahoo.com), January 06, 1999.

"Earlier today I read information referring to the power supplies in Texas. It stated that outages in Mexico can result in the power in Texas to be brought down because of the grid connections...adios amigos."-Terri

Same goes for the northeast states Terri, power outages in Canada could result outages in the east coast. The USA's electric grid is interconnect with both countries on either side.

-- Chris (catsy@pond.com), January 06, 1999.