Full PDF at: ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/4-30-y2k-report-to-doe.pdf

Executive Summary

Background

This report is the third in a series of comprehensive quarterly status reports on preparing the electric power supply and delivery systems for operation into the Year 2000 (Y2k). The North American Electric Reliability Council (NERC) is facilitating this Y2k Readiness reporting process in response to a May 1998 request from the United States Department of Energy (DOE). Previous quarterly progress reports were delivered to DOE on September 17, 1998 and January 11, 1999.

A final report describing the readiness of electric systems to operate into the Year 2000 will be delivered in July 1999, as requested by DOE. Subsequent updates after July will focus on tracking any remaining issues from the July report and review of operational preparedness activities.

Results from the First Quarter 1999

Key Result 1: Substantial Progress in Remediation and Testing  The owners of electric power production and delivery systems in North America made excellent progress during the first quarter 1999 in the arduous task of testing and, as needed, repairing or replacing mission-critical components that may be susceptible to Y2k. Remediation and Testing progress has jumped on average from 44% complete as of November 30, 1998 to 75% complete as of March 31, 1999. The electric power industry has made a strong commitment to meeting the NERC-established target date of June 30, 1999 and most facilities will be ready by that date. The industry is applying extensive resources and its best people to methodically address the Y2k issue.

Key Result 2: Minimal Operational Impact Discovered  With more than 75% of mission-critical components tested through March 31, 1999, findings in the field continue to indicate that the transition through critical Y2k dates is expected to have minimal impact on electric system operations in North America. It is estimated that fewer than 3% of items tested before remediation have any difficulty at all with Y2k date manipulations. The types of devices that do experience trouble with Y2k date manipulations exhibit mostly nuisance errors, such as incorrect date displays and date-time stamps used for data logging and reporting. In most cases, Y2k does not affect primary device functions related to keeping generators and power delivery facilities in service and electricity supplied to customers.

Key Result 3: Y2k is Real  Despite the good news that interruptions of electric service due to Y2k appear unlikely, Y2k issues are real and require completion of the ongoing comprehensive testing and remediation programs. In power generators, the types of devices that may exhibit date manipulation difficulties include Distributed Control Systems (mostly date display problems), computer controllers, data loggers, alarm systems, emissions monitoring, and precipitator controls. In power delivery systems (transmission and distribution) the types of devices that may exhibit date manipulation or display difficulties include EMS and SCADA 1 systems, protective relays (time stamp, not primary protective functions), digital electric meters, event recorders, and substation controllers. In the area of telecommunications, issues may exist in routers, PBXs 2 , and fax machines (date stamp). Heating and ventilation systems, as well as physical security access systems may also be affected. The bottom line is that Y2k must and is being taken very seriously by the electric power industry.

Key Result 4: Y2k Contingency Planning  Although the impacts of Y2k are expected to have minimal effects on the ability to reliably operate electric power systems, the industry is taking proactive steps, under its defense-in-depth strategy, to prepare for possible operating contingencies. NERC and its ten Regional Reliability Councils reviewed first drafts of Y2k operating plans in January 1999. The results from this review indicate that operating entities are taking the appropriate steps to evaluate Y2k operating risks and assure that contingency response plans are suitably adapted to Y2k conditions. Y2k operating plans, including contingency response strategies, are to be ready by June 30, 1999, and will continue to be refined through the remainder of the year.

Key Result 5: Y2k Drill  On April 9, 1999, the electric power industry conducted a first-ever North American drill simulating the partial loss of voice and data communications. This drill involved several thousand personnel at more than two hundred electric power organizations using backup radios, satellite voice systems, and other backup systems to simulate safe and reliable operating practices with loss of primary communications. Although the drill is a successful milestone for the industry, there are lessons learned that will be addressed prior to future drills. Some of the lessons learned include the need to: expand the geographic coverage of radio systems, establish priorities to address heavy voice traffic on some systems, provide further training on satellite voice systems, and practice voice communication protocols. A second, more comprehensive NERC drill to prepare for operations under Y2k conditions is planned for September 89, 1999.

Critical Issues

Issue 1: Some Facilities Y2k Ready after June 30, 1999  As was recognized in the previous report to DOE, some facilities may achieve Y2k Ready status after June 30, 1999. Some organizations may not complete the final few items on a generator until a scheduled maintenance outage in the fall of 1999. In other cases, vendor upgrades may not be available until later in the year. To address this issue, NERC initiated an exception reporting process in January 1999. The exception reporting process allows NERC to precisely identify the number of facilities affected and the amount and schedule of Remediation and Testing work to be completed after June 30. Identification of these exception details has had two desired effects in the first quarter of 1999:

1. Many utilities were able to modify their outage schedules and accelerate their Y2k work, thus reducing the number of items on this exception list between January and March 1999; and

2. NERC has been able to confirm that the remaining items on the exception list, which may not be candidates for acceleration, are a very small subset of overall system capabilities and do not pose operational risks to electric system operations into the Year 2000. Details of the exception reporting process are provided in Section 3 and Appendix A.

The exception reporting process was applied effectively for the 18% of organizations reporting one to a few exceptions. There remains about 10% of the organizations reporting to NERC that have expected Y2k Ready dates between June 30, 1999 and October 31, 1999. NERC is notifying these organizations of the need to achieve a Y2k Ready status for mission-critical electrical systems by June 30, 1999 or identify specific exceptions for NERC assessment.

Issue 2: Readiness of Distribution Systems for Y2k  Distribution systems are less susceptible to Y2k issues and the data provided in this report indicate a high degree of awareness and responsiveness by distribution systems. At the same time, a portion of distribution systems intends to complete Y2k testing and remediation after the June 30 industry target. About 42% of public power organizations are reporting they expect to be Y2k Ready after June 30, 1999. About 35% of cooperatives are reporting expected completion dates after June 30, 1999. All of these organizations are reporting they expect to be Y2k Ready by December 1999 or earlier. The readiness of distribution systems will continue to be a focus area for the NERC process.

Issue 3: Independent Verification of Self-Reported Data  The data used by NERC and its trade association partners to assess the readiness of electric systems is principally self reported. Interviews with Y2k program managers have been encouraging because they lend credibility to the reported data. Over half (53%) of the bulk electric systems reporting to NERC indicate use of an external contractor to audit their Y2k program. Another 29% report use of an internal audit function to review the Y2k program. All nuclear facilities are reported to be subject to an external review. DOE has been providing an external review of federally owned electric power facilities and many state utility commissions are conducting reviews of electric utility Y2k readiness programs. To reassure the public of the validity of the data reported to NERC, DOE is proposing to sponsor additional independent reviews of a representative sample of systems.

Issue 4: Public Communications  The first quarter of 1999 witnessed an increase in public awareness of the Y2k issue and increased scrutiny by the press. The NERC-facilitated April 9 Y2k drill, for example, was widely covered in television, radio, and newspapers. Being at the center of attention for Y2k provides an opportunity to share with the public what the electric industry is doing to methodically address the Y2k issue and how electric systems will be that much improved as a result of this thorough review. Each utility needs to be proactive in communicating the results of its Y2k efforts to its customers.

Issue 5: Interdependencies with Telecommunications  This issue was raised in the previous report, but is so important that it must remain a top priority for the electric industry. Operation of electric systems is highly dependent on voice and data communications, some of which are operated by external service providers. The dependence on voice and data communications directly affects real-time operations and control of electric systems and, therefore, requires the greatest attention in contingency planning and preparations.

The electric industry has taken several steps to address this key vulnerability. First, interindustry meetings are being held at the national, regional, and local levels to share contingency plans. These meetings include representatives from electric, gas, oil, telecommunications, and transportation industries. Second, NERC and others are performing integrated electric system and communications system testing, in cooperation with several telecommunications service providers. Third, the electric industry conducted a drill, which spanned North America, on April 9 to practice operations with limited voice and data communications and will perform a follow-up drill on September 89, 1999.

Continuing Industry Efforts

This report updates the industry work plan for continued coordination of Y2k efforts across North American electric systems:

1. NERC and its Regional Reliability Councils, in a cooperative partnership with several trade associations, will continue to facilitate electric industry preparations for Y2k. These efforts include ongoing readiness assessments of all sectors of the industry and sharing of information.

2. The NERC Y2k readiness assessment process will continue to track exceptions to the June 30, 1999 readiness target and periodic updates will be provided to DOE on the status of those exceptions. NERC will closely monitor the progress of any organizations completing Y2k readiness after the June 30, 1999 target.

3. NERC will cooperate with DOE in the conduct of independent, onsite reviews of Y2k readiness status at a sample of electric power organizations to validate the self-reported data received by NERC and its partners.

4. NERC and its ten Regional Reliability Councils will continue to review operational and contingency plans, with a goal of having plans initially ready by June 30, 1999.

5. The industry will conduct a second Y2k drill on September 89, 1999 to rehearse Y2k procedures, communications, and contingency response.

6. NERC will continue to coordinate efforts with the telecommunications industry to better prepare for interdependencies between the telecommunications and electric industry.

..end summary...

-- Hoffmeister (hoff_meister@my-dejanews.com), April 29, 1999

Answers

Thanks, Hoff. Just skimmed it, but will give it more time later.

Key Result 3: Y2k is Real  Despite the good news that interruptions of electric service due to Y2k appear unlikely, Y2k issues are real and require completion of the ongoing comprehensive testing and remediation programs. In power generators, the types of devices that may exhibit date manipulation difficulties include Distributed Control Systems (mostly date display problems), computer controllers, data loggers, alarm systems, emissions monitoring, and precipitator controls. In power delivery systems (transmission and distribution) the types of devices that may exhibit date manipulation or display difficulties include EMS and SCADA 1 systems, protective relays (time stamp, not primary protective functions), digital electric meters, event recorders, and substation controllers. In the area of telecommunications, issues may exist in routers, PBXs 2 , and fax machines (date stamp). Heating and ventilation systems, as well as physical security access systems may also be affected. The bottom line is that Y2k must and is being taken very seriously by the electric power industry.

I believe this is the first such detailed list that I've seen.

Issue 3: Independent Verification of Self-Reported Data  The data used by NERC and its trade association partners to assess the readiness of electric systems is principally self reported. Interviews with Y2k program managers have been encouraging because they lend credibility to the reported data. Over half (53%) of the bulk electric systems reporting to NERC indicate use of an external contractor to audit their Y2k program. Another 29% report use of an internal audit function to review the Y2k program. All nuclear facilities are reported to be subject to an external review. DOE has been providing an external review of federally owned electric power facilities and many state utility commissions are conducting reviews of electric utility Y2k readiness programs. To reassure the public of the validity of the data reported to NERC, DOE is proposing to sponsor additional independent reviews of a representative sample of systems.

"Representative sample of systems"? Who decides which utilities get audited?

More later....

-- regular (zzz@z.z), April 29, 1999.

From Rick Cowles forum:

NERC has placed the latest report to DOE online (24 hours ahead of when it was scheduled to be there).

(Provides same link as Hoff)

Of course, as usual, you'll need the Adobe Reader to view it.

And also, as usual, I'll provide some useful/useless (depending on your point of view) commentary after I've had the opportunity to review it.

-- Rick Cowles (rcowles@waterw.com), April 29, 1999

-- FM (vidprof@aol.com), April 29, 1999.

(sound of slapping head)

I cannot *believe* I hadn't thought of PBXs as being a Y2K/power issue. Duh. And double duh.

-- Drew Parkhill/CBN News (y2k@cbn.org), April 29, 1999.

Generally good news - detailed enough in the specifics to confirm the other data availble: this means that the "interface problems" and support systems that were excluded in earlier reports (from NERC and DOE) where gaping "black holes" of information existed are better filled in. Think of it as a former "tank trap in the road of life" that has been partially filled in - now, it NOT a bump in the road yet (as far as only power goes), but now it more like a simple open trench. You still can't drive through (yet) - but at least you can see the bottom of the trench, and the bottom appears to be slowly filled in.

Second point for discussion - note the more honest reporting of schedule finish dates. true, they are claiming all done by december, but are no longer blatantly claiming a rosy picture of completion by mid-June. Instead, some in June, more in July - Aug, about the same in Set - Oct, a few EMC's finishing as late as Dec.

This slipping is not in itself good news - it puts too many people at risk if any other problems occur - but it is more realistic. (Means most likely somebody up there got tired of being lied to.)

Drew: One early symptom of PBX trouble was reported from the IRS - they "set ahead" the system in one eastern office, and had no problem for three days. Then it shutdown completely and totally on the fourth day. Why four days? Don't know - computers fail in wierd and wonderful ways.

This report COULD be the reason why Mr. K is suddenly very, very aprehensive about overseas markets and power and infrastructure. (Obviously, he gets the thing before we do.) This report makes it very clear that if you do NOT remediate and test successfully, your process will break. No waffling words at all about that fact.

Score one for Yourdon last year - the precentages are tracking about as he forsaw then. But overseas, there is apparently too little real progress for even Mr. K to pretend the situation will be rosy.

A worrisome thing to note. Electric utilties are not the only game town, though they are important. You can assume that MOST factories and businesses that build and machine things are more highly automated and more threatened than utilities. (GM and Northwest Airlines, for example, are spending about double what they originally budgetted.

But too few businesses are close to finshing to even pretend most will succeed. the 80% success rate in the US may drop to as little as 45-55% if things don't change quickly. Overseas? Could be anything from 25-65%, depnding on how much is automated, and how much is manual labor.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), April 29, 1999.

Robert, I don't see the "slippage" you mention, in regard to completion dates.

All of the previous spreadsheets indicated numerous instances of dates later than Jun 30th. We could do a comparison, but I don't think any significant changes have been made to later dates.

Maybe I'm missing something, but what "slippage" and "lying" are you referring to?

-- Hoffmeister (hoff_meister@my-dejanews.com), April 29, 1999.

The big three... power/water, telecommunications and financial services. This triad is a bellweather. Progress in theses areas is the most positive of Y2K news.

Regards,

-- Mr. Decker (kcdecker@worldnet.att.net), April 29, 1999.

Thanks for the summary, Hoffmeister. It does appear to demonstrate significant progress.

Bummer I won't get to read it until this weekend...I'll provide my views as soon as I digest the document.

Glad to hear you'll be commenting on it, Mr. Cowles. As I mentioned on an earlier thread, are you interested in debating the report with myself (and any others who want to dicuss) on this forum?

-- Dan the Power Man (dgman19938@aol.com), April 29, 1999.

Key Result 3: Y2k is Real ...Duh...

-- Mike Lang (webflier@erols.com), April 29, 1999.

Hey Hoff,

I don't know if you were around the Y2K thingamajig in 1997 or 1998, but originally everyone was saying, "Done by Dec. 31, 1998, leaving a full year for testing." This, my friend, is slippage.

-- Jim the Window Washer (Rational@man.com), April 29, 1999.

From above:

<>

Only 42% ready by the deadline they expected to meet as recently as early Feb. This is slippage (which I expected), but it is "realistic" values: you'd natturally expect a very few early, more on time, and a large amount late. With software projects, a very few early, some on time, most late by varying amounts.

But "they" have (in the past) blantantly claimed "full compliance" and "completely ready" and "fully confident" and the like. Those words and phrases are meaningless and "lies" - these schedules are at least realistic-sounding. The actual completion may still slip from these given dates, but at least "they" are now admitting that most won't finish by the "deadline"

By the way, 43% scheduled by June, plus the 35% ready between Jun and Dec leaves ??? unaccounted for. When are these going to be done, and who are they?

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), April 30, 1999.

Must have missed where "they" claimed they would "all" be ready by June. Funny, every spreadsheet I've looked at since November has had many dates past June.

Also must have missed where "they" claimed to be "fully compliant".

And the 43% and 35%? Seems like you're misreading the numbers here. Both are statements regarding different subjects being ready after June. Again, my reading must be off.

-- Hoffmeister (hoff_meister@my-dejanews.com), April 30, 1999.

To you electric Guru's from a definite none-power enlightened person (I know that if I turn the switch and the light comes on - the power is on and the bulb is good)

If, as Mr Cooke says, 43% will be done by June, and 35% done between June and December, that leaves 22% where? Now, how much power production loss does it take to pull a grid section down? 2%? 5%? And, IF once section goes down, will that pull the whole grid down? Can one single section of the grid stay up if the other 2 sections are down? I live in Western Washington. We get most of our power from BPA who seems to be pretty compliant (they say). Our local power company Puget Sound Energy says they only produce 32% of their own power so they are buying 68% of their power mostly from BPA but some from Canada. If the NE grid goes down, can/will they take some of the power from out here to keep the other Washington (DC) up and running so that we are left with minimal power? I guess what I am asking is how detailed can the power that is available be directed to points far from where it is produced? If a grid is down, can power still be funnelled to population centers within that grid to "keep things from running amok"?

Thanks in advance - power is most worrisome to me since everything we do seems to be connected to power, from food to gas to water to everything else. If the power stays on, I think we will be OK even if it takes a while to fix everything. Without power - no spare parts and no way to manufacture more and bad things will go on for a much longer period of time.

-- Valkyrie (anon@please.net), April 30, 1999.

Well, I think that's a big "IF". I don't know where Robert got those numbers. Again, must be bad reading comprehension.

-- Hoffmeister (hoff_meister@my-dejanews.com), April 30, 1999.

I see what you mean: the 42% are those public companies that will be ready after June, and the 35% is the percent of co-op's ready after the June deadline.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), April 30, 1999.

I have 2 questions/comments. May need to repost at euy2k if noone here knows the answers.

1. At the bottom of p. 32:

Of those APPA respondents who have tested critical components, over 60% indicate results that show Y2k would not cause interruption of electrical service.

Am I correct in interpreting this to mean that nearly 40% would have service interruptions?

2. In the Appendix, over 20 CEMS (Continuous Emissions Monitoring Systems) Components are listed as Exception Items. All justifications are for "Vendor availability". The expected date of deivery (?) ranges from 6/30 to 11/1, with most indicating September or October. Is this delivery schedule doable? What about installation and testing? Is enough time allotted? Are there multiple vendors for these systems?

Sorry, that's more than 2 questions. Anybody have any answers?

-- regular (zzz@z.z), April 30, 1999.

Interesting spin.

-- regular (zzz@z.z), April 30, 1999.

I posted the same questions at the euy2k forum, if anyone cares to read some of the responses.

-- regular (zzz@z.z), April 30, 1999.