From ftp://ftp.nerc.com/pub/sys/all_updl/docs/pressrel/NERC_AT&T_BellAtlantic_Interoperability_Test_Press_Release_8-4-99.pdf

The North American Electric Reliability Council (NERC), AT&T, and Bell Atlantic Corporation today announced the successful completion of interindustry verification testing on communications and electric power control equipment. The tests on telecommunications network and electric utility equipment validate the premise that date-time transitions in the new millennium would not affect the ongoing operations of the electric power or telecommunications industries.

The test participants include AT&T, Bell Atlantic Corporation, Central Hudson Gas & Electric Corporation, New York Power Authority, New York Power Pool, PJM Interconnection, L.L.C., and Public Service Enterprise Group.

All of the equipment tested had previously been subjected to extensive testing by each participant, and the value of this testing was to verify interoperability among the various systems and to test end-to-end continuity of service.

We are confident that, as a result of these tests, the telecommunications facilities we need to operate the electric systems across North America will be there when we need them, and the lights will be on to usher in the new millennium, says Michehl R. Gent, President of NERC. The type of equipment included in the electric utility tests include end-element metering devices, supervisory control and data acquisition systems, remote terminal units, multiplexor systems, communications network processors, routers, network time protocol servers, and energy management systems.

We consider Y2k preparation a cooperative effort, not a competitive activity, said John Pasqua, Vice President, AT&T Year 2000 Program. Interoperability testing such as this gives our customers added confidence that the electric power systems and telecommunications networks will continue to function normally when the new year arrives.

Bell Atlantic is pleased to participate in such an important, joint-industry effort to corroborate the Y2k readiness and interoperability of telecom and power company systems, said Keiko Harvey, Vice President of Central Office and Interoffice Facilities Engineering for Bell Atlantic. Bell Atlantic recognized early on in the Y2k effort the special relationship that exists between electric power providers and telecom providers like Bell Atlantic. We each rely heavily on each others services and this type of testing helps to increase the high level of confidence we each have in those services.

Although the tests were conducted in the Northeast, the equipment tested is representative of that used nationwide by both the communications industry and the electric power industry.

All tests were executed in accordance with the plans by the participant companys personnel. The test teams initiated nearly simultaneous date-time transitions in all equipment under test, and conducted post transition evaluations of the data being transmitted and received. Overall, the equipment and services tested functioned normally through the various date-time transitions in all test scenarios. The participants concluded that for the tested systems and dates the AT&T and Bell Atlantic communication equipment and the electric utility monitoring and control systems would interoperate correctly in the year 2000.

More information about the telecommunications interoperability verification tests is available in the report entitled Year 2000 Interoperability Test Evaluation Report. A copy of the report is available on the web sites of the following test participants: NERC  http://www.nerc.com/y2k AT&T  http://www.att.com/year2000 Bell Atlantic  http://www.bellatlantic.com/year2000/

Details of the tests can be found at ftp://ftp.nerc.com/pub/sys/all_updl/docs/y2k/NERC_AT&T_BellAtlantic_Interoperability_Test_Report_8-4-99.pdf

-- Hoffmeister (hoff_meister@my-deja.com), August 04, 1999

Answers

(DR EVIL)

Hoff, throw me a frickin' bone here.

(/DR EVIL)

From NERC report, 8/3/1999, page 23 The electric industry owns and maintains a majority of its voice and data communications facilities. However, external service providers are used for a significant portion of voice and data communications. These external providers may be local telephone carriers leasing dedicated circuits to carry monitoring and control signals to power plants and substations. They also may provide long distance services, satellite systems, cellular systems, and wide-area networks. The electric industry, like many other industries, is dependent on a complex set of integrated communications systems.

Most entities report satisfactory progress in testing their internal communications systems, as reported above. Like EMS/SCADA and DCS systems, communications is an area that often requires support from vendors. Entities report making Year 2000 upgrades on older network equipment (e.g. routers, hubs, and switches). Often, testing procedures or results have been achieved with the assistance of or information available from equipment vendors.

It is apparent that extensive integrated testing with external voice and data communications service providers is not practical. Typically, these service providers are working hard to complete their own program and cannot dedicate substantial resources to joint testing with individual customers, including electric utilities. Also, these service providers typically cannot provide live circuits for end-to-end testing with electric systems, leaving most testing for the laboratory.

So, which is it, Hoff? NERC can't have it both ways.

-- Inquiring Mind (throw.me@frickin.bone), August 04, 1999.

whatever. for the 3rd time in about 4 months, when i try to call long- distance i get a recording saying the number i'm calling FROM is disconnected. the 1st 2 times were with an MCI subsidiary. this time i'm with AT&T. the MCI place told me they were having some problem with their carrier. i'm beginning to hate all of them.

-- sarah (qubr@aol.com), August 04, 1999.

Not a telecommunications expert, Doc.

But from reading the report, it appears they used production services to connect the utilities to AT&T and Bell Atlantic Test Labs, where the actual equipment and services were tested. The report gives a fairly detailed layout of the test environments.

-- Hoffmeister (hoff_meister@my-deja.com), August 04, 1999.

I don't understand the relationship between this test and the fake test they're running on September 9. Do they invite the press for the fake test?

-- Dog Gone (layinglow@rollover.now), August 04, 1999.

About time.

Now that they have checked the operability for one part of one city, why do they claim this "proves" the whole country is compatible?

<>

"Industries"? This is one little area only.

____

But - how could they loudly declare themselves "fully compliant" at the end of June, if they haven't gotten around to testing interoperability until the first week in August?

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), August 04, 1999.

Now - actually, all they can really conclude is included in the report - but as usual, it is not in the "headline" nor will you ever see it in the sound bites quoting by Mr. K:

<>

Again - what they tested, under the conditions they tested it, appears to work according to how they tested it.

It's good news - but the grid isn't providing power next year yet either.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), August 04, 1999.

The more I wade through the forum, the more it's deja vue all over again (love this english oxymoron.)

A snipet taken from Kevin's post in March http://greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000WMz:< p>

[snip] 

    Background: 

    The electric industry in North America is preparing to conduct a 
Y2K drill on April 9, 1999. The purpose of the
    drill is to prepare for operation with limited voice and data 
communications. A drill guide has been prepared
    defining the objectives and steps to prepare for and conduct the 
drill. 

    1. The April 9 drill is intended to install public confidence 
through success and at the same time be a real test of
    our ability to operate with limited communications capabilities. 
How can these two goals be balanced to provide
    the greatest value from the exercise? 

    * Start planning for it now 

    * Prior to drill, test system(s) that will be exercised during the 
drill. 

    * Document drill procedures. 

    * Identify system(s) to be drilled. 

    [snip] 

    * Verify that there are no real security issues during the time of 
the drill 

    [snip] 

    Individual companies who choose to should develop their own plans 
to notify and coordinate with their local
    media 

    [snip] 

    * Do not make the drill to complex. We want to have a successful 
and meaningful story for publication. 

    [snip] 

    * The drill should test partial loss of voice and data 
communications and partial loss of EMS functionality. 

    [snip] 

    * Do not overload communications channels with drill 
communications. Identify specific phone numbers that will
    be used during the drill. 

    [snip} 

    * Was NERC aware that April 9th is a real Y2K sensitive date? 
Answer: Yes, April 9th is the 99th day of 1999. 

    [snip] 

    -- Kevin (mixesmusic@worldnet.att.net), February 20, 1999 

Answers

    An important snip I forgot to put in the last message... 

    [snip] 

    * What will the final report look like. Work backwards from this 
in the development of the drill procedures. 

    [snip] 

If they were putting us asleep back in March, why would it be any different now?

-- Chris (%$^&^@pond.com), August 04, 1999.

Robert, do you understand the Y2K process and the difference between remediated and tested? This kind of testing has never, ever, been done anywhere for any system put into production. They have done more than they normally do for software development to further reduce risk. That's not good enough? Why? Why do you believe that this doesn't prove anything? What would be different in some other little town using the same equipment, besides the name of the town? The "proof" you're looking for is impractical, exponentially expensive, unnecessary and you'll never get it.

-- Maria (anon@ymous.com), August 04, 1999.

Hoff, just want to thank you for all the digging/posting of on topic information. Seems to give Mr. Cook a reason for popping up every now & again!

Best Wishes,

-- Bingo1 (howe9@pop.shentel.net), August 04, 1999.

No, Maria, its obvious that YOU will never get it. Until next year, that is. (And even then, I wonder. "Gee, thats funny, my phone is dead. Must be a fluke -- a line down or something. Gee, thats funny, no lights. Must be fluke -- a squirrel ate a transformer or something. Gee, thats funny, sewage bubbling up from my sink. Must be a fluke -- too many people barfing into toilets after all the New Year parties to welcome in the bright new century of hope, prosperity, and happiness. Gee, thats funny....")

-- King of Spain (madrid@aol.com), August 04, 1999.

Point of order, Mr Speaker, point of order.

The honourable member for the Iberian Pennisula has breached guidelines set out by this House to ensure debate is free from gratuitous personal attack. I move that he be removed from this House until such time as he sees fit to debate in a manner befitting the guidelines under which we all work.

[Low rumbles of "Hear, hear" are heard from the back benches]

Our next debate refers to Bill C134 "An Act to Affirm Mud-Wrestling as the National Pastime of this Dominion"........

-- Johnny Canuck (j_canuck@hotmail.com), August 04, 1999.

Johnny: LOL dude!

-- King of Spain (madrid@aol.com), August 05, 1999.

Milady Maria -

It has been established that there are frequent and unpredictable differences in manufactor, in sources (country of origin) and in response to simulated y2k data even between identical controllers (processors and sensors) purchased with the same part number from the same vender.

Change the year of purchase, the installing company, the frequency and degree of maintenance, and the upgrades/conversion/replacement of systems, and you quickly (within two-three years) get a "mixed" system of new parts, replaced parts, and original parts.

In water systems, even more than power (where venders for SCADA and 120,000 vac systems are few in number and less likely to change randomly) each "system" (defined from "water in" all the way through "water out") is going to response completely different.

Some even will be so "old" they won't notice (other than power loss - if it occurs). Others, more at risk, are the ones built in the 1880's and 1890's, renovated in the 10's-20's-30's-40's-50's....amd partially or fully automated in the 80's and 90's. Only those with no electrical interfaces or controls or billing and payroll and tax and monitoring systems at all (none ?), or those built entirely from scratch since 1996 (none) are not at risk.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), August 05, 1999.

To continue, more towards power/phone interoperability in the big systems. (Smaller phone/utility interoperability is more akin to water systems, larger are more likely to have been gradually upgraded in the 80's and 90's, and a massive change in the distributed systems as fiber optics came in to form in the mid-90's.)

Same conclusion though - each system (as installed and as in actual operation) is different, and may or may not respond the same way that the modelled system in the "lab" responded.

You're right - this form of testing I "require" is expensive. The penalty for failure to succeed is even greater.

Think of the whole process involved: what the big phone companies appear to have done is exactly right and they are following the only path they realistically can follow. (In that we agree completely - if I understand your previous postings.)

BUT - the difference is: you think that because a system has worked in the lab that it will work in the field in all (most/some/many) circumstances. My obsevation (based on a lot of actual trouble calls and field support trips) is that actual field installations are much more difficult to actually get operating. What we still have not been told is whether the remediated programs and control processors have actually been implemented in the ALL of the major and minor phone systems - that is, are the phone systems and utilities actually RUNNING today the programs that have been "tested" in the labs?

If they have, good. It means the revised programs actually work and are installed. IF they have NOTactually been "reinstalled" and actually placed in service, the administration is lying again. Their supposed "fixes" are actually "vaporware" or "demoware." (Demonware ?)

(This is the status of the FAA now - they have programs that appear to work, but have not gotten finishewd installing them yet, nor training the operators in how touse them yet.)

ONCE the revised programs have been actually installed and actually proved to be running correctly now, then we can beginning trying to find out if they will run next year. This "test" - for this one utility group and this one phone system - indicate that the revised program they tested might run.

This conclusion cannot be extrapolated to other programs and other installations and other cities. Not safely extrapolated, that is.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), August 05, 1999.

Robert, as for the status of implementation at the phone companies, my guess is we'll have a better view later today. The NRIC is holding a conference right now presenting the current status.

This is the second time you've made the claim about the FAA. I asked you about a week ago to back this up; you didn't. I'll ask again. What evidence do you have that the FAA has not implemented Y2k ready systems?

-- Hoffmeister (hoff_meister@my-deja.com), August 05, 1999.