>From Wired Magazine

http://www.wired.com/wired/archive/7.04/texaco_pr.html [W I R E D] [Image]Archive | 7.04 - Apr 1999 | Feature

This Is Not a Test

For Texaco's millenium commandos, the war against Y2K is being fought one RTU at a time.

Back in July 1998, Jay Abshier, Texaco's Year 2000 project manager, wanted to alert people to Y2K. "I want to show that Y2K is not a hoax," he said on the phone. Six months later, he wanted to calm everyone down: "Now it's gone too far the other way, into overhyped fears."

Abshier is one of the very few corporate Y2K managers who wanted to speak on the record. He had an agenda, of course - "I want to show that Texaco is doing a good job," he said. But his larger goal was to deliver what surely is the most difficult message to convey about Y2K: that Year 2000 problems are real and may indeed be locally severe; and also that hard work, engineering good sense, and intercompany cooperation can minimize the damage.

To prove his point, he invited me to Texaco's Stormac center in New Orleans, a control facility that monitors 32 offshore oil platforms in the Gulf of Mexico. During storms, workers are evacuated from the platforms, and the center is used to control the rigs remotely. The center's controls exactly duplicate those on the offshore platforms, providing the Year 2000 team with an ideal Y2K laboratory. (The control center's name is an acronym that stands for System for Texaco Offshore Remote Monitoring, but one suspects all those words are just an excuse for the impressive-sounding Stormac.)

Abshier turned out to be a nicely suited man with a weary yet patient manner. He has worked for Texaco for 18 years; he wrote code for control systems for eight years before becoming a manager. He took over as the Y2K manager in 1997, two years after Texaco started its Year 2000 program. He brought me into a small office, where he introduced Robert Martin, another 18-year Texaco veteran, and Fred Cook, advanced technician at the center, who has 19 years with the company. Martin and Cook wore chinos and polo shirts with Stormac logos. They spoke with broad Southern accents.

The small office was Martin's. Lying flat on the desk - positioned for viewing from the visitor's seat - was a large gold-metal cross. Later, I asked Abshier about that part of Y2K - the religious fervor, the millennial expectation of apocalypse. "I'm religious," Abshier said, "and a lot of people on my team are, too. I have a couple emails asking me, 'Is this the end of time?' Well, if you subscribe to the Christian belief that there is an end of time, it also says that no one knows when it's going to happen. So I say no, Y2K can't be the end of time - it's too obvious."

After we chatted for a while, the four of us went into the center's machine room, a windowless box filled with hardware, cables, and the hot air blown out the back of electronic equipment. I asked them if they had any idea of all the embedded code running in all that gear. Martin laughed, ruefully. "Oh yeah. We know. We wrote most of it ourselves." As there was no storm on that particular morning, the machine room was mostly given over to the test Abshier had invited me to see, a re-creation of one of the first tests Texaco had run on an embedded system.

The precise embedded system to be tested was a remote terminal unit, or RTU. An RTU is something like a small, single-purpose computer, the Stormac team explained. In a paperback-sized box mounted on the wall were several integrated circuit boards, each containing chips with embedded logic. Unlike programmable logic controllers, or PLCs, which can contain complex programs to control industrial processes, an RTU is fairly primitive, usually confined to doing one task. This one measures the flow of liquids and gases through a pipeline. Simple as its work sounds - it measures the instantaneous flow rate, stamps the measurement with a date and time, and stores it temporarily in its internal memory - it's a crucial piece of gear for Texaco. This little box is how it knows how much fuel it's delivering through its pipelines - and how much to bill the customers who are getting that fuel.

This RTU is just one small data-collection point in a wider universe of intelligent devices that communicate with a centralized computer system. Via microwave, hardwire, and radio, hundreds of devices like this one are constantly sending data to the Supervisory Control and Data Acquisition system.

The Scada host computer sat on the other side of the machine room - nothing exotic-looking, just an Intel-based PC with specialized OS and software. But the Scada system is the heart of Texaco's embedded-system network. If it can't collect data from the field devices, the company has no idea what's going on in its operations, can't analyze its production, can't bill customers - can't function as a company. By law, if Texaco loses contact with its field devices, it shuts down in four hours. Right at that moment, the Scada system was polling hundreds of embedded-system devices, collecting and storing about 30,000 points of data.

Cook attached a laptop to the RTU, which gave him a direct interface to the logic in the device. He was, of course, about to do the one thing everyone wanted to do: set the date on the device to December 31, 1999, wait for the year to change, and then see what would happen.

Using a handheld interface terminal, he entered the date and time: 12/31/99 23:59:45.

Then we all watched the display on the face of the RTU as the seconds counted up to midnight. 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59 - then the date rolled over.

01/01/:0.

"Colon zero," said Cook. "It's like, what is that?"

Then he tried entering the date 12/31/00. Again the seconds counted up to midnight, this time to 01/01/:1.

But nothing terrible seemed to happen. No flashing lights, no buzzers, no equipment shutdowns. Was it just a weird date-format problem? A lot of hype over a display? Cook then took me over to the terminal for the Scada system and tried to collect information from the RTU. He entered the command to retrieve the device's idea of the date and time, and the Scada console displayed:

01/01/101

Then he tried to retrieve the crucial information from the device, the date-stamped flow measurements stored in the unit. And the Scada system answered:

METER DATA NOT AVAILABLE - CONTRACT HOUR NOT CURRENT

"It can't get the data," Cook explained.

Gas and oil continued to flow, unmonitored and unmeasured. If you can't read the data, said Cook, "you don't know what you've sold, and you can't get paid for it." How long could Texaco continue to function without being able to bill for the oil and gas delivered through its pipelines? Abshier and Martin looked at each other and let the question go.

Texaco has hundreds of RTUs like this one out in the field. Fixing the devices involves going out to each unit, changing the chips inside it, and installing new software - about an hour's work per unit. The first round of replacement chips the RTU vendor sent them didn't work; they had to wait for another. Then the Scada system needed upgrading. And that was just for this one device. There are all those other devices in the field, with their chips and their embedded logic - setting valve positions, measuring pressure - hundreds of them.

Even so, Martin and Abshier were reassuring. In the face of serious system failure, Martin said, "We could be back online, with the proper personnel, probably within a week." Abshier made a point of saying that Texaco was finding Y2K problems in only 5 percent of its embedded systems - enough to take Y2K seriously but not so many as to cause panic. And they found no problems in life-critical systems, those related to safety, health, and the environment. Texaco got a relatively early start on its Y2K work; Abshier has a large budget (Texaco estimates it will cost about $75 million to fix its systems); and he said many times that the problems they're finding are "not showstoppers." And he still retains the plainspoken confidence from his days of writing code. "Engineers know all these systems are not going to fail. Engineers aren't stupid."

And yet, as the day wore on, I became aware of an edginess in Abshier. Maybe it was the uncertain atmosphere in the Stormac control room itself, dim and quiet like a radio station, with seven consoles showing data readings from the offshore control centers. Suspended above the consoles was a muted television permanently tuned to the Weather Channel. Despite the hot hazy sunshine outside, a tropical depression was developing in the Gulf, and Martin, who would have to supervise the platform personnel brought here in case of evacuations, kept sliding his eyes over to the TV. "We're waiting to see if it's named," he said, meaning they were waiting to see if the depression became a tropical storm.

"Interdependency" was the word Abshier kept saying. He had used the word several times during the test demonstration, but now, as we talked in the control room, he seemed to give in to its implications - the vast, interconnected economic machine that lay beyond his control. The nitrogen vendors he depends on. The thousands of other critical suppliers. The subsidiaries in South America and Indonesia. The big customers - airlines, other oil companies, utilities, outside pipeline operators, the automobile industry ("every car off the assembly line has oil in it") - what will happen if they succumb to Y2K and stop working, supplying, buying? Abshier's composure seemed to waver as he let himself consider all the possible points of failure. "I'm aware of the interdependencies," he said, "the cascading effect. One pipeline going down - what's the cascade effect?"

It was right there that I caught some of Abshier's edginess. Suddenly it was all too easy to envision Y2K problems propagating themselves, system to system, like some ice-nine of the computer age.

Even as Abshier was worrying over the outside pipeline operators, they were worrying over him. Two weeks before meeting Abshier, I'd met Steve Wilson, Abshier's counterpart at El Paso Energy, a pipeline transmission company for oil producers like Texaco. Wilson exactly mirrored Abshier. The same acknowledgment of real problems. The same engineering optimism about their own systems. Then, inevitably, the same fear of the outside world. "It's everyone else I worry about," he said.

Worry about everyone else: This was the core issue people involved in Y2K work raised over and over again. What was unnerving everyone was their sudden realization of critical dependency on other systems they knew so little about. Abshier, like everyone else, was trapped in a paradox. He complained about the faulty information he was getting from some of his vendors; meanwhile, he complained about the questionnaires vendors were sending to him. "You wouldn't believe the data they want from Texaco," he said. "I say, 'Are you kidding? You're a vendor!'"

One of the greatest interdependencies for Texaco, as for all of us, is electrical power. And there again came the paradox: Texaco relies on the utilities for power, Abshier explained, and the utilities rely on Texaco for fuels. "They're just as worried about us as we are of them."

While we stood in the darkened control room, Abshier contemplated the likelihood of losing electrical power. The sensible guy in him had a "high" confidence level in the utilities. He thought the probability of a failure that will take days to recover from was "pretty small." Then again ... He could imagine the way a catastrophic, cascading failure might happen: manufacturing plants shutting down unexpectedly, causing power fluctuations on the grid; electrical cogeneration facilities simultaneously going offline; all of this exacerbated by embedded-logic failures in the electrical transmission and distribution system.

What, finally, is Texaco going to do about electrical power? Abshier admitted he'll probably take Texaco's data center off the grid at the turn of the millennium - that he'll "warm up the generators and go to emergency backup several hours before midnight." He tried not to make too much of this decision. "Might as well do it just to be safe."

- Ellen Ullman

Copyright ) 1993-99 The Condi Nast Publications Inc. All rights reserved.

Copyright ) 1994-99 Wired Digital, Inc. All rights reserved.

-- Flint (flintc@mindspring.com), April 25, 1999

Answers

I'm having a hard time figuring out the date on this article. It seems to be archival--perhaps Feb 1999? Do you know in what issue it appeared?

:)

-- FM (vidprof@aol.com), April 25, 1999.

Flint

You like to walk the Y2Knifes edge eh? No one has the answer to the problem below. Tech or lay people can just look at this and shake those dice.

What, finally, is Texaco going to do about electrical power? Abshier admitted he'll probably take Texaco's data center off the grid at the turn of the millennium - that he'll "warm up the generators and go to emergency backup several hours before midnight." He tried not to make too much of this decision. "Might as well do it just to be safe."

>>>>>>>>>>>>>>>>>>>>>>>>

Technology Problems and Industrial Chemical Safety

http://www.csb.gov/y2k/y2k01.pdf

For some managers of facilities that draw high power loads prudent safety practice may determine that the plant be shut down during critical time periods and restarted at a later date. However, such decisions should not be made without communicating these planned actions with their utilities in order to prevent problems on the power grid. As a further complication, cumulatively, small power consumers can impact on power distribution through the nearly simultaneous shut down of many facilities without coordinating with their utility. Utilities can bring up or shutdown generators as demands vary, but they have trouble responding to unexpected changes in load or demand.

-- Brian (imager@ampsc.com), April 25, 1999.

FM: The above article appeared in the April 99 issue of Wired.... the one with the black cover. It is probably on line now since the May issue is already out.

-- jeanne (jeanne@hurry.now), April 25, 1999.

FM,

Here's the link to Wired's April 1999 issue:

http://www.wired.com/wired/archive/7.04/

-- Kevin (mixesmusic@worldnet.att.net), April 25, 1999.

The guy tells her he is fixing it, shows her how. Then tells her about the work on their SCADA system and so forth. Then tells her - of course we will have problems if our suppliers have problems, we do depend on them. The rest of the article is meaningless emotion directed at thin air, made up from that one bit of talk out of what sounds like hours worth of explanation.

And you guys wonder why companies don't have anything real to say about Y2K! Why bother? Doesn't matter what you say, or what you show - a determined D&G can make it sound like TEOTW is coming fast.

-- Paul Davis (davisp1953@yahoo.com), April 25, 1999.

"The first round of replacement chips the RTU vendor sent them didn't work; they had to wait for another. Then the Scada system needed upgrading. And that was just for this one device.

Right at that moment, the Scada system was polling hundreds of embedded-system devices, collecting and storing about 30,000 points of data."

Interdependency in action. What if they forget to update Scada for one of these 30,000 points, and that one is CRITICAL? What if they update Scada for one of these 30,000 points and forget to update the RTU, and that one is CRITICAL? This is ONE site. I rest my case. <:)=

-- Sysman (y2kboard@yahoo.com), April 26, 1999.