Bruce Beach, author of an excellent paper on the natural gas industry, sent the following message today. I'm sure he would want you to know. And I will look forward to your intelligent discussion.

Faith --------------------------------------------

Y2K People Finding People - http://www.webpal.org/list.htm

As you know, I have been working for months on getting an interview with a Petroleum Refinery Engineer. Happened Wednesday.

A TV Producer under contract with an ABC affiliate let me be the on camera INTERVIEWER at a large Oil Company Refinery Research Facility.

The Interview is On the Record, but I am not going to give you the company name, because some of things that I am going to tell you were off the record.

You can read my report and study at:

http://www.webpal.org/Gas.htm

It was an EXCEPTIONAL interview, and by the way, if you have not read the TEXACO interview on page 130 of the latest all black cover "Lights Out" issue of Wired Magazine - DO SO! That too is EXCEPTIONAL!

I asked the TV Producer why others are not doing this kind of reporting. He said he wouln't have understood the problem, if I hadn't explained it to him, and that he just happened to have exceptionally good contacts with the Refinery Research Center because he used to do their training and PR videos.

Still, there were back and forth lengthy, lengthy lawyer letters (at the end so lenghty my email wouldn't download them). They wanted the interview completely scripted out, with all the responses written out and reviewed ahead of time. But, in the end, the Research Manager fought through and we were able to do a VERY good on camera open interview within certain guidelines, and I was allowed to ask all the questions that I wanted to off camera. GREAT!

The manager said that he would not have been permitted to give the interview at all - six months ago. But, the government had passed the Good Samaritan Legislation and now they could talk without fear of being sued.

Still, you may never see the show. Why? Because of the networks. A producer produces, but a program manager schedules, and he may just decide that the subject is too complicated and that the public couldn't understand it or wouldn't be interested. OR above the program manager there is the station management, and they may have a policy of not treating Y2K seriously, because they consider it a "fringe" subject, or they don't consider it socially responsible to broadcast anything that might PANIC the public.

And even if it were to get broadcast locally, then there are the same hoops above that for the Network. Program managers, network management, network policy and so forth. The networks have done a LOUSY job of covering Y2K, but they do a LOUSY job of covering many things. We have MARVELOUS Freedom of the Press in this country, but just lousy press that only likes short sound bites or doing SENSATIONAL subjects or something that involves sex. THEN they can spend MILLIONS of dollars and send HUNDREDS of reporters for a single interview.

I get many calls a week from media wanting to interview me. But they all just want to do the "survival nut" stories. (Although they don't put it in those terms). When I tell them that there are REAL stories and that I will help them with them they say, "No, just point me towards the survival nuts". If I try to prevail, they say, "Well, I am just a reporter and I will have to check with my producer", and I never hear from them again.

The subject of EMBEDDED Processors is very complicated. The implications are VERY serious, but I think that I am fairly good at explaining complicated issues, and that most people can understand what I explain.

Still, the explanation is too long to put in an email, so I will put it on a webpage called:

http://www.webpal.org/Gas.htm

and you can read it if you wish, and tell others about it if you wish.

---------------------------- Well, onto a different subject. I am planning to schedule 3 days later this month to check out the Black Helicopter scene. There are tens of thousands of people in this country that think there are millions of foreign troops concealed in this country, and that they fly around in black helicopters or use vehicles with UN markings and that Y2K may tied into a plot to take over this country. I am NOT a believer. There are also the stories about the 15 million body bags (oops make that 20, no 25, - every batch of email brings a higher number) that have been ordered from Dow Chemical, and how the Sherrifs are now all turning their fence toppings inward. etc. etc. etc.

Now, the people who believe in this stuff don't like me checking it out. Really! You do really want the un-believer checking out your evidence. That way you will get a more objective report. So, if you have any sites in the Kansas, Oklahoma, Missouri areas, that you would like to have checked out, or any other verifiable evidence you would like to have shared, please let me know.

--------------------------------------- And finally, a suggestion from Jim Warner----->

I have what I consider a serious and legitimate question: If Y2K causes severe losses and perhaps even deaths, can the corporate and government lawyers who have recommended or forced the information blackout be sued for malpractice for causing this blackout, when useful information might have prevented or limited the losses or death? I think this is a very important idea that needs to be brought to public dialogue, and I would like to offer it to you if you wish to start this dialogue.

Jim Warner -------------------------------------

Go for it.

Peace and love, Bruce Beach survival@webpal.org

______________________________________________________________________

-- Faith Weaver (suzsolutions@yahoo.com), April 09, 1999

Answers

This Article is Great - Gas Company Interview
Thanks

-- WebRNot (webrnot@ncap13k.com), April 09, 1999.

We have a dedicated National Weather Service that works feverishly around the clock to forewarn us of events which could endanger us and our families. Preparing us for major events like tornadoes and hurricanes, and the inconvenient winter "storms"(3"-12").

It would be nice to think that our Federal Gov't would act with such dedication. Instead, they are more like the Mayor and Council Members in the movie Jaws. "If we tell them there is a shark in the water we might lose money or control".

Unfortunately, if events go terribly wrong in 2000 we won't even have the means by which to "hold their feet to the fire". They have developed an escape hatch called FEMA which, when empowered removes all possibility of retribution from their constituency.

It would be nice for all of us if we could act without the worry of consequences as they do. Note:Just a thought from a conspiracy Nut/hoarder.

-- WebRNot (webrnot@ncap13k.com), April 09, 1999.

Thanks Faith

It is for information like this that I read this forum. Recently there has been all too little of this type of posting, and I wish there was more of it.

I am not a techie and rely on others in this forum to explain some things to me that are complicated. I have read other essays on this problem but this seems much more real. One reason is the manner that the author explains his credentials and then explains how he came by his information.

I had not heard of the "laddering" before and that certainly shows that testing these "chips" is very ,very difficult. It would seem that many are unaware of the extent of the problem and not testing for the proper failures anyway. Ouch!

I have also read the article the author wrote on the gas pipleline problem. Both are well written and quite conservative. Am I correct in assuming that this information is accurate as far as others (techies) are concerned? If not, please explain what is incorrect in the essay.

Thanks.

-- Mike Lang (webflier@erols.com), April 09, 1999.

Faith,

Thank you for posting this!

- - - - - -

Mike,

I am a techie. I grew up in a city that had an Oil & Gas page in the daily newpaper and a world-class university school of petroleum engineering. I worked five years in the computing department of a major oil company's research center.

Bruce Beach's article seems quite genuine and well-informed. I see nothing in it that contradicts what I know from experience, and quite a bit that matches.

-- No Spam Please (No_Spam_Please@anon_ymous.com), April 09, 1999.

Mike,

Just a few comments:

Bruce uses the terms primary clock, secondary clock, and external clock. He clearly describes what he means by the term primary clock; it is simply an oscillator. His term secondary clock seems to refer to what most would call a real time clock (RTC). Unlike what he calls the primary clock, the RTC would keep track of dates, and could have Y2K problems. Whether or not the RTC might have Y2K problems, software that uses it could have Y2K problems if the software uses two digit years. Bruce also refers to an external clock, but is not clear what he means by that term.

In PC terms, when you set the time and/or date on your PC, the software sets the RTC, in Bruces terms, the secondary clock.

Regarding the discussion of ladders:

My impression of the use of the term ladder in the context of PLCs is that it derives from "ladder logic" diagrams used to describe networks of relays that were commonly used before digital computers got into the PLC business. What I have been told by process control engineers with whom I worked in a previous life :-) is that as digital computers replaced the networks of relays, much of the terminology was retained, and that the software was designed to blend as much as possible with the previous experience of the engineers who would be using it.

Jerry

-- Jerry B (skeptic76@erols.com), April 09, 1999.

Stuart Rodman, whom Art Bell interviewed on 1/12/99 about his analysis of the January NERC report, pointed out that one of NERC's biggest concerns (at least as of the beginning of the year) was the potential for "common mode failure" in the power grid. In other words, the potential that one overlooked or misunderstood device commonly used throughout the system, could create widespread havoc. One of NERC's contingencies (it's in their plan, but I don't remember the URL), is power rationing to minimize the likelihood of energy transfers between plans. Of course, that was before the more recent announcements that the experts don't think the embedded system is nearly as widespread as they had once thought! Looking forward to an analysis by forum experts on Beach's contribution.

-- Brooks (brooksbie@hotmail.com), April 09, 1999.

A nice article. Thanks, Faith, and also Bruce Beach.

-- dave (wootendave@hotmail.com), April 09, 1999.

My IT expertise has nothing to do with embedded systems. But I'll repeat what I've been saying all along (with many others, natch): the news is not all in about embedded systems. I had become more optimistic, but even so, have assumed that they could, all by themselves, tip us into TEOTWAWKI.

This post marks an end to my "optimism" unless someone can poke meaningful holes in this. I'm waiting eagerly ......

-- BigDog (BigDog@duffer.com), April 09, 1999.

Faith,

Thanks, for posting this, it was a great read. LOL Big Dog.

I hate being at the mercy of experts.

Now everyone will probably have an ugly argument over this info. & I will still not be sure who is correct.

Feel lucky?

-- Deborah (infowars@yahoo.com), April 09, 1999.

I'm no expert on refineries. I can easily understand how date bugs might creep into control software in many ways. I'd be surprised if there weren't problems, a few of them catastrophic but most of them fixable by various means (shutdown-and-restart, turning clocks back 28 years, etc.)

I *do* know about RTC chips. All are y2k compliant if properly 'windowed' by the software accessing them. The bad news is that replacing the RTC chips won't change a thing. The good news is, you don't need to change any chips - just the software that reads them. The bad news is that some of this software is in ROM, so you still need to change a chip somewhere. The good news is, very little offending software is in ROM, and even then, resetting the RTC date will fix the majority of those problems.

Once again, the fixes are pretty easy. Finding *what* to fix may not be.

-- Flint (flintc@mindspring.com), April 09, 1999.

Good evening Flint. I spent a couple of years working on a "black box" terminal controller. My job was writing (some) of the Z-80 assembly code that was "burnt" into the ROM. I gotta argue with you on this one. When you say that the good news is that you only have to change the ROM to update the software, you are making the assumption that the software is available. Just about every "embedded system" has it's own custom written program. Usually many revisions were made over the lifetime of the system. We usually had a new REV about every two months. The hardware for our "black box" was only updated two times over it's life, but we had at least 15 different versions of the software. Same box, totally different function. For example, one of the REVS was to add data compression.

I would venture to say that once a box is no longer being manufactured, it is next to impossible to find the original, correct version of the source code to update. I would also say that many of the older "blank" ROMS are no longer being manufactured, hence also next to impossible to find. Any comment? <:)=

PS to all, Why are none of our regular pollys responding to this post?

-- Sysman (y2kboard@yahoo.com), April 09, 1999.

Sysman - I think it exceeds their attention span! :-)

Arlin

-- Arlin H. Adams (ahadams@ix.netcom.com), April 09, 1999.

Sysman:

Quite so. I've written numerous times here about the 'replaceable unit'. The ROM itself needs to be replaced, but it isn't the replaceable unit as you point out. The entire black box is the replaceable unit at best (once it's been determined that no workaround short of complete replacement is feasible). Of course, sometimes there's no snap-in black box available (much less compliant), and whatever the old box snapped into needs to be replaced as well. This can get expensive.

I've wondered for a while if one of the key reasons why especially manufacturers haven't spent much of their y2k budget yet, is because they're either waiting for expensive replacement equipment to arrive, or postponing paying for it. I keep expecting to hear of more facilities being decomissioned because fixing them is more expensive than they can possibly repay over their remaining lifetimes.

-- Flint (flintc@mindspring.com), April 10, 1999.

Good point Flint. I wonder what the ratio is of boxes that are current, and can be updated just by a new ROM from the manufacturer, vs. a more complex solution like you mentioned. Maybe I'll put up a new thread if we don't attract a few answers here. See ya. <:)=

-- Sysman (y2kboard@yahoo.com), April 10, 1999.

Isn't this PNG's and Mr. Cooks' area of expertise?

A few points.

1. A lot of explosions at refineries this year - testing?

2. It is still the case that at least one country - Venezuela - has admitted that it will shut down refineries that it deems will not make it in time to avoid 1. above. the last count I heard was two refineries.

3. Thanks faith - this is what we need - information.

-- Andrei-hung-lo (2000EOD@prodigy.net), April 10, 1999.

To support Bruce and add another element that he did not touch on is the fact that the "microprocessors" will fail at differant times. Below is a quote from the IEE which is the benchmark of the industry. Here in black and white is the issue we are up against. Not weather chips will fail but when. I beleive this is an issue of long term risk rather than short term.
 

http://www.iee.org.uk/2000risk/guide/year2k12.htm

2.4 CATEGORIES OF EMBEDDED SYSTEM

2.4.1 Individual microprocessors

2.4.1.1 These may be found in small devices such as temperature sensors,
smoke and gas detectors, circuit breakers, etc. It is highly unlikely but
nevertheless possible that they will be affected. If they are (i) it will
not be evident until after the date and (ii) the only possible action is to
replace the whole device with one known to be compliant and otherwise< br> satisfactory. 

2.4.2 Small assemblies of microprocessors with no timing function

2.4.2.1 These may be found in flow controllers, signal amplifiers, position
sensors and valve actuators. It is unlikely that these will be affected.
However they may depend for their internal operation on a clock which might
be affected by the Year 2000 problem. 

2.4.3 Subassemblies with a timing function

2.4.3.1 Devices such as switchgear, controllers (e.g. for traffic),
telephone exchanges, lifts, data acquisition and monitoring systems,< br> diagnostic and real time control systems may fall into this category. These
systems may be local elements in a larger system to which they pass data
collected by their sensors. They may incorporate a PC [ The display screen,
keyboard and the box containing the other hardware may not be used.] , and
may involve some kind of database (e.g. of events). In these the Year 2000
problem may affect their systems or application software, the database, and
the networks and data transmission systems they use to communicate with the
larger system. The error may become apparent before the Year 2000 (because
the system may attempt to make a record of when next a particular action
should take place), on the date 01/01/2000 and for some time after that. 

2.4.4 Computer systems used in manufacturing or process control

2.4.4.1 This relates to cases where the computer is connected to plant or
machinery in order to control it. In such systems the computer is used for
overall control and monitoring, rather than for direct control of
individual devices within it, which almost certainly involves other kinds
of embedded systems. These systems are liable to be affected in exactly the
same way as commercial data processing systems, because of course the< br> hardware and the systems software are the same, and because the
applications software may have been developed along similar lines.& nbsp;

2.4.4.2 There is a developing trend to link process control with business
systems (for example to enable sales figures and stock levels to determine
automatically which quantities of which products should be produced). This
raises the possibility of knock-on effects from one to the other. There are
also off-the-shelf ("ready-made") hardware/software packages in this< br> category.

-- Brian (imager@ampsc.com), April 10, 1999.

Some GM Data

Year 2000

Year 2000 Test Procedures

           & nbsp;   Test Procedures

This version of the GM "Year 2000 Test Procedures" may be
revised in the future.

    1.This document is being used globally in GM as a
       reference for all Year 2000 Testing.
    2.This document is being shared with all of our suppliers
       and other manufactures to raise the level of awareness
       industry wide for year 2000 testing.
    3.As a condition of use, we request feedback on the
       document including:
           & nbsp;  Additional tests and / or examples of failures
           & nbsp;  Test plan examples for systems, clusters, and
           & nbsp;  enterprise testing.
           & nbsp;   Please e-mail any feedback to the authors. (see
           & nbsp;          & nbsp;        section 1.5)

Big Snip
 
 
 
 
 
 

1.9.1 PC Compatibles 

Tools have been choosen that General Motors will use to
test "PC Hardware" for Y2K compliance. Please refer to
the General Motors corporate Y2K Library for the latest
versions of these tools.

Note: When using test tools to perform the Year 2000
testing on personal computers (PCs) the tester must
understand the architecture of the PC. There are two
main clocks in the PC, the real-time clock (RTC) and a
software clock in the operating system. The recommended
PC test tool tests the operation of the RTC which is not
directly visible from the operating system user interface
and may report different results from manual testing. What
does the test tool test? Once this is understood, the tester
needs to assess the impact of a test 3.2.1 failure on the
application. Real-time applications which will be operating
through the Year 2000 transistion and that use the RTC
many be impacted by failures that desktop applications
which access the operating system clock would never
encounter. There are several BIOS implementations which
recognize this hardware limitation of the PC/AT
architecture and correct the time and date on reboot when
the year 2000 arrives. When a PC which fails test 3.2.1 of
the RTC will be powered off during the year 2000 date
transition and passes test 3.2.3, no remediation is required.

1.9.2 Industrial Controllers

At this time no test programs have been identified which
apply to industrial controls. Several may be in development
for scanning PLC programs for potential year 2000
problems. ( Please send information to the author about
this subject.)

2. Test Planning

This section of the document provides guidelines for
writing specific test plans. Example test plans for plants or
sites and individual systems are included in the appendix.
The resources required to conduct the tests are also
described. Testing shall be conducted on large systems
using the divide and conquer method. Unit testing
addresses the smallest division for which tests are
reasonable, the number of test cases is smaller and the
likelihood of finding all the errors is increased. Once a unit
has been tested, integration testing begins by assembling
larger systems from tested units. At this level the number
of test cases may increase and the likelihood of finding all
the Year 2000 errors decreases. Balance and judgment are
required of the test designer to select the system boundaries
and appropriate test cases. Systems which are unique may
share many common components and similar architectures.
Two systems using identical hardware in a similar
architecture may have different applications and use date
related functions in completely different ways requiring
separate testing.

Testing Levels:

    1.Component testing, testing a single component
       controller or application in isolation
    2.Combined Component testing, testing an
       assembly of standard components and custom
       application programs or configurations
    3.System testing, testing a collection of
       Combined Components and Components which
       are assembled into a complete system. An example
       is a welding cell with PLC's, Robots, HMI's,
       PC's, and weld controllers. Readiness Testing is
       System Testing in a Y2K compliant environment.
    4.Cluster (Integration) testing, testing a
       collection of Systems and Common Systems and /
       or Business Systems that constitute a complete
       process. An example is a test of the paint shop
       systems taken as a whole including the interfaces
       to any higher level support or scheduling systems.
    5.Cross Cluster (Enterprise) testing, the
       highest level of integration testing where the test
       cases are organized around specific functions, or
       across functional boundaries. An example is the
       entire process of Automobile Order through
       Automobile delivery.

2.1 What to Test

Prioritize test scheduling based on the inventory of controls
and computers to identify the most suspect components or
systems. Suspect components are those with unknown or
undocumented testing results. Components that are part of
a system that shares date information or that uses date
information for product marking in the manufacturing
process (encoded date information) should be considered
suspect. Unique systems which have been locally built and
programmed must be tested individually, test results for
commercial off the shelf hardware or software may be
shared industry wide.

2.1.1 Resource Availability

Evaluate the required resources prior to scheduling the
tests:

       The system is available, not in production
       The system cannot be tested in a lab environment
       The system is functional and can be dry cycled
       Ease of dry cycle operation may influence choice
       of cell where similar systems exist
       Appropriate personnel are available to oversee and
       conduct the testing
       External Systems interfaces can be disconnected,
       simulated, or modified for testing purposes
       Backup has been performed prior to testing
       Support personnel are available as required
       CPU time or storage capacity available for test
       execution and test data storage

End of snip---------------------------------------------------------

-- Brian (imager@ampsc.com), April 10, 1999.

Brian:

If I'm reading you right, you raise a good point. There is of course a very large number of different ways that devices can fail in theory. People have a tendency to assume that all possible mechanisms are more or less equally likely. In turn, this leads to the impression that if we can think of twice as many possible failure modes today as we thought up yesterday, things are twice as likely to fail as they were yesterday. This impression is hard to overcome.

This logical error seems natural to our brains, somehow. As a baseball fan, I get tired of hearing that there are 9 ways a runner might score from third that he can't score from second, and therefore the runner is 9 times as likely to score from third as second! This myth persists despite over 100 years of statistics indicating quite otherwise. This becomes important in baseball because these exhaustive statistics show conclusively that sacrifice bunts *always* reduce the probability of scoring! Yet we still see this 'strategy' executed all the time.

Once we start combining long lists of possible (but vanishingly unlikely) failure modes, with the implication (generally false) that any failure is a total catastrophic failure, we jump straight to the conclusion that catastrophic failures must necessarily be extremely common, and that the infrastructure must collapse. This is a big overreaction. We know that Windows 95 has many thousands of known bugs. On that basis alone, we'd conclude that it was worthless. Yet it is quite usable, if occasionally unreliable.

So please don't confuse the vast potential for failure, with the practical observation of infrequent and minor glitches. The dangers are real enough. Exaggerating the dangers in our minds doesn't increase them in reality.

-- Flint (flintc@mindspring.com), April 10, 1999.

Flint --- If you are an Atlanta Braves fan, this would, IMO, explain all.

-- BigDog (BigDog@duffer.com), April 10, 1999.

Flint

The point below is that know one knows but the possibility exists that there is a risk over time of failure and not just on the rollover. This one would assume is because when the systems were designed and when they were powered up is differant. As to to the end result of failure that is just going to be known at the time of failure. No one knows, pollys or doomers. But it is the risk that exists. I didn't give any impression on the results of failure just on the risk. I am NOT a tech person but consider the unknown factors have to be documented and I am a bit of an archivist and the IEE states this and they are the last word in my books. To me this indicates that NO ONE KNOWS what is in the critical devices that run our utilities. "Highly likely but nevertheless possible that they will be affected" Is still speculation. But the risk is there or they would not indicate it as such. And of course then the statement "If they are (i) it will not be evident until after the date" Indicates they can't test for the internal clock.

Seems like a gambling to me rather than a baseball game. Like looking at the great unknown.

It is highly unlikely but nevertheless possible that they will be affected. If they are (i) it will not be evident until after the date and (ii) the only possible action is to replace the whole device with one known to be compliant and otherwise satisfactory.

-- Brian (imager@ampsc.com), April 10, 1999.

Brian:

I think you are mostly (not entirely) wrong on both counts. Yes, there are devices out there that no one really knows about. These are rare. Most devices in refineries are well understood by those who designed and programmed them. There aren't many mysteries left.

And yes, one of the optimal solutions for those relatively few that fail after rollover is complete replacement. But this is only one of many solutions, not the only one. I expect it won't even be the most commonly required solution.

There is genuine reason to be concerned, because rollover hasn't come yet and remediation isn't finished yet. But don't exaggerate your concern by exaggerating our ignorance or limiting our options.

-- Flint (flintc@mindspring.com), April 10, 1999.

Thanks Faith and Bruce,

Snip

"I have what I consider a serious and legitimate question: If Y2K causes severe losses and perhaps even deaths, can the corporate and government lawyers who have recommended or forced the information blackout be sued for malpractice for causing this blackout, when useful information might have prevented or limited the losses or death? I think this is a very important idea that needs to be brought to public dialogue, and I would like to offer it to you if you wish to start this dialogue.

Jim Warner ------------------------------------- "

The same thought has crossed my mind MANY times.

Ray

-- Ray (ray@totacc.com), April 10, 1999.

Ray:

Now flip it around. IF y2k turns out to be a big nothing, just how grateful will you be to the government for telling us the truth and preventing panic?

Or will you say, well they *thought* they were lying so it doesn't count? Or will you just move on to the next big conspiracy?

-- Flint (flintc@mindspring.com), April 10, 1999.

Flint,

Your premise that the government is telling us the TRUTH has already been shown to be FALSE. The government HAS lied about y2k. I know that is difficult to believe!!

Ray

-- Ray (ray@totacc.com), April 10, 1999.

Flint,

"for telling us the truth and preventing panic?"

I wish we wouldn't dwell so much on this "panic" angle - when have you last seen a population "panic" over anything - especially in the USA.

The average American is so boozed up, doped up, stressed and just plain tired and/or all of the above to care much about anything other than day to day realities for THEMSELVES.

And this is how it is designed to be.

The gubbmint are not playing fair as usual. They ARE lying, brazenly, and will continue to do so.

-- Andy (2000EOD@prodigy.net), April 10, 1999.

Burp! (hic) wazzat, Andy? y'mean I'm followin some plan? Thash too compl.. uh, hard. To deal with. I got imPORtant stuff goin on, see? Too busy.

-- Flint (flintc@mindspring.com), April 10, 1999.

I promised a fellow who emailed me about this thread that I would post an analysis/answer to the Beach paper. Tomorrow, not tonight, it is midnight here and I am too sleepy. Did read it and realized that I had explained the whole thing before the first of the year. Guess everyone forgot it or can't relate it to BB's home grown terminology. Hate it when people do that - I may fuss with you but I damn well refuse to talk down to anyone unless they are being intentionally obtuse.

-- Paul Davis (davisp1953@yahoo.com), April 11, 1999.

Flint:

"Most devices in refineries are well understood by those who designed and programmed them. There aren't many mysteries left."

they are long gone .... we are on our own.

Have a sunny and pleasant day

pete

-- Peter Starr (startrak@northcoast.com), April 11, 1999.

Peter:

OK, I haven't seen any detailed refinery assessment results. If you have these data, can you tell us how many total devices were found, and of these, how many were 'orphans' no longer understood by anyone? I admit I was extrapolating from a large manufacturing plant, where very few such orphans were found among a great many total devices.

Paul:

Good luck. I found Beach's terminology very hard to relate to my knowledge and experience. He might as well have been talking about papa bear, mama bear, and baby bear clocks. The average board has several different oscillators, and each of these might be subjected to multiple dividers depending on purpose. Is he talking about synthesized clocks? Sampled clocks? Software clocks? It was weird.

-- Flint (flintc@mindspring.com), April 11, 1999.

Flint - at first I thought he might be talking about software clocks of some sort. But close reading of the material on the mystery oil company convinced me he was talking about hardware clocks - since the clock had to be interfaced to the hardware to make it work. Sounds like a RTC to me. I admit he is vague as all get out but that is the best I can do. Frankly, this was written to scare people, not for techs. So I address RTC's in particular here - though it would really apply to almost any sort of hardware clock function.

My reply to Bruce Beach and his essay on The Embedded Processor SECONDARY Clock Problem. First and foremost, he does not use any semblance of standard terminology. This makes the whole thing very hard to read. I am going to talk in the real terms, if this makes anyone drop out, too bad.

His first few paragraphs tell you about him, though if this is the Bruce Beach I know about, his degrees are in economics, not Computer Science. That hardly matters, and if this is another Beach, I apologize. So, let us have a look at what the meat of this actually says.

After we skip over the usual "I know about this and I don't think anyone else does" stuff (WHERE do these people come up with this much EGO?) he starts describing the insides of a microprocessor. OK - rather over simplified but not too bad. In addition, he is quite correct that CPU's do not keep time. Support chips are necessary. To cut to the chase, he is talking about RTC (Real Time Clock) chips. An RTC chip may or may not be used as a dating device; some are just used as interval timers. Depends on the application. Moreover, almost none of them use four digit years. All an RTC does is keep time and date - and it puts this out on the bus that connects it to the CPU. If you want a world of tech info on RTC chips just look on the web - here is Motorola's chip site

http://mot-sps.com/sps/General/chips-nav.html

for an even bigger thrill, do a site search for Real Time Clock.

Therefore, the RTC puts out date and time. That is its sole function. So he asks if the PROGRAMS running on a machine, whether or not they directly access the RTC (doesn't matter if it is a ladder program or not, whether or not it is burnt into a PROM or EPROM or EEPROM, a program is a program), whether or not the RTC even can be set by a program or hardware function, he asks if these programs can fail to act as expected when the clock rolls over.

My answer is - well of course, they could. AND it doesn't matter in the REAL world whether they could or not - and we will get to that in a moment. You see, the REAL WORLD is my concern. I don't live in a world where I obsess about possible ways things can fall apart.

Now this RTC matter has come up before in different clothing. However, the chips are still the same, and the answer is still the same - doesn't work that way. AND, BTW, the way RTC chips work and hook into other devices IS well known among computer scientists, technicians and engineers - despite Beach's claims to the contrary.

On to why it doesn't matter if the program would fail at a rollover if you can't set the clock. WHERE will the clock start? IF IT HAS NO INPUTS ON STARTUP, IT STARTS AT ZERO - THE SAME PLACE ANY HARDWARE COUNTER STARTS WITHOUT INPUT. The BIOS on a PC sets the date to PC zero date - 1/1/80 - when it starts up without any date on the RTC. That is why you always get the 1/1/1980 thing (depending on the exact BIOS) if your PC battery fails. So if you start a device containing an RTC and provide the RTC with no input, it will start at day one, month one, year one. That gives you over NINETY NINE YEARS of continuous operation without power failures or batteries running down before the rollover event will actually occur. Now if you are not a total newbie to computers, you probably have had a battery run down. So it had to be replaced. And your time had to be reset. Happens to PLC's and such too. AND CAN YOU, EVEN IN YOUR WILDEST DREAMS, IMAGINE KEEPING ANY PIECE OF ELECTRONICS GOING FOR NINETY NINE YEARS? OR EVEN TWENTY YEARS IN THE CASE OF A PC IN WHICH YOU COULD NOT INPUT A DATE? AND industry is not a friendly place for electronics. Power surges are common - just start all the pumps at once at Camp#9 mines and you caused a power surge that could drop the voltage in my lab to nearly 90 volts. Dust, water, heat, humidity, insects, metal grindings, heavy duty accidents involving forklifts and such - the life span of a given piece of electronics in heavy industry is pretty short - ten years would be a long survival time for anything on the line. Moreover, equipment is constantly being updated and replaced as new stuff is brought out. THEREFORE, IN THE REAL WORLD, IT DOES NOT MATTER AT ALL. THE ROLLOVER WILL NOT OCCUR.

Having gone that far, why did the mystery engineers at the mystery oil company find so much trouble? Frankly I have doubts that they did - and you may reach whatever conclusion you wish from that. BUT - if such a place really exists - they introduced a number of really bad failure modes into their tests that would not be present in real life.

FIRST - they replaced a component with another component WHILE THE COMPONENT WAS UNDER POWER. THAT IS A NO NO. First day of the first class in any electronics based technical field you are told - DON'T EVER DO THAT. I have to say they were blowing stuff out by bad testing methods. Did not catch that in Beechs paper? Here is the snip - says exactly what I said it did.

!QUOTE> But, there is one that I can remember. It is where they could they would interface the system with an advanced clock. They would watch the system go through the clock change from 99 to 00. Even if the system successfully performed in that function they would then turn the system off and see if they could restart it. Much to their initial surprise, (and I had previously heard this from other engineers) some of the systems would not then restart. !ENDQUOTE>

I almost fell out of my chair when I read that! And they are in there soldering away, causing all kinds of thermal spikes and Lord knows what other problems. If it ain't broke, don't fix it.

You cannot interface the system with an advanced clock properly, using the SAME TYPE OF CLOCK AND ASSOCIATED CIRCUITS, without the clock running while you do this, since he specifies a clock that can not be set from the existing circuitry. If you connect it under power you invalidate the test because YOU NEVER, EVER CONNECT CIRCUITS WITH POWER ON. (printer cables and such are different - the input circuits are designed to allow you to plug in a cable or such - he is talking INTERNAL circuits on a l IF YOU USE ANY OTHER TYPE OF CLOCK OR CONTROL CIRCUITS TO ALLOW YOU TO SET THE CLOCK ON STARTING THE MACHINE BACK UP, YOU HAVE INVALIDATED THE TEST FROM THE GET GO. Therefore, you really cannot perform a test on a RTC that has been hooked up in such a blind way. There is no valid way to test the program - any way you can test it will introduce new modes of failure and invalidate the test! AND WHY WOULD YOU WANT TO TEST IT? That one really blows my mind. The machine has been running since power up X many years ago. You KNOW the date in the RTC does not match the real date in any way. You worry about failures if the RTC should somehow rollover. SO FOR HEAVENS SAKE, DO THE LOGICAL THING. PULL THE CLOCK BATTERY AND POWER THE DEVICE DOWN FOR A FEW MINUTES FOR GOODNESS SAKE. THEN IT HAS TO START UP AT THE SAME DATE/TIME IT STARTED AT X MANY YEARS AGO. THEN YOU KNOW BEYOND A SHADOW OF A DOUBT THAT YOU HAVE ANOTHER X MANY YEARS OF RUN TIME.

I just have a hard time believing any group of professionals would behave the way Beech describes - unless they are being directed to by Dilbert's pointy haired boss.

Anyway, don't get your shorts in a bunch over this particular theoretical mode for a microprocessor/PLC/PLA hooked to a 'blind' RTC to fail. In the REAL WORLD, it is not a worry.

Paul Davis

-- Paul Davis (davisp1953@yahoo.com), April 11, 1999.

Paul, Paul, I feel your frustration...I was ready to post but bothered to read through the ignorance to the end and found what you posted. That person, Bruce Beach, has written something totally silly. Secondary clock my big assets! He writes like he is a debunker...he is off now to prove the wierd ideas like the ufo's and black copters wrong...which he thinks will validate this pile of dog doo doo. Guess he just decided he would jump in and act like he is going to inform people from his limited 3 months of "research". Man, is he dumb. He will not last long *grin*. But I do understand how you feel about seeing this total caca and people feeding off of it.

-- Cherri (sams@brigadoon.com), April 12, 1999.

Point and counterpoint.

More infornation from Webpal on embedded chips

Bruce replies to some queries.

Will we ever know the truth?

-- Nick Laird (sharefin@cairns.net.au), April 13, 1999.

Well, one truth is self-evident, Paul and Cherri's baseless and condescending arrogance as compared to Bruce's gracious effort to analyze the problem, including all critiques of his position.

-- BigDog (BigDog@duffer.com), April 13, 1999.

Interesting analysis Paul - thanks for taking the time "to do right".

Makes you wonder - if the "test" was correctly reported and improperly done, or improperly done and improperly reported, or properly done and improperly reported. Doesn't appear the fourth option (properly done and properly reported) is viable.

Also - if improper" tests are being done - what could the secondary impact be of electronic damage to the processors and controllers. Flint (elsewhere) pointed out the potential problems of "excessive" testing - testing that introduced more problems than were found by the test. Certainly, excessive failure rates will be more likely if tests are improperly performed.

-- Robert A Cook, PE (Kennesaw, GA) (Cook.R@csaatl.com), April 13, 1999.

Big Dog:

You really crack me up sometimes. Here we have Beach, an economist, potificating about embedded systems. And we have Cherri, Paul Davis, Robert Cook, myself, and *all* the embedded people in csy2k unanimously saying that much of what Beach wrote is incomprehensible garble, and the remainder varies from vanishingly unlikely to outright ludicrous.

And you describe Beach as "gracious" while those who actually work in this area are spouting "baseless and condescending arrogance." And you consider your description to be self-evident truth! What a hoot!

Rarely does one see such a blind bias exhibited quite so nakedly.

Maybe I should read one COBOL textbook. Then I'll "graciously" expound on big IT systems (using my own incomprehensible terminology), and you can "baselessly and arrogantly" point out that I'm being an idiot. Hehehe.

-- Flint (flintc@mindspring.com), April 13, 1999.

Flint -- Glad to be providing such delightful entertainment for you on a beautiful spring day.

-- BigDog (BigDog@duffer.com), April 13, 1999.

Flint, IBM pub numbers:

SC28-6478-5 Cobol Programmer's guide

GC26-3998-0 Cobol Language reference

<:)=

-- Sysman (y2kboard@yahoo.com), April 13, 1999.

Flint, on a serious note, I never did get your opinion of the third "external clock" that Beach discusses. Please forgive my small waste of bandwidth here, and allow me to repost this:

---------------------------------------------------------------------------

Jon:

You're right. Microcontrollers contain ROM, RAM, interrupt mechanisms, etc. It is possible for the ROM programmed into a microcontroller to access a real time clock (RTC), and to be the only interface to that clock. It is essentially impossible to roll such clocks forward, since the only interface is to read it, and there is no code programmed into the microcontroller to set it. So far, so good.

Now, in this environment 'real time' is necessarily not synchronized with real time as we know it. These clocks aren't very accurate, typically losing or gaining up to 2 minutes a day. Furthermore, the initial time in the RTC is usually undefined at power-up. In other words, we're not looking at actual dates here, we're using the RTC as an interval timer over long intervals.

If there were some necessity for keeping the RTC in such devices synchronized with the real world, then there *must* be a mechanism for resetting the RTC to prevent drift and set initial time. So as a rule, if the time can't be set, the date isn't relevant.

This doesn't mean that a rollover error might not happen some day. I'm guilty of this myself, having written code that will have a 1- time error 25 years after initial power-up, assuming a perfectly accurate clock. But these errors even when they lurk inside the microcontroller ROM, won't all happen anywhere near the same time. But they will probably happen, here and there, mostly between 2010 and 2020. Some may have already happened.

Hope this helps, and isn't too technical.

-- Flint (flintc@mindspring.com), April 10, 1999.

------------------------------------------------------------------------

Thanks Flint, that helps. I still have a question about the role of the "external clocks" he mentioned. Does this keep everything in sync, and if so, doesn't this mean that any failures will also happen in sync? This isn't my area, just trying to figure it out! <:)=

-- Sysman (y2kboard@yahoo.com), April 11, 1999.

-- Sysman (y2kboard@yahoo.com), April 13, 1999.