A friend related to me an incident this past January.

At 1:30 AM, he got a call from his supervisor. He was to come down to the Chicago loop immediately. The "loop" is the heart of the downtown area. It contains a huge number of the commercial and financial firms in this city, including the stock exchange, options board and several international financial service centers.

My friend's company provides monitoring and protective services (security, HVAC, power, flooding) for buildings around the country. The regional phone company is a customer of theirs. Probably their largest customer. They manage several hundred telecommunications facilities for this utility in a five state area. It turns out that the power had failed in a small section of the city servicing one of their largest switching centers. In fact this center handles nearly all the wire-based phone / data traffic for the entire "loop".

Normally this would not be a problem. Back-up generators are installed in these centers for just such an occasion. However, this happened in the middle of a Chicago winter. The cold temps caused a water pipe to break in the generator room, flooding the room, contaminating the fuel tank and disabling the generator.

As a result, the entire facility was running on a back-up battery. The problem was that they were unable to get the generator running again. The battery would only last a short time. One of the largest business districts in the U.S. was about to lose their entire communications system. No phones. No faxes. No e-mail. No electronic funds transfer. No internet commerce. Silence.

Frantic calls were made. ComEd was working heroically at their end to get power back up, but couldn't say how long it would take. Through the night the battery continued to run down. A mobile generator, for whatever reason, was not available. TV crews got wind of the situation and were setting up their cameras on site to film the expected disaster as it happened.

With only 20 minutes of battery power to spare, electrical service was finally re-established.

SO WHAT ON EARTH DOES THIS HAVE TO DO WITH Y2K???

First, it is completely irrelevant whether the phones did or did not go dead. The issue here is that a single broken water line nearly shut down a significant chunk of our local economy, if only for a short while. A single water pipe!!!

The point I am trying to make is that our infrastructure is growing brittle. Small malfunctions are having large ramifications. A lightning strike in Florida shuts down rail traffic up and down the East coast. A satellite malfunction disables pagers and credit card scanners over a large part of the US. A fallen branch in Idaho causes wide spread electrical blackouts over a multi-state area in the west.

In the world of software, programmers describe their applications as being "robust" if they are fault tolerant. In other words, they don't come to a flaming crash every time you hit the wrong key. These programs are designed to anticipate problems and have extensive error processing algorithms imbedded within their logic. Of course, this robustness takes time and talent. It costs money.

That is why so many systems are brittle. They are written on a shoestring IT budget. The end user pressures the programmer to produce the most code for the least cost. That's the nature of a market driven economy.

Our economy is also a system. It includes every industry we have: manufacturing, financial, telecommunications, health care, transportation, etc. It is subject to intense market pressures. Competition drives out the wasteful. Every corner that can be cut is cut. Every resource is optimized to the greatest degree possible. If redundancy and fault tolerance cost too much, it is slashed with a vengeance.

The economies of scale demand centralization and consolidation. This principle applies equally to both corporate mergers as well as telephone switching centers. In the name of efficiency, we are putting all our eggs in one basket. Closing down redundant hospitals. Laying off redundant personnel. Consolidating the vendor base. Selling off excess production capacity. Making the corporation dependent on a single software system.

General Motors is brittle. They may not know it, but their fortunes may well depend on a single bolt. A key bolt that must stay in place on a key die (one of a kind) that is used to stamp a critical part by a sole-source vendor. Should that bolt fail and ruin the die, it will take days to replace. Production lines would be shut down in a number of plants for some period of time until the die is replaced. Sound far fetched? Remember how a small union strike at a sole-source supplier making a critical component shut down several GM plants a few months ago?

This phenomenon is universal. Businesses around the world are stretching internal and vendor resources to the limit. There is little excess capacity! There is little redundancy! There is damn little room for error! The tach is red-lined.

As a result, we have lost "robustness" in our economy. That is why seemingly small problems can have such a terrible and widespread effect.

And that is how a small and insignificant software problem, such as Y2K, can be so cataclysmic.

-- Hawthorne (99@00.com), June 12, 1999

Answers

Hawthorne, high praise: you have a well-developed sense of consequence.

-- Old Git (anon@spamproblems.com), June 12, 1999.

Hawthorne wrote: "SO WHAT ON EARTH DOES THIS HAVE TO DO WITH Y2K???

First, it is completely irrelevant whether the phones did or did not go dead. The issue here is that a single broken water line nearly shut down a significant chunk of our local economy, if only for a short while. A single water pipe!!!

The point I am trying to make is that our infrastructure is growing brittle. Small malfunctions are having large ramifications. A lightning strike in Florida shuts down rail traffic up and down the East coast. A satellite malfunction disables pagers and credit card scanners over a large part of the US. A fallen branch in Idaho causes wide spread electrical blackouts over a multi-state area in the west. " END OF QUOTE

Yes, but all of these small malfunctions were totally unexpected. Y2K is a bird of an entirely different feather.

-- Joe Six-Pack (Average@Joe.Blow), June 12, 1999.

Six pack,

B.S. Your implying that all impending failures are known. Again B.S. This guys describing 1, got it, 1 failure, and yes the fact that it was unexspected DID increase damage. What do you thinks gonna happen when a 100 thousand "unexspected" failure occure all over the world in the same 24 hr period?

Think man Buy some tuna...

-- MidwestMike_ (midwestmike_@hotmail.com), June 12, 1999.

It is just fascinating -- pollyannas CANNOT comprehend an argument based on generalizing from a single event (usually not Y2K related, since after all year 2000 isn't here yet) to many such events all happening in a very short span of time. This is why I find Infomagic's writings so on target -- our civilization can be thought of as a spider's web, with interconnecting strands. When one strand breaks, such as the Chicago incident described above, the other strands are strong to hold while enough resources are mustered to repair it. But if too many strands break, the web collapses. Isn't that obvious?

Makes sense to me. But can pollys comprehend this? Nahhhh.

-- King of Spain (madrid@aol.com), June 12, 1999.

King of Spain and Midwest Mike, obviously there will be "unexpected" failures. The difference is everyone is expecting "unexpecting" failures. I suppose I have to explin the difference to you, but I have to run right now, so I'll expain it later.

-- Joe Six-Pack (Average@Joe.Blow), June 12, 1999.

As a follow up to the first post-telco switching centers all have strings of deep cycle batteries which need to be maintained. Rectifiers can go bad, fuel can go bad in the diesel tanks if not used for long periods of time or protected by fuel additives, fuel deliveries can be interrupted by storms or breaks in the supply chain, generators can break down or deteriorate from long periods of idleness. As a part of Y2K readiness, has every telecommunications provider (I guess we could ask this question of any of our vital service providers) covered this aspect of their service provisioning? If I recall correctly, something related to a generator was one of the factors that caused the shutdown at the Peachbottom facility test. Even if the telecommunication generators are all ship-shape and able to maintain the load, how long will they be able to keep them up if the power grid goes down and the fuel supply is interrupted-two, three weeks???????

While you are out there buying that tuna, you'd better find yourself some wheat, a grinder, and get that pot of sourdough starter going.

-- Sharon L (sharonl@volcano.net), June 12, 1999.

So many polly's, so little time...

Six Pack, what kind of logic is that? Are you trying to say that "exspecting" an UNKNOWN failure amounts to the ability to deal with it? Here we go again...NON-SENSE. I'd guess that one third of the potential failures world wide have not even been identified let alone rectified. Please respond to this position with logic my freind.

Tuna, tuna, tuna. Got mayonese?

Mike

-- MidwestMike_ (midwestmike_@hotmail.com), June 12, 1999.

I think what Joe is trying to say is that come rollover, the emergency teams won't be home in bed or celebrating, they'll be on the spot waiting for anything. The mobile generators won't be unavailable for some reason, they'll be there and ready to use. Every key infrastructure element (power, water, phones, police, you name it) will be overstaffed, and armed with circuit diagrams, maps, tools, parts, whatever they need. This event definitely won't be treated as Yet Another Sleepy Friday Night.

Now, whether this maxed-out state of readiness will be able to make any noticeable dent in the universal catastrophic breakdowns you take for granted, who can say? But if the whole thing turns out to be a false alarm, *that* will be the only real surprise.

-- Flint (flintc@mindspring.com), June 12, 1999.

Flint, I can't believe this "maxed-out state of emergency readiness" you are describing, not even in America, let alone in Russia, Nigeria, Saudi Arabia, Venezuela, etc. It's not just willing people that you would need for such "Fix On Failure" plan. All sorts of resources would be needed, spare parts, obsolete chips that don't exist any more, bankrupt OEM manufacturers that have gone out of business.

The US imports more than 50% of its crude oil Flint from y2k non- compliant countries that are begging for funds and programmers right now in front of our very eyes.

Besides, if plans for that "maxed-out state of emeregency readiness" ever take place Flint, bank runs should to be expected everywhere. In troubled economies for sure. People are not stupid. It's just that sometimes they take longer than experts "in the know".

Flint, if your Y2K strategy is based on what you've described above, your probabilities of being dead wrong about Y2K are very high.

Your Pal George

PS: I admire your spotless English. It reminds me of Allan Greenspan's style (No sarcasm intended). Take care

-- George (jvilches@sminter.com.ar), June 12, 1999.

Flint,

Hi buddy, your estimation of the personell on hand rings true in my ears. That being said, the pending y2k failures will overwhelm the men and woman at their "posts". How can it be otherwise? The entire world is "connected". The company I am currently working for (y2k remediation project) will, in my estimation be ready, their vendors on the other hand may (read will) not be. This is a global investmant company. Any breakdown will snowball into very damageing consequences (sp?) for them. I imagine you've seen some of my postings and know I'm not exspecting a 10, but the possability of a 2 should be enough to motivate anyone to start SOME preparation. I know you are not a pure bred polly, help us out by sharing your preparations with the rest of us.

From the "show me" state.

Mike

-- MidwestMike_ (midwestmike_@hotmail.com), June 12, 1999.

George and Mike:

I tried to be careful to say that the *effectiveness* of a maximum state of readiness is debatable. Depending on what goes wrong, of course there may very well be nothing anyone can do about it, and as I said, their efforts won't make a dent in the problems. But these organizations *will* be overstaffed, as prepared as possible and ready for anything. Certainly there may be problems these SWAT teams can't address effectively, or perhaps at all.

IF this condition had prevailed at the time of the incident that got this thread started, it seems very likely that it would have been caught quickly, and consequently not been nearly as dangerous.

By the way, I most definitely did NOT say that I thought this would be the sole means of addressing y2k problems on the part of ConEd, or the phone company, or anyone else. From what we can tell, all of these infrastructure elements have been remediating and testing, and certainly should continue this process right up to rollover. But when the time comes, everyone knows that the detailed ramifications of y2k are unknown. So no matter how thoroughly you *think* you've addressed all possible problems, you're still aware that we're heading into the unknown. So you're as prepared as possible to react quickly and to handle anything you *can* handle.

Kind of like knowing that IF an armed robber should ever break into your house, it *will* happen tonight. You're quite likely to spend the night differently from usual, and this may make all the difference. It may not, but forewarned is forearmed.

-- Flint (flintc@mindspring.com), June 12, 1999.

Flint,

<>

The point I tried to make in the original post is that companies are already stretched to the max. They keep just enough emergency resources on hand to cope with ~80% of "normal" emergencies. If this 1/1/2000 is a "normal" emergency, fine. But if it exceeds that, all bets are off.

Note: even in normal emergencies, utilities borrow crews from other unaffected areas. If predictions are right, there will be no unaffected areas.

-- Hawthorne (99@00.com), June 12, 1999.

off?

-- __ (off@please.com), June 12, 1999.

Flint, Flint

I don't feel your reply is anywhere close to your usual standards. Could you please try again? Please re-read my logistics objection and international lack of preparedness. Y2K will be a global, instantaneous test of the world's entire IT. Y2K's attack will be systemic, with concurrent, generalized and parallel flaws which will strike anywhere and everywhere. Please address that vis-a-vis your maxed-out readiness philosophy. Thank you

I repeat, I envy your English

-- George (jvilches@sminter.com.ar), June 12, 1999.

Hawthorne and Mike:

It sounds like we're violently agreeing here. I freely admit that IF there widespread and/or serious failures, they'll be pretty helpless. Can either of you understand that if you have a lot of people standing ready, there is often *something* they can do, if not to solve the problems than at least to prevent them from getting worse (water leak -- cut off water fast. Power out -- people stationed everywhere can locate it quickly. On and on).

In the case of failures of key embedded systems (what we're really talking about with rollover failures), there's often collateral damage. Being on the spot can reduce this damage, so that later repair/rebuilding can go faster. The primary purpose of this overstaffing is to prevent things from getting out of hand as much as possible, and prevent 2-day blackouts from becoming 2-month blackouts.

-- Flint (flintc@mindspring.com), June 12, 1999.

As it was said of yore--

"For want of a nail, the shoe was lost;

For want of a shoe, the horse was lost;

For want of a horse, the rider was lost;

For want of a rider, the battle was lost;

For want of a battle, the kingdom was lost!"

-- Tom Carey (tomcarey@mindspring.com), June 12, 1999.

All too true Flint. Maybe the damage will be lowered enough to change the outcome by 1/32 on a 1-10 scale. Neither of us knows. It might be a night of catch the falling dominoes.

It has taken me some time and a considerable amount of reading to grasp the global consequence of the issues. It seems to me that they should be given more weight and tactics such as over-staffing need to be looked at in a global context.

The rest of the world may not have access to the resources to respond as quickly. Of course, neither may we. Oh, and two days versus two months is dealing in extremes and that is unlike you.

-- Mike Lang (webflier@erols.com), June 12, 1999.

Mike:

Not entirely unrealistic, according to those over-referenced smelting plants. Finding that bug took hours, rebuilding the plants from the emergency shutdown took six months. If everone had been ready for anything, that bogus shutdown could have been overridden on the spot, and the downtime would have been zero rather than six months.

Tom Carey:

I don't know what case you're making. If you're implying (and you really seem to be) that *every* nail will cost *every* kingdom, you need a sanity check. By that reasoning, computerization has been a total failure and isn't used anywhere, because nails are lost constantly. Yet we see that this isn't so.

George:

I can't understand why you continue to miss this simple point. There are emergency stopgap measures that can be used. No, of course they are no substitute for proper remediation and testing. I'm not saying they are. No, these measures won't be much help against all the global systemic failures predicted by the extremists. But if you feel that appropriate emergency responses where necessary are a total waste of time, then I can't agree with you. If we don't know what to expect, but we *do* know that we can do some good in some cases, I believe it's worth the effort to do so. Having somebody in the right place doing the right thing at the right time has a very happy history.

-- Flint (flintc@mindspring.com), June 12, 1999.

Flint,

Hello good buddy, I'm very happy to see we can have an debate without personal attacks (even though I still can't spell). My compadre Will continue is notably absent. Well I'm sure you'll live, eh?. I do enjoy your banter (it's a given I'm more in agreement with Will isn't it?).

Your freind.

Mike

-- MidwestMike_ (midwestmike_@hotmail.com), June 12, 1999.

Flint,

Thanks for your comments,

To repeat Mike's question: Would you care to share with us your level of preps?

-- Hawthorne (99@00.com), June 12, 1999.

Mike:

So long as you support your position thoughtfully, you have a lot more in common with me, even though we disagree on the future.

-- Flint (flintc@mindspring.com), June 12, 1999.

Flint is more ready (for potential troubles) than most who post here. Exact details I'll let him describe, if he wishes.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), June 12, 1999.

Robert:

Anyone who cares can look it up in must be a dozen places here by now. And I'm getting tired of the illogic that says everyone who thinks y2k will be a disaster is preparing, therefore anyone who doesn't think so isn't preparing.

This is the same argument as saying "all dogs have four legs, therefore everything with four legs is a dog."

-- Flint (flintc@mindspring.com), June 12, 1999.

Flint,

I was unaware you are frequently asked this question. But I do think it is an acid test. BTW the logic as you stated it is faulty. It should be:

If you don't prepare for Y2K, it may be for several reasons. You may either be lazy, poor, too busy or believe nothing will happen.

However, if you do prepare for Y2K, you MUST think something is going to happen.

If you have prepared to any extent, as Robert has said, good for you. But it would seem hypocritical for Pollys to criticize Doomers while stacking cases of spam in their basement.

If that is truly the case, what is your overall point? That it will be a 3 instead of a 7? That we should only prepare for two weeks, not two months? I think the loyal opposition on this forum would have more credibility if they disclosed their own level of preps.

I am prepared for four months.

I refer you to Tala's comments to me in a prior thread under utilities : "Preparing for a 9: Water". (Sorry I am URL-challenged!)

-- Hawthorne (99@00.com), June 12, 1999.

Flint......are you suggesting that individuals standing around, picking their noses and looking at their watches, just *prior* to overwhelmingly destructive failures due to a lack of complete remediation (or perhaps the lack of *end to end testing* alone) is going to make all the difference? It almost appears to me that this is a condensed version of your 'debate". I'm sure you'll deny this, I hope.

-- Will continue (farming@home.com), June 12, 1999.

You can run Flint, but you can't hide. This forum just isn't big enough for the two of us.

-- Will continue (farming@home.com), June 12, 1999.

Will Continue:

I'm hoping that if organizations are psyched up, staffed and ready, they can make *some* difference. I know that it all depends on the scale of what they face. If the scale is too vast, this readiness will be a spit in the bucket. So it might be a waste of effort, and it might come in real handy. Since nobody has anything to lose, and may well have something to gain this way, it's worth a try. If a single life can be saved because ALL the police and firemen are on duty, then it was worth it.

Hawthorne:

I'm prepared for at least a year in most respects. But I don't regard this as a guarantee that my preparations will be needed. I'm covered if my house burns to the ground with all contents, but that doesn't mean I expect a fire either. What I expect and what I'm prepared to handle are entirely different things. I think a 2-4 level event looks most likely, but a 7-9 cannot be ruled out by any means. So I've tried to prepare for the worst scenario I can, without taking steps I'll regret if not much happens.

-- Flint (flintc@mindspring.com), June 13, 1999.

Say what you will about Flint's wishy-washy beliefs about the impact of Y2K, his preparation strategy is probably better than 70% of the doomers that post here. Time is getting real short. Prepare.

-- King of Spain (madrid@aol.com), June 13, 1999.

You've prepared for a year! And your expecting a 2-4!!! That does it. I'm movin' to Alaska!

-- Hawthorne (99@00.com), June 13, 1999.

Hawthorne....your attempting to see the logic. Forget it, just gooo with it, just "be the flint, nanananananna, just be the flint nananananan"

-- Will continue (farming@home.com), June 13, 1999.

Just to put my two bits in.

It amazes me actually that society has little idea of the risks of cold weather. While I might not be tech oriented I know the cold. And there is very little information on contingency plans dealing with cold. This is a VERY weak link in the Y2K chain. I know because looking for information on this subject is a passion of mine. So if anyone would like to prove me wrong please do. The Eastern Storm has not really gotten folks focused on the problem. There has been lip service, but in my mind it is a risk like fires. Your apartment block may never get a fire but sprinklers are manditory at great expence. For less expence building could be fitted with emergency propane tanks and heaters. Right now cities are at the mercy of the power grids.

In the Eastern Storm the weather wasn't "that" cold. Flint I think you know that I am on record with an explination of my observations that we may have a very cold winter. It would be interesting to see what the farmers almanac says. Its a pretty good indicator. There is just no way society is set to handle the cold for extended periods of time. This is of course because of the standards set by the power industry and they are very consistant. But compliacancy is not an option in my mind.

Its funny that this happened in Chicago, for some reason I worry about that place.

-- Brian (imager@home.com), June 13, 1999.

"What I need is a list of specific unknown problems we will encounter." (Lykes Lines Shipping)

Another "recent" problem in Chicago.. totally unrelated to Y2K of course, was the completely unexpected complete shutdown of the decades long area sewer project called "DEEP TUNNEL"

These engineers planned for everything....except when it rained "just a little too much" in one area and this overflow caused an unanticipated strain on the system which manifested itself as a FIFTY FOOT WAVE (I am not exaggerating) that was created when the water slammed into the end of the tunnel, (its a BIG tunnel), and the wave backflowed through a huge valve, breaking 24" pipes apart and filling a large room full of electrical equipment, essentially rendering the entire Tunnel system useless until they could manually drain the system slowly and spend about 1.5 million dollars to repair the damage, which they did in about 10 days.

It's fixed, but they still don't know what caused the wave. The "wave" could happen again .

sounds like the system crossed a failure threshold of some sort.

What other systems will cross a similar threshold with more failures that "normal"?

Hey, it was just a 1/2" of rain. how much damage can that do?

-- PLONK! (realaddress@hotmail.com), June 13, 1999.