BERGEN RECORD
12/18/99

Y2K fix costlier than first expected
http://www.bergen.com/news/y2k19199912193.htm

By LOUIS LAVELLE, Staff Writer

Talk about rude surprises.

North Jersey companies thought they knew what it would cost to repair their year 2000 computer problems last year. And it wasn't pretty then.

Fourteen local companies who made predictions last year were looking at nearly $1.2 billion in Y2K remediation costs.

But in recent months, many have discovered that those original estimates, as high as they were, were optimistic -- in some cases extremely so -- and have been forced to readjust them upward.

The new estimates for those 14 companies: more than $1.4 billion, a 17 percent increase.

North Jersey is not alone. A soon-to-be-released survey on Y2K spending by U.S. companies shows that 90 percent now consider their original estimates too low, up from 82 percent in September 1998.

Eight out of 10 companies are spending more than a fifth of their information technology budgets on year 2000 remediation.

"It's taken a while for people to gain awareness of why it's a big problem and that's why they've been so surprised," said Noah Ross, a vice president for Cap Gemini America, the New York company that conducted the survey. "Once they started digging into it, they found there was more to it. That's a common finding."

Local companies have been struggling for several years to repair or replace computer systems that may be vulnerable to the Y2K problem, a computer error that may cause software to misinterpret dates, resulting in system failures or corrupted data.

In most cases, costs have mounted as a result of decisions made to replace equipment ahead of schedule -- as opposed to making less expensive repairs -- or simply because new, unanticipated problems cropped up.

Peter Rogers, vice president of information systems for American Standard Cos., said the $6.7 billion Piscataway-based global company turns up new computer systems that need to be repaired or replaced every time its technology experts gather for a monthly conference call.

The most recent discovery: 10 servers in the headquarters office that were not Y2K-compliant. The repair cost: $50,000.

"It seems like in every conference call, every month, something new crops up. That's where these costs are coming from," Rogers said. "We keep finding new areas over and over again that need year 2000 compliance."

The additional Y2K costs have driven up the company's estimate to $22 million, an increase of $5 million or 29 percent over its year-ago estimate of $17 million.

But American Standard is far from the worst-case scenario. Schering-Plough of Madison now expects to spend $95 million on Y2K remediation, an increase of $45 million or 90 percent from last year, and A&P in Montvale expects to spend $10 million, an increase of $5 million or 100 percent.

A&P did not respond to requests for information, but Schering-Plough in a statement said the increase in its Y2K budget was the result of additional computers in the company's research operation that required repair or replacement.

As costs at North Jersey companies mount, some -- such as Warner-Lambert in Morris Plains, where Y2K cost estimates have risen a modest 4 percent since last year -- have been able to keep other information technology projects on track.

But some have had to reallocate internal resources to the Y2K problem, delaying implementation of new technology initiatives until 2000.

At American Standard, the installation of new financial and management s Ross said he expects companies that have put off new technology initiatives in order to devote resources to Y2K finally to turn their attention back to profit-making and innovation.

"They've been spending their money on this instead of other things," Ross said. "The firms are poised for post-Y2K initiatives. There's a big, pent-up demand for new Internet activities."

Not all companies were taken by surprise by increased Y2K costs.

At AT&T in Basking Ridge, Y2K remediation costs increased 23 percent, from $600 million to $739 million. But the increase was the result not of unforeseen Y2K expenses, but because of its $53 billion purchase this spring of Tele-Communications Inc. TCI's Y2K costs have been folded into AT&T's, said company spokesman Dave Johnson.

"Y2K spending at AT&T has remained level and is closing right on target," Johnson said.

The company's extensive inventory of potential Y2K problems -- everything from mainframes to electric staplers -- precluded most surprises, he said.

"Because we did that inventory . . . from right up front we had a pretty darn good focus on what was time- and date-sensitive and what needed to be done."

Some experts say the latest round of Y2K cost increases won't be the last.

While the repairs to so-called "mission critical" systems will be finished by Dec. 31, unanticipated computer failures, lawsuits over Y2K issues, repairs to non-critical systems, and replacing temporary Y2K fixes with more permanent solutions will continue to drive up costs.

John Hayes, editor of Mealey's Year 2000 Report, which tracks Y2K lawsuits, said litigation costs alone could dwarf remediation costs as companies, insurers, suppliers, and customers struggle to assess liability for computer and business failures.

Keith Rhodes, director of computer and information technology assessment at the federal government's General Accounting Office, said the vast majority of U.S. companies have used a temporary solution to their Y2K problems and will need to make permanent repairs over the next 50 years.

"The bill comes due," Rhodes said. "If anybody thinks the price of year 2000 has stopped, they're incorrect. . . . Many people think on Feb. 1 God will be in heaven and all will be good with the world. That's not the case."

Company officials acknowledge the limitations of Y2K remediation estimates, which do not include litigation expenses and can not anticipate unforeseen computer failures or their consequences.

"We have lawyers, we have financial people, and we brought in outside people to make sure we're doing it right," said Rogers. "I honestly do think we'll be in good shape and we shouldn't have any major, major problems. We're very optimistic, but we're not taking anything for granted."

-- doomed (wallet@sucked.dry), December 19, 1999

Answers

Economy will collapse!

-- dead (in@the.water), December 19, 1999.

Imploding depression, not a recession

-- bad (worse@terrible.times), December 19, 1999.

Just 12 days left...8 'till when Martial Law perhaps gets declared.

I know that talking about an "average" company may be misleading, but a forecasting technique that I use will accept that phraseology. The sense I get is that the centerline of the Bell shaped curve is about 6 months behind on really thoroughly debugging their mission critical (roughly 9% of total) systems. The same technique is still forecasting a 5+, but definitely not a 7 or greater.

Question...what would be the implication for operating an entity for 6 months with barely functional mission critical systems?? (BTW, spot shortages of 3 part order pads in selected sizes have already appeared here in Central Arizona)

-- K. Stevens (kstevens@ It's ALL going away in less than two weeks.com), December 19, 1999.

"It seems like in every conference call, every month, something new crops up. That's where these costs are coming from," Rogers said. "We keep finding new areas over and over again that need year 2000 compliance."

The solution is obvious. Quit holding those conference calls and declare yourself Y2K Ready!

-- Vegas Boy (uno@who.com), December 19, 1999.

I have no problem finding the parts about companies ending up spending more than they anticipated. I can't seem to find the part about "can't make it", though. It sounds like they bit the bullet and spent what it took, which was more (sometimes a *lot* more) than they expected to spend. And sometimes they had to drop more productive efforts to concentrate on remediation as well. Not surprising.

But where does anyone say they can't make it?

-- Flint (flintc@mindspring.com), December 19, 1999.

But where does anyone say they can't make it?

"We keep finding new areas over and over again that need year 2000 compliance."

Note that they don't say "we kept finding new areas", but "we keep finding new areas". Do they have to say they can't make it? Take a look at the calendar and tell me how much time they have left (with a full day for testing). Then tell me whether they're going to make it.

-- Steve Heller (stheller@koyote.com), December 19, 1999.

Doomed,

Your email address says it all:) My wallet is also sucked dry, and now faces the danger of implosion.

-- Ocotillo (peeling@out.===), December 19, 1999.

Steve:

In the sense of finding, fixing and verifying every last date bug, I sincerely doubt that any organization will ever "make it". Those bugs will continue to crop up forever. Especially once we start hitting hard-wired windows. So there's no doubt they keep finding problems, and *will* keep finding problems for decades.

But there seems, at least to me, to have been a kind of verbal sleight-of-hand going on here. The term "make it" is used interchangeably between "fixing all bugs" and "remaining in business". So we read that businesses won't "make it", followed by visions of dominoes as businesses fall over one after another. The hidden (and completely false) implication is that remaining date bugs mean inevitable business failure. This implication arises from using the same term to mean both conditions -- imperfect remediation and business failure.

So as I said above, I'm not surprised the problem is turning out to be more pervasive and difficult than many companies anticipated. It's a nasty problem, to be sure. But it can't be entirely bad news that companies have realized this and boosted their budgets to compensate. It would be worse news to discover they refused to admit it and did nothing about it.

-- Flint (flintc@mindspring.com), December 19, 1999.

Flint,

I have some programmer friends who work at one of the companies mentioned in the article (a biggie and no, I won't say which). I just spoke to one of them on Friday to make some lunch plans for next week and was told they are SCRAMBLING to try and get the mission critical stuff done. They are *FAR* from being done as they keep 'discovering' systems they had overlooked in their initial inventory, systems that WILL shut down production lines if not finished (it's NOT American Standard). Will they go out of business? I doubt it, they are too large. Will their lines be shut down for at _least_ several weeks? Count on it.

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), December 19, 1999.

TECH32:

Sounds like damage control time. Will all their lines shut down, or just some of them? Does anyone else produce competitive products on (hopefully) working lines who can (hopefully) ramp up?

I'm NOT trying to minimize the problem -- I recognize that such things will be all too common. I sure you, just like I, have had to put on your troubleshooter hat and say "OK, we're in deep shit, what do we do now?" I regard the first few months of next year (at least) as a recovery period, whatever it takes. That's when the *real* triage starts.

-- Flint (flintc@mindspring.com), December 19, 1999.

Flint,

Sounds like damage control time. Will all their lines shut down, or just some of them? Does anyone else produce competitive products on (hopefully) working lines who can (hopefully) ramp up?

I don't know if all lines will shut down or if competitors can pick up the slack. All I know is they ARE in damage control big time and the bosses at the top are NOT happy with anyone at the moment (even those not involved in production). If it were one or two lines I don't think my friend would have sounded as worried as she did.

We still have tentative plans for lunch though it might have to be in the company cafeteria since a lunch 'hour' is, for the time being, a thing of the past there. I'll see if I can find out more if/when we get together.

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), December 19, 1999.

1. I agree w/Flint. Some failures are probable, but total failure in US is improbable.

2. If a few production lines are down a few weeks, okay; isn't that why people have prepared for Y2K disruptions.

3. My disaster training taught that the Recovery Period is about ten times longer than the Emergency Period. So a few weeks of disruptions and 2-3 month of recovery seems to me to be best case scenario for the US.

4. Bigger problems overseas, likely to affect finance and energy in the USA, may extend the Emergency and complicate Recovery. So probable case appears to be 4-6 weeks of disruptions and 12 months of recovery.

Disruption in some of one's production does not equal a systemic breakdown. I think people that expect TEOTWAWKI underestimate the robustness of our economy and the ingenuity of goal oriented people.

-- Bill P (porterwn@one.net), December 19, 1999.

Flint: I'd ceratinly agree with you that these underestimated remediation efforts do not equate to "won't make it". Indeed, these 14 companies may well have taken Y2K seriously enough to be able to maintain continuity of business. But clearly, even organizations who are taking it seriously have underestimated the costs. This is one prediction made here in the forum that was pretty much on the money. But it didn't take any magic ball or insider information to guess this one right - one only needs to acknowledge the relatively short but perfectly clear history of such large projects.

But the real story, the one that peeks out between the lines, is not the impact of Y2K on the 14 companies profiled above, but rather the implications of this article for the thousands of organizations whose remediation efforts didn't even come close.

Bill: I follow and basically agree with your logic here, but I would add that the real wild card is confidence. Our economic systems (banks, markets, etc.) are highly dependent upon the confidence of the participants. I believe the situation you describe could easily result in a widespread loss of such confidence. A lot of money, in all its various forms, would begin to shift - at a much, much faster rate than it does today. Such a move toward 'safe havens' would be enormously disruptive and therefore, I would agrue, has to be factored in.

Only if the levels of any actual problems/disruptions do not exceed the tolerance of the main financial participants (as a whole) will confidence in our overall system be maintained.

Banking could come through Y2K totally unscathed (technically) and still suffer immensely if confidence is lost due to other issues not directly under the control of these financial institutions.

The trillion-dollar question is, what is our collective threshhold of pain?

-- Arnie Rimmer (Arnie_Rimmer@usa.net), December 20, 1999.

Flint:

Link is way opff my radar screen, but an interview of the CEO of Massey Ferguson (sp?) tractor maker had a startling comment about alternative suppliers. Somewhat paraphrased from memory, "If our suppliers fail us, we are finished. It can take us up to four or five years to reestablish suppliers." While, guessing at the wording, the meaning was that direct. Thus, the alternative supplier model may be a fiction in many cases.

-- Dave (aaa@aaa.com), December 20, 1999.

Flint,

Ok, I just got the skinny. Middle management (ie. the Pointy Haired Ones) continue to proclaim that all is well. The technicians/programmers on *some* of the production lines are standing firm that *some* of their systems will fail (enough to stop production for at least a few weeks). They have demonstrated this to their bosses but the denial continues. They are ALL supposed to report to work by 6:30AM on the 2nd. Anyone who's not there by 6:45 will get a call from the Senior VP of their division.

So, not ALL production lines will fail, but if the people working on these systems are to be believed at least *some* of them will (more than one or two). Hope this helps.

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), December 21, 1999.

But it can't be entirely bad news that companies have realized this and boosted their budgets to compensate. It would be worse news to discover they refused to admit it and did nothing about it.

Okay, but the problem is that one can play both sides of that game:

1. If they admit they underestimated the problem, then they should be praised for this admission and taking it seriously, so that's good news.
2. If they continue to claim that they have estimated the problem correctly, then that means they will get done on time (with one day for testing?), so that's good news.

Can I ask you what you would consider bad news, if anything?

-- Steve Heller (stheller@koyote.com), December 21, 1999.