Aging IT systems still trundle along in state and local government back offices.

By Shane Peterson

August 2004

Like many other governments, Greensboro, N.C., relied for years on obsolete mainframe technology to handle many agency processes and applications. IT staff knew the clock was ticking on the system, said Larry Kerr, ERP director, so the city hired Gartner to analyze its IT environment in 1998.

Imagine their surprise when analysts told city officials only two people in the entire United States could support their mainframe's operating system.

The consultants recommended the city migrate off the mainframe pronto because so many critical lines of business depended on it.

Greensboro took their advice and began implementing an ERP package, and the last of the old mainframe applications will be gone this year. Although Greensboro is updating its technology, plenty of jurisdictions continue to rely on aging systems for mission-critical functions.

A combination of factors -- such as tight budgets and fear of failure -- prompts some government agencies to hold on to systems well past their prime. Some technology is so obsolete, however, that vendor support is evaporating.

That was the case in Greensboro.

"We finally hit that time when the vendor said, 'No thanks,'" Kerr said. "Our MIS folks had great difficulty getting a one-year contract for the year we're in now -- this calendar year -- so our support plug is going to be pulled at the end of this year."

All of the city's mission-critical systems were on that mainframe, he said, and the only stand-alone systems in the city's infrastructure were operational systems for particular departments.

"The core that drives the whole organization was on that system," he said. "Every one of them were custom-written programs. Every one of them ran on that same operating system. We were extremely dependent on that combination of hardware and software. We were fortunate in the last 10 years of its life it was very, very stable."


Behind the Curtain
Typically the public isn't aware of the archaic nature of some government IT systems. Some of the oldest applications are in drivers' services, vehicle title and registration services, unemployment insurance, and child welfare.

As long as people get their new drivers' licenses in a timely fashion, nobody realizes the data on that shiny new license comes from 20-, 30-, and in some cases, 40-year-old mainframes.

According to the Center For Digital Government's Integrated Registry: Motor Vehicles, a 2003 survey of 33 states and three provinces of Canada, Indiana relies on a 43-year-old system to process drivers' services information (its last major enhancement was in 1999); Louisiana's system is 34 years old (a major enhancement is in progress); and three other states' systems are 33 years old.

The average drivers' services system in states surveyed is 19 years old, according to the center's research.

On the vehicle title and registration side, according to the same survey, California is using a 38-year-old system (last major upgrade in 2000); Missouri's is 35 years old (last major enhancement in 2003); and three other states are using 33-year-old systems.

The average age of vehicle title and registration systems in states surveyed is 18 years, the center's research found.

While the numbers are close, the majority of states surveyed are planning major enhancements to their drivers' services systems rather than system replacements. Yet for vehicle title and registration systems, the opposite is true -- more replacements are planned than enhancements.

State and local governments may not be eager to publicize the gory details of the old, and potentially cranky, IT system processing such things as driver's license information or unemployment insurance claims. This isn't to say CIOs or IT managers have their heads in the sand -- if a system goes down, everybody knows there will be hell to pay -- but the systems work well for the most part, even if they're not exactly stuffed with 21st-century functionality.

At a certain point, old systems' idiosyncrasies -- relying on batch processing, the limited (or nonexistent) ability to run ad hoc reports, new program requirements that translate into costly and labor-intensive programming changes -- simply outweigh their usefulness.

This is when IT departments find themselves painted in the corner.

If they had their druthers, agencies would replace the systems. Who doesn't want new technology? The decision to migrate or not to migrate comes down to comparing the cost of supporting aging IT systems with the expense of implementing new technology.

Adding a cloud of uncertainty is that moving to modern systems is clearly a process fraught with danger: What if implementation goes awry? What if costs shoot through the roof? What if users resist the new system? What if you bet wrong on a particular type of technology? What if things don't work?


History Lesson
Perhaps the most inglorious attempt to scrap an aging IT system and move to a new one happened in California. The state's Department of Motor Vehicles (DMV) relied on archaic mainframe environments and databases to process driver's license and vehicle registration transactions. Further complicating the situation was the fact that the databases were written into the operating system, which made for fast searches but caused headaches when attempting to program changes into the systems.

The state reached a decision in the mid-1980s, after undertaking a feasibility study, to replace the system -- a process that dragged on until late 1993, when the project was finally killed by then-DMV director Frank Zolin.

When he became director of the DMV, Zolin inherited the ambitious systems-replacement project, which sought to merge the driver's license and vehicle registration databases into one gigantic database.

Zolin eventually resigned, along with two other key state officials, because of the failed project's fallout. The mess generated widespread anger and frustration as a slew of newspaper stories detailed the failure's costs (anywhere from $43 million to $50 million, depending on the newspaper); alleged malfeasance on the part of state officials; and strained relationships between the state and the vendor chosen to implement the new system.

"The status of the project was quite different than my preliminary briefings," Zolin said in discussing his initial surprise when reviewing the project. "I became director in March of 1991. I probably didn't realize the project was in trouble until that summer -- July or maybe even August."

He said the hardware -- Tandem Cyclone mainframe computers with a proprietary architecture -- was already purchased when he took over. Though the selection of Tandem hardware later caused plenty of controversy when the project was in its death throes, Zolin said the hardware was never the problem.

"The problems all stemmed from the software and the inability to program our business practices properly," he said. "The bottom line is it was the software development that failed."

The original contractor handling software development continuously fell behind and missed benchmarks, he said, and the decision was reached before he took over as director to buy out that contract and transfer software development responsibility to DMV data processing staff.

Zolin said if he had not taken over direct control of the project when he took the job, he might never have known its true status.

"I would have probably never killed the project the way I did if I hadn't had that personal involvement and gained that knowledge," he said. "If I had just depended on reports, I wouldn't have done it.

"I didn't know the project was in trouble until the summer, four or five months after I got the job, but that doesn't mean people didn't tell me," he said. "Because a few people did walk up to me and tell me, 'You know that database redevelopment project you have, it's never going to work.' I think it's like a disgruntled employee or a staff person's opinion, plus you've got all the people that have already invested in the project, whether it be the vendors or staff, they're all telling you it will work."

Zolin said he tried for 18 months to save the project, partly because of the investment the state had already made, and partly because he kept hearing how the new technology would solve a lot of operational problems in the DMV's driver's license and vehicle registration records processing.

There's also no denying this sort of project takes on a life of its own after enough time passes -- if money keeps getting appropriated to the project -- nobody wants to kill it. The project's sheer weight and mass becomes a protective shell.

"In the reality of state government, it's tough to kill a project," Zolin said. "I know that after a career in public service. That's why I took 10 to 12 months developing the case, checking out every argument to save the project, and I felt at the time I killed the project, I was 100 percent justified. If you go back and read [accounts in the press], I was criticized for a lot of things, but nobody said the project would work and nobody criticized me for saying, 'This thing is a failure. Let's not spend good money after bad.'"

It's been 10 years since the ill-fated DMV project, and the aging IT system still hasn't been replaced. Clearly California is having budget trouble, but whispers from the Capitol intimate that nobody wants to touch a renewal project because memory of the spectacular flame out still lingers.


In With the New
Though such projects can be dangerous, several states, including Massachusetts, have embarked on campaigns to replace old systems before it's too late.

In early April, The Boston Globe ran a story titled Massachusetts' Scariest IT Project that explained the state's decision to replace its existing Medicaid Management Information System (MMIS), a 20-year-old Medicaid claims-processing system that was last upgraded in 1991.

Oversight of the replacement project was given to Louis Gutierrez, CIO of the Massachusetts Executive Office of Health and Human Services (EOHHS), who said he knows the task will be complex.

"When you look at the number of interfaces, the volume of business, the number of things that have to go right every day, and how you work your way through the organ transplant, you've got to be really careful with the complexity on these things," Gutierrez said, adding that the state hopes to finish the replacement by July 2006.

The existing system is fully depreciated, Gutierrez said, and though Massachusetts isn't thrilled to undertake a costly project -- the state is willing to spend upward of $70 million on the replacement -- it's not a matter of should it be done, it's a matter of when to do it.

"You ask yourself, 'How much longer do we continue patching the past, versus really setting the new foundation for the future?'" he said. "We felt that not doing it was just shifting the issue to the future. There's a kind of 'progressive entropy' that happens with large systems. They start out with some focus as to purpose, but after 20 years of patching, workarounds and change in the industry, you've got a situation that really needs radical attention."

He compared problems with the state's existing claims-processing system to similar problems in the private sector with health-care information systems, noting that a lot of health care these days runs on undercapitalized, very complicated application foundations.

"The situations are very analogous," he said. "One thing that raises, and it's an important thing, is people don't think about turnarounds in a public-sector context because the public sector will always be here.

"We very much look at the current situation in human services as a big turnaround operation," he said. "We're in the middle of a huge reorganization here. We've been through the bleakest of financial times. We've got some very aged business support systems we're trying to tackle. I think there's more commonality than difference."


Discretionary Spending
The other tough part of replacement projects is finding funding. Greensboro IT staff had to justify the significant expenditure on a new ERP package from Lawson Software to the City Council when the old system was still chugging along, Kerr said.

"Everything worked well for years," he said. "There were no warning signs. There was nothing falling apart. There was nothing smoking every time you ran payroll or something like that. There really was nothing to express urgency, until we had somebody who could look at this thing through different eyes."

Though the Gartner review helped, the city conducting its own IT strategic planning process probably helped more, he said. During that process, agencies expressed frustration over the limitations of the mainframe environment and the applications developed for it by the city's programming staff.

The problem for the agencies, Kerr said, was that the applications were isolated and unable to share information. The programs and mainframe environment also wouldn't support new technologies. As a result of the internal planning, agencies agreed to make some sacrifices to free up money for mainframe replacement.

Another factor was the MIS director's promotion to assistant city manager.

"He was really the key person in dealing with the council," Kerr said. "Essentially, the message started going to the council, well over a year in advance, 'We think we're going to make Y2K OK, but we've got to have a pretty massive infusion of money for our basic technology, and it's got to start at the foundation level. It's got to start in building our mission-critical systems over again because we just have not maintained those in a way that they should have been for a lot of years.'"

It took a year, he said, but the City Council supported the mainframe replacement, knowing full well that money would take away from other, more politically attractive projects, such as a new park or similar public works.

"Our council was very good in dealing with that," he said. "They accepted that we hadn't put the money into technology we probably should have in recent years, and it was going to take a change in priorities to do that."

In Massachusetts' case, it helps that the state can use federal funding for some portion of the replacement project. In addition, leaders of the state's EOHHS support the project, said Patricia Wada, project manager for the new MMIS project, and they have been key lobbyists for the initiative, with individuals in charge of the budget process or technology initiatives statewide.

Project team members also were given the opportunity to discuss the project's goals, costs and time frame with a number of individuals in the state Legislature, she said. Before the EOHHS released the request for response, the project team explained potential trouble spots and listened to outside concerns about how to deal with them.

In addition, the health-provider community in Massachusetts has not been shy about telling the state's Medicaid office about the current system's shortcomings, said Rosemarie Day, COO of MassHealth operations.

"They interact with our system daily, and have very visible and regular complaints I get to hear about a lot," Day said. "We know the ugly truth of our system. They know it, and it's not any big secret."

She cited the state's desire to charge a co-pay to its members for services as an example. It took 18 months to create a co-pay option in the existing system for only two health services, and the state was forced to scale back from its original plan of extending the co-pay to all its health services.

Medicaid expenditures account for a large chunk of Massachusetts' budget -- approximately $6 billion of the state's $24 billion total budget -- and the state, like many others, would dearly like to whittle that chunk down. The current claims-processing system is costing Massachusetts plenty of money and presenting plenty of obstacles to front-line staff, said Day.

"We've found that because our system is so outdated, the tail is wagging the dog," Day said. "So when you're trying to do some creative new program initiatives that might yield some cost containment, we don't have the ability to tweak things. We tend to have to do very simplistic things because that's what our system can do. That's a problem for our members and taxpayers as well."


House of Cards
Another Massachusetts agency going down the replacement path is the Department of Labor and Workforce Development (formerly the Division of Employment and Training), which relies on a 30-year-old Unisys mainframe system to process unemployment insurance benefit payments.

Jeff Ritter, CIO of the Department of Labor and Workforce Development, said the system's age isn't necessarily the problem.

"It's maintenance of the business logic," he said. "The Defense Department uses this equipment. The IRS uses this equipment. NASDAQ uses this equipment. It's not the hardware or its OS, per se. It's the 25-year-old, give or take, business rules invented in COBOL code that are the issue."

When the benefit payment system was written, he said, people seeking benefits went to a local unemployment office to file a claim. As a result, the office from which the claim originated was a critical piece of information, and the database used to process the payments contained a field for the office number.

Staff then used that field to drive some business logic, Ritter said, such as processing wages, adjudicating a claim or scheduling a hearing. If the state wanted to reach the person filing the claim for any reason, a message was sent to the office, which was instructed to contact that person.

"All of the business logic associated with 40 offices made complete sense in 1984," he said. "Today, we have precisely zero offices where you can walk in and file a claim because it's done over the phone or the Internet. All of the business logic associated with those 40 offices and the way data gets structured in the database is, at best, moot.

"If it were just moot, that would be fine," he said. "But some of the processes run off of which office you send the paperwork to. We have four virtual call centers where adjudication is done, and we can't send all the paper to every call center. We continue to build a house of cards on top of that data structure -- that doesn't make sense in the current business environment."


Full Circle
Still, the nature of replacement projects -- long-term animals that cost a lot of money to feed and care for until they can walk on their own -- often keeps politically sensitive agency leaders from making the commitment.

"Nobody is going to get elected to office or be deemed more popular politically because they replaced the mainframe down at [the Department of Labor and Workforce Development]," Ritter said. "Any savings that accrue due to reduced staffing, more efficiency and better outcomes may accumulate to your successor in office, not you.

"The timing is such that it takes a real visionary willing to commit to a project that you know you're not going to be around to see the end of and benefit from," he added. "That is one of the most important reasons these projects are hard to get started."

Govenment Technology

-- Anonymous, August 04, 2004