WATER: BIG DISTRICT: ORANGE COUNTY SANITATION DISTRICT

CALIFORNIA WATER RESOURCES BOARD:

http://www.swrcb.ca.gov/y2k/y2k.html

http://www.swrcb.ca.gov/y2k/epaocsd.pdf XXXXXXXXXXXXXXXXXX FROM PAGE 10 (12/19 pdf.)

BOTTOM LINE: "The District considers itself a leader in contingency planning and is taking Y2K preparations very seriously so that it will be as ready as possible for any unforeseeable problems."

RE: THE SYSTEM: However, thus far no embedded equipment has been replaced due to Y2K. RE: THE LAB: The older laboratory equipment had a very high percentage of Y2K compliance problems. About 40 percent of the equipment was found to have potential Y2K problems. Many could alsobe confirmed as non-compliant.

......SNIP.......

The Laboratory staff took a different approach to implementation than the other departments in the District. For the most part, the Laboratory did not spend as much time doing the assessment of their equipment. Because the size of the Laboratorys inventory and the complexity of some of the instrumentation, the equipment was divided into three categories: 1) equipment that must be replaced without testing because there were known Y2K problems, 2) equipment that must be assessed because they were critical to the Laboratory processes, 3) equipment that would not be assessed because they were of little Y2K risk or were not critical to the operation of the Laboratory. Approximately $420,000 was spent on new equipment in category 1. Much of this equipment was older and approaching the end of their useful life. Proof of Y2K compliance was a condition of the purchase for all of the replacement equipment. The assessment of the equipment in category 2 is nearly complete and no apparent Y2K problems have been found.

XXXXXXXXXXXXXXXXXXXXXXXXXXXX

1. SYSTEM OVERVIEW The Orange County Sanitation District is a wastewater treatment utility located just south of Los Angeles on the Pacific coast. The Districts service area covers 471 square miles and serves a population of approximately 2.2 million. Founded in 1948, it became a formal sanitation district in 1954. Much of the infrastructure in use today, however, was built and/or rebuilt in the 1970s and 1980s. The District consists of two large treatment plants, which include primary treatment and secondary treatment with activated sludge and trickling filters; 650 miles of sewer trunk lines; and 22 pumping stations. The smaller sewer lines, totaling approximately 7,000 miles, and connections are maintained by the cities within the District. Effluent from the District is discharged out a five-mile-long, ten-foot-diameter ocean outfall pipe. The last mile of the pipe is the diffuser section, which is about 190 feet below sea level. The effluent is half primary treated and half secondary treated wastewater. Average flow into the plants is 243 million gallons per day (MGD), with a total capacity of more than 480 MGD.

The winter is the areas rainy season, and winter storms can cause the plants to reach full capacity. About 10 MGD of water is reclaimed and sent to the neighboring Orange County Water District for final treatment. It is then used in the seawater intrusion barrier, landscaping, industry, and in the Districts own operations. The District produces an average of 500 wet tons per day of biosolids, which are digested, dewatered on belt presses, and used for local landscaping and agriculture. The methane gas from the digesters plus supplemental purchased natural gas is used in the Districts onsite electrical generation plants, which generate an average of 10,500 kW of electricity. The electricity generated is more than enough to run the entire Districts plant operations.

The excess power is sold to the local electric utility. The District has approximately 520 employees and is governed by a 25-member Board of Directors. The Board consists of mayors or elected council members from the 21 cities in the District, three members from Sanitary Districts located within the service area and one member who represents the unincorporated areas. The Districts budget for its 1999 fiscal year is $154 million, $45 million of which is for operations, $72 million for capital projects, and $37 million for debt service. 2. MOTIVATING FACTORS District management is determined to do everything possible to stop the Y2K bug from affecting its business or operations. The ramifications from disrupted service could be disastrous. Sewage could flow into the streets or into the Santa Ana River and coastal waters, causing risks to public health and local ecosystems.

XXXXXXXXXXXXXXXXXXXXXXX

4. SYSTEM REVIEWS This section describes the inventory, assessment/testing, and implementation processes of major systems in greater detail.

There are four major systems:

The control system includes everything responsible for controlling operations equipment, but does not include the equipment itself.

The embedded systems include the operations equipment that have embedded computer chips in them.

The Laboratory system includes all of the Districts equipment responsible for sampling and testing, inside and outside of its laboratory.

The business system includes all the hardware and software used in District administration and management. These sections will be followed by a discussion of influences external to the District, such as its energy suppliers, and a discussion of contingency planning.

XXXXXXXXXXXXXXXXXXXXXXXX

software, which has already been upgraded to a compliant version. 4.2 Embedded Systems The embedded systems review process was complex because of the great number of items to be reviewed. Throughout the process, there was the chance of something being missed, so the District tried to give every device in the plants at least two chances to be reviewed.

Figure 1, Embedded Systems Review, illustrates the embedded systems review process.

4.2.1 Inventory The District had an existing asset inventory that contained 80,000 to 90,000 items. Every item in the inventory was categorized, and the District decided that the only items that could be affected by Y2K would be in the electronic or instrument categories. These two categories contained about 20,000 items. Most inventories of this type have about a 2 to 3 percent margin of error, and the District estimated that its inventory has somewhat more error than that margin. Therefore, throughout the inventory and assessment processes, personnel attempted to find items that were not included in the inventory.

All items that had been assessed received a sticker.

Midway through the assessment process, a team walked through both plants looking for items that had not received a sticker and were not in the asset inventory. About 48 devices in need of review were found in this walkthrough.

When assessing items in the field, the team always asked the maintenance or operation staff for the area whether anything had been missed. Numerous items were found in this way.

The inventory and assessment processes not only helped with the Y2K assessment but also helped refine the Districts asset inventory for future use. Any item that was the responsibility of another department (such as the PLCs, which are reviewed by the PCI Division) was transferred from the embedded systems assessment. These transferred items were put into one of two categories at the end of the assessment: By Others -Miscellaneous or By Others - PLC. These categories will be described in more detail below.

4.2.2 Assessment/Testing The District received offers from Y2K database consultants to perform the assessment. These consultants have large databases of equipment items and compliance statuses, so that any District item that is in the consultants database could be quickly checked for compliance. The consultant would separately assess any District item not already in the database.

The District was skeptical of whether any of these databases would contain exact matches of the Districts devices, and it did not believe that an outside firm would be as able as the District in its assessment.

Therefore the District chose to assess the embedded systems in house, with the help of Bristol Systems, Inc.

The assessment was conducted in three phases:

Field Survey

Research on Potential Non-compliant Items

Testing Field Survey.

The database of 20,000 items was given to the maintenance and operations staff, who were asked to answer 12 questions about each item. The 12 questions were:

1. Does it have a battery?

2. Does it have a date/time display?

3. Does it have a keyboard?

4. Does it have a serial port?

5. Does it have a net connection?

6. Does it have a microprocessor?

7. Does it have a system management interface?

8. Is it programmable?

9. Does it have date dependent functionality?

10. Does it schedule its own maintenance?

11. Is it critical?

12. Is it still in use (not retired)?

A yes answer to any of the first 11 questions coupled with a yes answer to the last question sent the item on to further assessment. All items that received all no answers to the first 11 questions and yes to the last question were sent to management for review.

The Districts Operations management knows the risk of failure of each piece of equipment, so if there were any doubts about a particular item being taken out of the assessment process, management could add it back in for further assessment.

This management review proved to be very helpful. Some items that were taken out of the assessment process because they were considered to be too simple to have Y2K issues, such as power relays, were added back in by management and were later found to be non-compliant.

Approximately 4,000 items needed further assessment. These items were all physically examined by Bristol Systems experts for date functionality, clock chips, and CPUs known to have embedded clocks. Items that appeared to have no chance of being affected by Y2K were taken out of the assessment for management review.

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Research on Potential Non-compliant Items.

Approximately 2,100 items were passed on to Phase 2, but many were redundant. Some items were redundant because there were several instances of the same model in the District, while other items were redundant due to data entry problems. When the redundancy was removed, 553 unique items remained.

In Phase 2, personnel sought to collect at least two sources of information on each item. Manufacturer data was collected by accessing the Y2K section of the manufacturers web page or by contacting the manufacturer directly and asking for written information about the compliance status of each model the District owned. In some instances District personnel asked to talk to qualified engineers about the possible Y2K issues a device could have.

Technical information was also sought for each device. This information was often found in the Operations and Maintenance manual, which the District either already had or obtained from the manufacturer. Circuit diagrams were also used for insight into the devices functionality. All information was rated for quality. Once all the information was reviewed, items were either put into one of the following five assessment categories or were passed on to Implementation (Section 4.2.3).

By Others - Miscellaneous: This category consists of items that are the responsibility of other organizations or departments.

By Others - PLC: All PLCs are the responsibility of the PCI Division, so they are not part of the embedded systems review.

Continue Phase II: These items need additional inspection or research before a recommendation can be made.

The District may be awaiting more information from the manufacturer or still reviewing the available information.

No Further Action: Reliable manufacturer and technical information was found for these items, and they indicate that the device will have no Y2K problems.

System Analysis:

These devices alone probably do not have Y2K problems, but they may encounter problems with the passing of bad dates from/to other pieces of equipment.

In these cases, the whole system must be analyzed and tested together, possibly with outside help. For example, the District plans to have the manufacturer test the two generation plants systems this summer.

For the most critical systems, like headworks pumps, Bristol Systems and District staff wrote detailed reports on everything that affects the system and could possibly cause it to fail on critical Y2K dates.

Although they found no Y2K issues, they did find some single points of failure that were not recognized before. (With critical systems the District attempts to have redundancy in every aspect of the system.)

Testing. These items cannot be confirmed as compliant and they are critical enough to warrant the cost of testing. These items were passed on to Phase 3 of the assessment.Testing can be done through a number of processes, but generally, one representative model will be taken offline and tested by forcing the date forward to critical dates. If it is deemed necessary, other like devices may be tested later. Since testing is expensive, Phase 3 is performed only on critical devices for which other options are more costly or unavailable.

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

4.2.3 Implementation In the Implementation step, items were placed in one of four categories:

Contingency Plan: These items show some indication of possible problems but are not critical to operations and can be easily fixed or replaced, if broken, with little or no disruption to business as usual. These items are scheduled to be checked for problems on critical dates. There are currently about 200 in this category, and the District expects that the category will stay this size through the end of the year as some new items are added and some are moved to other categories.

Upgrade Software: For these items, the current software is non-compliant or uncertain, and new, compliant software is available. Currently 17 items are in this category, most of which are uninterruptible power supplies (UPSs).

Upgrade Hardware: These are cases in which a chip or other hardware is non-compliant or uncertain, and new, compliant hardware is available. There are currently 45 items in this category, most of which are power meters and monitors.

Replace: These items may be non-compliant, but upgrades are unavailable. For these items it may be less expensive to replace them than to test them. For example, the auto-dialers at several lift stations are currently in the Contingency Plan category because the District does not have sufficient information to confirm them as compliant. The District is considering replacing the auto-dialers, however, because of their importance. Upgrades are not available, and it would be less expensive to replace them than to test them. However, thus far no embedded equipment has been replaced due to Y2K.

XXXXXXXXXXXXXXXXXXXXX

4.3 Laboratory 4.3.1 Inventory The District has a large, sophisticated laboratory that does the majority of their testing and analysis.

They perform some daily sampling and testing, quarterly testing of industrial dischargers, and some long-term sampling and testing in the ocean around the outfall pipe.

Personnel inventoried about 250 devices that were sophisticated enough to have Y2K issues. Some of the equipment is run by specialized software that runs on PCs that are several years old.

In addition, many pieces of field equipment for sampling and testing were inventoried. Much of the laboratory equipment is connected to a Laboratory Information Management System (LIMS) from LabWare. LIMS is linked to an Oracle database.

4.3.2 Assessment/Testing The older laboratory equipment had a very high percentage of Y2K compliance problems. About 40 percent of the equipment was found to have potential Y2K problems. Many could als obe confirmed as non-compliant.

LabWare stated that the LIMS is compliant, but the District has done some preliminary testing of it to be sure since the LIMS is one of the most critical parts of the Laboratorys operations. The District is currently debating whether to test any other items in the Laboratory. While the Laboratory is important to the overall Mission of the District, it is not critical to short-term treatment operations, so the District is not as conservative in assessing its Y2K compliance.

4.3.3 Implementation The Laboratory staff took a different approach to implementation than the other departments in the District. For the most part, the Laboratory did not spend as much time doing the assessment of their equipment. Because the size of the Laboratorys inventory and the complexity of some of the instrumentation, the equipment was divided into three categories:

1) equipment that must be replaced without testing because there were known Y2K problems,

2) equipment that must be assessed because they were critical to the Laboratory processes,

3) equipment that would not be assessed because they were of little Y2K risk or were not critical to the operation of the Laboratory.

Approximately $420,000 was spent on new equipment in category

1. Much of this equipment was older and approaching the end of their useful life. Proof of Y2K compliance was a condition of the purchase for all of the replacement equipment. The assessment of the equipment in category 2 is nearly complete and no apparent Y2K problems have been found.

XXXXXXXXXXXXXXXXXXXXXXXXX

p.11 11 4.4 Business Systems 4.4.1 Inventory Other Y2K non-treatment systems were put through the Y2K process. The building heating and cooling systems, security systems, closed-circuit television system, fire detection and alarm system, District vehicles and their smog testing equipment, fuel management system, landscape watering system, leak detection equipment, and portable gas detection equipment were also inventoried.

The administrative areas of the District have a great deal of sophisticated hardware to be inventoried for Y2K. The District has about 500 Dell PCs and approximately 50 Dell laptops. The laptops are difficult to inventory completely because some are connected to the District network only occasionally and may be kept in a maintenance shop or office when not being used.

The District also has one IBM AS400 computer that runs the financial software.

Other office hardware that must be inventoried includes the network infrastructure, printers, copiers, scanners, and uninterruptible power supplies.

There is also a great deal of communications equipment throughout the District that must be assessed, for example, fax machines, internet connections, a telephone system at each plant, pagers, and two-way Motorola radios.

All District PCs run on the Windows NT or Windows 95/98 operating system and have Microsoft Office software. Also approximately 200 other applications need to be inventoried.

The Districts software inventory has been updated for the Y2K review. An application called ZAC has been installed. ZAC inventories all software on the network on a regular basis, and this has been found to be very helpful.

Outside the NT network, the Human Resources and Source Control Departments have a few workstations that run on an older Novell network. These are no longer used for current data but are necessary for accessing historical data.

All financial software is in one software suite from J.D. Edwards that has several modules. The software is continuously being upgraded and added to, which can make inventorying and assessing difficult.

There is an expected lull in activity this summer, however, which will allow the District to freeze the system for testing.

Further system enhancement and upgrade will occur after the New Year. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

P.13 (15/19 pdf):

4.4.3 Implementation The District is currently in the process of either replacing its Pentium-based 100 MHz PCs or upgrading the BIOS, depending on which solution best fits the needs of each situation. Replacement and upgrading of laptops is also well under way. The Internet connection has been upgraded to be compliant.

Non-compliant software can usually be easily fixed by upgrading to the latest version. Some of these must be purchased, but many smaller Y2K upgrades, such as the latest service packs from Microsoft, can often be obtained without cost on the Internet. The Districts Windows NT software is currently running on Service Pack 3, and it is debating whether to upgrade to Service Pack 4 or 5. Bristol Systems, Inc. tried upgrading its systems to Service Pack 4, and many of its applications malfunctioned as a result, so it is advising the District against the upgrade. As was stated above, the District will probably wait until Microsoft is more certain of its compliance statements before deciding on a service pack. For statements on the latest Y2K compliance statuses of Microsoft products go to the Microsoft web page at (http://www.microsoft.com/technet/year2k/product/product.asp).

The IT Department has found that Y2K has provided the District with an incentive to update and clean up its software and information systems.

Before the District began its Y2K Program, it had many different versions of each application and some obsolete software. Now all applications are being brought up to the same version level, and obsolete software is being removed. The District expects the increase in software commonality and the greater capabilities that higher version levels bring will not only increase the sophistication of District users but also decrease the workload of the IT Departments help desk.

XXXXXXXXXXXXXXXXXX

15 6.2 Energy and Other Services

The District already has a plan for electrical outages, but this plan is currently being expanded to apply to all utility outages. Some of the systems are dependent on water supply from outside suppliers. Some of the Districts energy needs are met by natural gas since one of the central generation plants relies on about 35% natural gas as fuel. It also depends on telephone and cellular service providers for communications, and on other private service providers such as its biosolids trucking service. The District feels that its service provider contingency plan is useful to have regardless of Y2K, because it could experience disruptions with one or several of its service providers at any time.

Y2K has provided the motivation to ensure that this plan is in place.

The Districts plants do not rely on electricity from Southern California Edison since the District is a net exporter of electricity.

Electricity from the grid is considered a backup in case the central generation plants fail, but the District plans to disconnect from the grid on New Years Eve to protect its systems from any electrical surges.

Southern California Edison agrees with this decision because it is more concerned about having too much energy on New Years Eve, with many people cutting electrical load or switching to generators, than having too little.

Critical equipment at the plants are still connected to diesel generators as backup in case the central generation plants and Southern California Edison fails.

The District does not have fixed generators for 17 of its 22 pumping stations, however, and they do not receive power from the central generation plants.

The five pumping stations with fixed generators have automatic switches that turn the generators on immediately when voltage from the grid drops.

All the other pumping stations are equipped with quick-connect switches that allow mobile generators to be hooked up quickly and efficiently.

The District currently has three mobile generators, and it is purchasing several more so that it will be prepared for a substantial outage.

The District is currently working on a rotation plan for the mobile generators in case of an outage affecting multiple pumping stations.

Some of the pumping stations have only 10 minutes of capacity in their wet wells, so a rotational strategy for these stations may not be possible.

Nevertheless, the District is confident, that with the generators it has and the new ones it is purchasing, it will be ready to handle the most likely scenario of a series of rolling outages.

Although the District is preparing contingency plans for a total outage, from talks with Southern California Edison, it believes that the most likely scenario will be rolling blackouts or brownouts, not a total area blackout.

Therefore, it is talking with the electric utility to determine which pumping stations are on the same transmission lines so that it can determine the likelihood of multiple pumping stations in the same vicinity losing electricity.

The District cannot put fixed generators at every pumping station because in most places it does not have enough real estate to do so. The generators it currently has are well maintained with monthly tests and occasional full-load tests.

The District must monitor its generator use carefully because South Coast Air Quality Management District regulations allow it to run its generators only for a certain number of hours each year.

The District is also designing a fuel plan to accommodate the generators needs and the fuel needs of its trucks and other equipment.

It plans to top off all storage tanks late in December and will assess the need for more fuel in case the local fuel depots cease functioning due to Y2K.

It also plans to ensure proper inventory levels for other supplies such as chemicals, spare parts, laboratory supplies, and other supplies that will allow the District to stay self-sufficient for a few days.

The Finance Department is working on contingencies, but fortunately the timing of certain transactions will allow it a few weeks of leeway in working out any Y2K problems it encounters.

The District receives its quarterly receipt from the County in mid-December, and its biweekly payday is the Wednesday before the rollover.

It is currently working on moving other transaction dates away from New Years as well. The District plans to complete its contingency plans this fall, which will allow enough time to practice with tabletop exercises. The District considers itself a leader in contingency planning and is taking Y2K preparations very seriously so that it will be as ready as possible for any unforeseeable problems.

-- Cherri (sams@brigadoon.com), September 05, 1999

Answers

Also see this link for a table of the Y2K readiness (as of July) of water/wastewater treatment facilities in the largest American cities:

http://www.senate.gov/~y2k/hearings/990715/chart4_tbl.gif

-- Linkmeister (link@librarian.edu), September 05, 1999.

Wow!! Looks like they have put in a lot of work on this...Just wish my city ( one of the infamous 11 that will (/) finish in 4th Quarter had been as dilligent!

-- K. Stevens (kstevens@ It's ALL going away in January.com), September 05, 1999.

Wow!! Looks like they have put in a lot of work on this...Just wish my city ( one of the infamous 11 that will (?) finish in 4th Quarter had been as dilligent!

-- K. Stevens (kstevens@ It's ALL going away in January.com), September 05, 1999.

It is much neater and easier to read all of this info and more directly from the The Water Resources Control Board

-- Chris (%$^&^@pond.com), September 05, 1999.

Orange County sure appears to be getting their shit together, so to speak, albeit at a very late hour.

Their concern seems to support many of our positions that the whole y2k business is a very serious concern. I personally learned a lot about all the different types of problems which a large organization such as this can expect, and how huge a job it is to prepare everything.

I had not heard that So. Cal. Edison was expecting "rolling blackouts and brownouts". I wonder if this is their OFFICIAL position on y2k, shared with their stockholders and customers, or if it was only Orange County Public Works who received this unexpectedly negative announcement.

I also hope that Orange County's pumps don't burn up if from low voltage during the expected brownouts.

Al

-- Al K. Lloyd (al@ready.now), September 05, 1999.

K. Stevens,

Where do you live? Maybe something can be found out for you.

-- Cherri (sams@brigadoon.com), September 05, 1999.