Entergy christened its Performance Monitoring and Diagnostic Center several years ago to leverage the expertise of its most senior operators and technicians across the company’s entire fleet of plants. The center also makes use of advanced software tools that increase plant availability and reliability by identifying faults before they become major, unplanned outages. The center paid for itself for years to come with a single “big catch” last year.

Entergy has a long history in plant performance monitoring, beginning with the first implementation of its Operations Information System (OIS) over a decade ago. The introduction of market competition for reliable, low-priced power initially drove the investment in new data collection and evaluation tools at the plant level.

About the same time as the OIS was introduced, Entergy embarked on a best-in-class evaluation of its entire fleet. One result of that study was an adjustment of appropriate plant staffing levels that was based on only routine plant operations and maintenance, to reflect the industry’s long-term readjustment to a market-driven power supply system. The bottom line: Plant staffing was reduced to minimum levels during those early days of market competition.

The OIS at the time was a simple menu-driven program used to facilitate access to plant performance databases collected by OSIsoft’s PI Historian software. General Physic’s EtaPro, used for thermal performance calculations, rounded out the software toolbox available to the predictive maintenance (PdM) staff.

Unfortunately, the same initiative that reduced the O&M staff also reduced the use of the OIS system and other PdM maintenance tools so vital for monitoring plant equipment health.

Get it together

By 2001, Entergy had decided to reexamine the business case for a centralized Performance Monitoring and Diagnostic Center (PM&DC) to capture the expertise and leverage the skills of senior plant O&M professionals across its fossil fleet of 18 key power plants. Entergy also was not immune to the aging workforce pains that have hit virtually every utility, so it was seeking to leverage a shrinking cadre of highly seasoned power plant operations experts.

Entergy subsequently gave the centralized monitoring concept a green light, and the PM&DC was staffed and commissioned in 2002. The PM&DC was created in The Woodlands, Texas, a half-hour drive north of Houston. Its mission is to “support plant objectives to achieve fleet commercial excellence through improved unit performance, equipment condition, and operational risk management” (see sidebar).

According to Gary Barnes, superintendent of Entergy’s PM&DC, the center completed its sixth year of providing 24-7 coverage of routine monitoring and diagnostics the end of July. The OIS and PI systems still are the primary means of accessing plant data, but the center’s experts’ problem-solving is greatly enhanced by a suite of custom performance evaluation tools, including the latest version of SmartSignal’s EPI*Center, for early detection of performance anomalies and data mining.

PI servers are now located at 18 plants, and the OIS is at 36 units in the fleet. Special purpose utilities have been developed to further facilitate data extraction from PI for analysis. Barnes noted that the PM&DC has a strong can-do attitude, thanks to its seasoned staff, a full suite of software tools, and an in-house maintenance management system that tracks center activities.

The center also has implemented an alarm system for key performance parameters of critical equipment at each plant. Real-time performance monitoring and diagnostics through pre-built PI-Process Book displays and General Physics EtaPro have also been implemented in the PM&DC. Custom process books and data links have been built for trip analysis, unit/equipment problem diagnostics, and special monitoring. RBMWare is used by the PM&DC staff to access detailed vibration data.

Entergy recently upgraded its PM&DC to SmartSignal’s EPI*Center (see “Making PM systems sweat the small stuff,” POWER, May 2008), which provides a significant improvement in capability and performance with data access speeds that are up to five times faster. EPI*Center eases the model construction and maintenances task and allows links to other reference data such as drawings.

Anomaly detection and alerting via advanced pattern recognition (APR) software using SmartSignal’s EPI*Center, which pulls near-real-time data from the plant PI servers, is installed on 36 units. APR software filters data for subtle changes in parameter values or trends and alerts the PM&DC staff when they occur. The important difference is that these alerts occur much earlier than the plant’s standard supervisory systems alarm setting would, giving operators more time to react when a critical event occurs (Figure 1).

1. Early detection is the key. Early detection of impending equipment failures requires a custom mix of specialized software tools and highly experienced operating and technical specialists. The Entergy PM&DC routinely identifies equipment problems before the plant’s supervisory instrumentation reaches alarm limits. Source: Entergy

In addition to the software toolkit just described, a member of the PM&DC staff attends the morning Entergy dispatch meetings to gain perspective on system issues plus future dispatch plans, unit testing, outages, and the like. The PM&DC maintains a daily list of units to be started up, shut down, or tested, and a generation status page for all units in the fleet; it also responds to alerts of unit start-ups, shutdowns, and operating derates. The PM&DC even has connected to the company’s automatic generation control system to get real-time notification of unit trips (breaker trip alarms) and other important data.

Super staff

Staffing the center around the clock requires a total of nine PM&D specialists: two per shift—one with an operations background and the other a technical specialist with an instrumentation and control (I&C) or PdM background—for four shifts plus one relief person to cover absences. The background of the specialists is necessarily broad, and they represent all areas of Entergy’s service territory and all fuel and unit types.

Staff experience levels are also impressive. The typical operations specialist has 20 to 30 years of operating experience on multiple units. Three-quarters of the technical specialists have an I&C or electrical background plus 20 to 30 years’ experience and cross-training in operations. The staff vibration guru has more than 30 years’ experience. An IT specialist rounds out the technical staff.

Barnes noted that in the early days, the PM&DC was careful to avoid being perceived as “Big Brother” looking over the shoulder of plant operators—just waiting for someone to make a mistake. Instead, PM&DC operators took great pains to show that they are another set of calibrated eyeballs available to the plant staff to watch start-up and help identify problems early.

To put a face with a telephone voice, PM&DC specialists visit several plants once or twice a year to share operating information, available services, and to build a close working relationship with the plant operations department. Plant and field office staff are also welcome and are encouraged to spend a few days working with specialists in the PM&DC. Specialists work closely with subject matter experts in the Plant Support and Tech Support departments to solve more specialized problems.

Plant operators soon learned that the PM&DC staff was, according to Barnes, “not looking over their shoulder, but rather, watching their back.” Today, the PM&DC specialists all lend assistance to the shift supervisor and control room operator during every unit start-up and during most shutdowns. There is an audible alarm in the PM&DC with every unit circuit breaker trip.

Barnes noted that the PM&DC has an equal emphasis on operations and equipment monitoring. The typical shift for a specialist in the Entergy PM&DC is spent monitoring operations at several assigned plants using the OIS/PI as the primary means of accessing plant data for routine monitoring. As mentioned earlier, the specialist will also carefully monitor unit start-ups and shutdowns, which are complex, especially for supercritical plants. The expertise of the operations specialists focused on monitoring key functions and data can be especially helpful during the controlled chaos in a plant control room during start-up or shutdown. When all doesn’t go as planned, the data collection and diagnostic capability of the PM&DC is critical to assessing corrective actions that affect plant operations.

PM&DC specialists also continuously look for early signs of emerging equipment problems with failed instrumentation, respond to requests by plant operators to help identify areas where lost generation can be recovered, and assess other equipment and performance issues.

If an anomaly is identified by the software tools and is confirmed based on the specialist’s experience and expertise, then the first step is to compare the data with expected values calculated by the OIS, when those are available. The specialist investigates by reviewing historical trends and by making comparisons with similar units or equipment to determine if they’re dealing with bad instrumentation, process change, or an equipment problem. If the specialist determines that there is a bad data point or an equipment problem, the next step is to query their automated integrated maintenance management system, or AIMM, to determine if the plant is already aware of the problem or not. If it isn’t, then the plant is contacted and alerted. If the problem is minor and doesn’t affect equipment or operator safety, the alert is made by e-mail to the shift operators and other key staff. If the alert is critical (meaning a unit trip or equipment damage is imminent or staff safety is endangered), then the specialist will call the control room to report the situation.

Catch of the day

Entergy has made a significant investment in establishing and operating its PM&DC and has an enviable track record of early identification of impending failures. Spotting the steam turbine bearing damage on Sabine Unit 3 and the broken forced draft fan vane regulating arm on Independence Unit 1 are just two examples. These and many other early alerts allow PM&DC specialists to identify problems before they became catastrophes, saving potentially hundreds of thousands of dollars with each early alert and preventive repair.

When asked about the most spectacular success of the PM&DC, Barnes was quick to discuss what has become known inside Entergy as “the catch.” The Waterford Unit 2 generator failure intercept is perhaps the most profound example of why a monitoring and diagnostic center is vital for any utility focused on reliable and safe operation of its power plant fleet. The catch undoubtedly not only saved the plant from catastrophic damage that would surely have required a months-long unplanned repair outage, but it also could well have saved lives.

The catch also demonstrates that there are possible failure modes that plant supervisory instrumentation cannot identify and alarm in sufficient time to prevent damage.

Anything but routine

Routine monitoring of Waterford Unit 2 at the PM&DC, including an evaluation of the turbine generator system using SmartSignal’s EPI*Center software, was completed on December 12, 2007. No abnormalities were found at the time (Figure 2).

2. Routine operation. PM&DC staff evaluated the Waterford Unit 2 generator operating data the afternoon of December 12, 2007, and found nothing out of spec. Courtesy: Entergy

The next morning, bearing number 8 vibration (horizontal plane, exciter bearing) levels began to rise, and the EPI*Center placed this reading on the watch list for the first time (Figure 3) at 12:59 p.m. Note that the vibration levels were still lower than the alarm limits at the plant, yet the rate of change trend away from the expected level of vibration was of concern even though all readings were still below the plant’s 5 mil alarm setting. By 1:39 p.m. a second alert and watch list incident was posted on bearing #7 at the PM&DC.

3. Vibration levels rising. On the morning of December 13, slowly rising vibration levels in the Waterford Unit 2 generator were visible. The vibration levels triggered additional scrutiny by PM&DC staff in the early afternoon. Courtesy: Entergy

The PM&DC specialist on duty contacted Waterford’s control room by phone to alert operations staff about a developing vibration issue that was “uncharacteristic” on the Unit 2 generator. Specifically, number 6, 7, and 8 bearing vibration readings were increasing, while number 5 was decreasing. The specialist recommended to plant operations staff that vibration readings be taken as soon as possible, because the unit was schedule to come off-line after the evening load peak.

At 1:56 p.m. the PM&DC specialist also contacted the Plant Support vibration specialist for consultation and followed up with the plant production superintendent on the potential seriousness of the situation. By 3:20 p.m. the vibration specialist was en route to the plant to take vibration data. The highest readings were still approximately 5 mils (the alarm setpoint) at this time.

The vibration specialist arrived at the plant and took additional readings as the PM&DC shift changed at 5 p.m. The plant shutdown began at 5:48 p.m. The PM&DC specialist again contacted the control room to advise operations staff to expect elevated vibration levels to appear on their supervisory instruments during the coast-down period, when the rotating speed passed through critical frequencies. The specialist also recommended that operations staff break condenser vacuum as soon as possible following shutdown to shorten the coast-down period and to reduce the turning gear speed to extend the cool-down period (Figure 4). Shutdown was completed without further incident.

4. Break vacuum early. The PM&DC staff recommended an accelerated shutdown of the steam turbine to minimize the time the turbine spent at critical speeds, where vibration levels peak. Courtesy: Entergy

Entergy engineers first expected the problem to be exciter-related, but an inspection found no problems there. The decision was then made to pull the generator rotor for inspection. A circumferential crack in the shaft at the “J” section was found that ran 180 degrees around the shaft (Figure 5) and at least 1.5 inches deep (Figure 6).

5. Cracked shaft. A shaft crack was found 180 degrees around the shaft when the generator rotor was removed. Courtesy: Entergy

6. Money shot. The location, length, and depth of the generator shaft crack meant a catastrophic failure was imminent. The depth of the crack was estimated as 1.5 inches. Courtesy: Entergy

The staff’s consensus opinion was that the crack was propagating while the generator was operating and was sure to have catastrophically failed if it had remained on-line to the normally scheduled shutdown later that night.

Repairs were later estimated at approximately $5 million. Costs for picking up the pieces and putting out lube oil and hydrogen fires—which would have resulted if the shaft actually had failed—were estimated at between $20 and $40 million, not including personnel injuries.