A September 2005 power outage that affected two million people in the California Southland was initiated when workers cut live electrical wires after consulting erroneous design drawings, but it was exacerbated by a number of extant problems with local generation and protection configurations.
The outage occurred while utility workers were installing new protective relay equipment for a transformer at a local receiving station, RS-E. The project was part of a systemwide effort to upgrade relays at 179 substations. Acting on a faulty drawing that had cable numbering errors, a utility worker cut a bundle of three wires that were supposed to be left intact, triggering a short circuit that led to the shutdown of other transmission and generation stations. Because the specific facilities are secondary to the sequence of incidents, the generation and transmission facility names have been made generic.
Sequence of Events Leading to the Voltage Collapse and Recovery
About 12:35 p.m. local time, Sept. 12, 2005, both 230-kV busses at RS-E relayed simultaneously due to a worker inadvertently activating breaker failure protection on both RS-E busses. This immediately de-energized 390 MW of load at RS-E and RS-S. All RS-E transmission circuits opened at RS-E. This resulted in 2,447 MW of load being supplied by 635 net MW of H Generation Station and S Generation Station and by four transmission circuits: T-O Lines 1 and 3, and V-C Lines 1 and 2. These circuits were loaded above their normal ratings. System voltage fell to around 90% of nominal, but sub-transmission dispatchers stabilized and restored the voltage by immediately energizing the available 34.5-kV capacitors.
All generation at H and S tripped off within 7 minutes of the disturbance. Voltage fell to 72% of nominal, but sub-transmission dispatchers again stabilized and restored it by manually shedding 395 MW of load and by energizing additional 34.5-kV capacitors.
Attempts to restore transmission at RS-E failed due to the large power angle, later estimated by system simulations to be around 40 degrees. Thirty minutes into the disturbance, T-O Line 3 relayed off line after sagging into a tree. The remaining three circuits became extremely overloaded, voltage fell to 77% of nominal, and sub-transmission dispatchers shed 195 MW of load in order to stabilize the voltage.
Five minutes later T-O Line 1 relayed off line due to relay misoperation. V-C Lines 1 and 2 then loaded so high that relay protection interpreted the heavy loading as a fault and tripped both circuits. The system blacked out due to loss of supply, de-energizing the remaining 1,584 MW of load not already manually shed. During the several cycles it took for the blackout to completely occur, underfrequency load-shedding (UFLS) relays automatically disconnected 1,040 MW of load.
T-O Line 3 was restored 25 seconds after the system blackout, picking up 544 MW of system load not already disconnected by UFLS action. After T-O Line 1 and V-C Lines 1 and 2 were restored, system load was low enough that the resulting small power angle at RS-E allowed RS-E transmission restoration to begin. Dispatchers fully restored all transmission and load within 51 minutes of the system blackout without further incident.
During the 84-minute event, eight generators and seven condensers either tripped off or became isolated, 23 bulk power transmission circuits either relayed or became open-ended, 39 bulk power transformers became either open-ended or de-energized, and 15 receiving stations blacked out. A major sewage treatment facility, a major international airport, four major oil refineries, and 897,992 customers experienced a service interruption.
Analyzing the Sequence of Events for Preventive Equipment Measures
The outage would have been avoided had the utility worker been trained to treat all existing wires as energized and cut them one at a time instead of in a bundle. Cutting three of the cables at the same time caused two breaker live wires to be connected to a grounded wire and opened the breakers to two busses at RS-E.
Thirty minutes into the disturbance, T-O Line 3 relayed after sagging into a tree. This demonstrates that even if a specific line is thermally rated to handle emergency loading, this capability is lost if appropriate tree trimming precludes the requisite sagging of the overloaded line.
Attempts to restore transmission at RS-E failed due to a large power angle. Load dispatchers were not initially aware of the large phase angle between two of the generators. Accurate knowledge of the phase angle at RS-E would have allowed them to more quickly analyze the situation and develop a remedial plan. This problem has subsequently been corrected by the installation of a phase angle monitoring device (a phasor measurement unit) at critical receiving and generating stations.
Fifteen transmission circuits terminate at RS-E. This is too many for reliable operations; the loss of RS-E will black out large portions of the service area. This situation was mitigated by conducting comprehensive planning studies that evaluated and implemented a method of bypassing RS-E with transmission so that any future loss of the station will be less likely to cause blackouts.
The H Generation Station was tripped out of service seven minutes into the disturbance. If this generation had remained online, it might have supplied enough reactive support to avoid the voltage collapse incident. What is troubling is that the unit that caused the H Generation Station to trip off, H Unit 10, was operating within its capability curve. More on the capability on H Unit 10 later.
Investigations into the H Unit 10 Trip
The generating units at H must remain online during and after disturbances to relieve loading on overloaded circuits and to provide dynamic voltage support. At the time it tripped off, 7 minutes into the disturbance, H Unit 10 was generating 156 MW and 104 MVAR, which is within its reactive capability.
A review of the records showed that the relay for the feeder to the excitation system tripped, and when this relay tripped it left the remainder of H’s units without a source of steam, thus tripping the entire plant.
Before getting into the specifics of this particular generating unit, some background on how the excitation system is supposed to function. A voltage sensor measures the output voltage of the generator. If the voltage is too low, the voltage regulator and power system stabilizer work to increase the excitation current, which is fed onto the field windings to increase the output voltage. The increase in output voltage provides extra reactive power to provide dynamic voltage support.
In H Unit 10, the exciter is a 380-V static exciter unit and is fed from a transformer that lowers the voltage from 4,160 V. It turns out that the 380-V feeder line relay tripped because of too much current on the feeder. To provide 104 MVAR output on the unit, the excitation feeder required 157 A. The feeder relay was set to accommodate the size of the feeder cables, which were rated at 100 A. As a result, the feeder cables were undersized. Had the feeder cables been sized correctly, say at 200 A, then the excitation feeder relay would not have tripped, and H Unit 10, along with the other H units, would have been able to supply reactive power to help support the collapsing system voltage. This problem has since been resolved by increasing the ampacity of the exciter feeder on this unit.
The Importance of Acceptance Testing
Besides the mitigation measures outlined above to avoid such cascading failures in the future, an important lesson on acceptance testing was learned.
The undersizing of the excitation feeders was a design flaw that should have been corrected during the acceptance testing of the generation equipment. Acceptance testing is a process in which all the generation systems are tested across their design range to ensure operation not only during normal operation but also during abnormal operations — like the disturbance causing this voltage collapse.
It is evident that the testing protocol followed during the acceptance testing of H Unit 10 did not include the full range of testing on the excitation system. Had this been done, the excitation system would have failed and a proper redesign would have been implemented.
— Contributed by Robert Castro (firstname.lastname@example.org) who teaches graduate-level power classes at the University of Southern California and negotiates wind generation contracts for a local utility.