The destruction of the turbines and auxiliary equipment at Russia’s Sayano-Shushenskaya Hydro Power Plant in August 2009 claimed the lives of 75 workers and wrecked an indispensable source of electricity that will take years to fully restore. The disaster, as this report explains, was predictable and preventable.
It was early Monday morning August 17, 2009, at the Sayano-Shushenskaya Hydro Power Plant (SSH), which is located on the Yenisei River, near Sayanogorsk in the Republic of Khakassia, in the southern region of the Russian Federation. By 8:00 a.m., technicians were operating multiple hydroelectric turbines at the plant, including Unit 2, and plant maintenance work on other equipment was continuing. Other members of the plant staff were arriving or having a final cup of coffee with colleagues before beginning the new work week. Overall, the plant was performing reasonably well in spite of the many nagging maintenance problems. In fact, just over a month earlier (July 2), RusHydro Public Corp. (RusHydro), the owner and operator of SSH, announced that the plant had set an all-time 24-hour electricity production record. Thirteen minutes later the plant lay in rubble and 75 workers had lost their lives.
RusHydro, with its 55 hydro plants, is the leading power company in Russia by installed capacity (25.4 GW) and the second-largest hydroelectric power company in the world, after Canada’s Hydro-Québec. RusHydro was established in December 2004 as part of Russia’s privatization movement, although most of its shares (about 60%) are owned by the Russian government. The energy generated by SSH, about 23.5 TWh annually, was about one-quarter of that produced by RusHydro’s resources for the Unified Energy Systems of Russia and Siberia. About 70% of the energy generated by RusHydro goes to United Company Rusal’s aluminum smelters, which produce about 12% of the world’s aluminum.
With its 10 turbines, SSH was RusHydro’s largest hydroelectric power station and the sixth-largest in the world, based on installed capacity. In fact, SSH was once included in Guinness World Records as a plant with the most durable dam in the world. At 6,400 MW, SSH was also the largest of the hydroelectric plants positioned along the Yenisei River basin. The Yenisei is a river that forms in Mongolia and flows through Siberia to the north before emptying into the Arctic Ocean. Other plants on the river include the 12 x 500-MW Krasnoyarsk (in seventh place) and the 3 x 107-MW Mayna hydro power plants (Figure 1).
|1. Global leader. The Russian Federation leans heavily on hydroelectric power plants located on the Yenisei River and its tributaries for power to run its industrial facilities. The 6,400-MW Sayano-Shushenskaya hydro plant (bottom left) was to be the largest in the world when construction began in 1961. The 10th and last 640-MW hydro turbine was placed into service in 1987. Before the accident, the plant was the sixth-largest hydroelectric plant in the world. Courtesy: Alexander Boyko|
At the start of construction in 1961, SSH was the largest hydro power plant in the world. The length of the completed dam is 1,066 meters (m), and it rises 242 m above the river floor. The width of the dam is 105.7 m at its base and 25 m on the ridge. The dam itself is of the arch-gravity type, a design where the center of the dam extends upstream and ends arch back in order to direct most of the hydrostatic force against the canyon walls (Figure 2). The maximum capacity of the spillway is 13,600 m3/sec (3.6 million gallons/sec). The project’s turbine gallery housed 10 turbine units, each rated at 640 MW but each capable of peaking up to 720 MW. Rated head for each of the hydro turbines was 192 m.
|2. Beautiful curve. The Sayano-Shushenskaya Hydro Power Plant is of the arch-gravity design and measures 1,066 m long. The turbine gallery is located at the base of the 242-m-high dam. Courtesy: Kirill Elesin, EKRA-Sibir Ltd.|
From 1978 to 1987, all 10 units were placed into operation in turn. Originally, Unit 1 and Unit 2 had a rated capacity of 155 to 400 MW and a rated head of 60 to 140 m, but newer units later replaced these. In actual operating practice, the generating capacity of SSH was limited to 4,400 MW because of a weak connection with the local electrical grid, which consists of four 500-kV transmission lines. SSH, in coordination with the Bratsk hydro power plant (18 x 250 MW), located on the Yenisei tributary from the Angara River, performs secondary control of reactive power and frequency for the Siberian power grid. The Mayna hydro plant, located 21 km downstream from SSH, performs tail water reregulating for the plant.
The Blame Game
The official report on the incident was released by the Russian Federal Service for Ecological, Technological, and Nuclear Supervision, or Rostekhnadzor (roughly the equivalent of the U.S. Occupational Safety and Health Administration but with added nuclear and environmental safety protection responsibilities) on October 3, 2009. The report blames six officials who were “conductive to the disaster,” including Anatoly Chubais, former CEO of Unified Energy Systems (UES). “Under order No. 690 dated Dec. 13, 2000,… UES chairman Anatoly Chubais approved the act of the central commission to launch the Sayano-Shushenskaya hydropower complex without a comprehensive evaluation of the data on Sayano-Shushenskaya’s operations,” says the report released by the Federal Inspection Service for the Environment and Technology. Chubais’ statement in response was posted on his website: “I’m responsible for everything that took place in the field during my time in office.” He went on to say that the plant had already been running for more than 20 years when he signed the papers and that he was under great pressure to make those decisions in 2000 because shutting down equipment while waiting for maintenance funding would have “meant a catastrophe for the economy of Siberia and millions of residents there.” No explanation was offered as to why this certification was finally signed more than 20 years after the plant began service.
From July 2008 until the accident, RusHydro was managed by Vasily Zubakin, who was included in the “List of persons bearing responsibility for accident disaster prevention on Sayano-Shushenskaya” along with 18 other RusHydro executives. (This list is a chapter of the official 140-page accident investigation report, although the report is no longer available on the Rostekhnadzor website.)
The industrial watchdog’s chief, Nikolai Kutin, described the service’s responsibility in the disaster probe: “For our part, we study the technical causes of the accident that were in the making for a long time, and that is why you will find a relatively large number of names, both from the plant management and RusHydro, as well as senior officials who made decisions affecting the stability and security of the plant’s operation.” In a televised statement, Kutin summarized what had happened the day of the accident. He said part of a turbine unit (for Unit 2) weighing 1,500 metric tons had flown 14 meters into the air after the screws (anchor bolts) holding it down had come loose, causing flooding and debris that killed dozens of workers in seconds (Figure 3).
|3. Turbine destruction. Before and after photos of the plant turbine gallery. Courtesy: Kirill Elesin, EKRA-Sibir Ltd.|
The accident of August 17 was not the first big accident at SSH. The power plant was still under construction in March 1979, with Unit 1 and Unit 2 already in operation, when a large spring flood overfilled the reservoir and rushed over the running generators and step-up transformers. In four months, the damaged equipment was repaired and the units were restarted. Another spring flood in 1985 destroyed 80% of the concrete spillway, an event repeated in 1988. A 1998 report by the Russian Emergency Situations Ministry found that the dam walls may not be capable of withstanding the cyclical pressures that result from annual spring floods. No significant flood control structures have yet to be placed upstream to blunt the force of those floods.
No further civil action has been taken against any state official or executive of RusHydro since the official report was released. The report was subsequently forwarded to criminal investigation authorities, and the investigation has been forwarded to the Russian Prosecutors’ Investigating Committee.
Follow the Timeline
The sequence of events that led up to the incident began early in 2009 when three months of repairs to the Unit 2 turbine were completed that included improved speed regulation controls. The turbine was resynchronized to the grid on March 16. Although the turbine’s vibration levels remained high, the unit continued to operate through July, when it was shut down for repairs.
The design life of the Unit 2 turbine was reported as 30 years by the manufacturer. On the date of the accident, the age of the Unit 2 turbine was 29 years and 10 months.
August 16, 2009. In accordance with usual generating schedule practices, all possible units at SSH were online and providing the grid with baseload and regulating (load-following or secondary control) power, including Unit 2, which was started up shortly before midnight. Bratsk hydro plant was operating in regulating mode (load-following or secondary control) under the control of the automatic load-frequency control system (ALFCS), which is overseen by the Siberian Unified Dispatching Control Center (UDCC).
August 16, 2009 at 8:31 p.m. A fire alarm tripped at Bratsk plant. Primary and secondary communication lines connecting Bratsk and the Siberian UDCC were lost due to the fire, and UDCC was unable to use Bratsk in regulating mode. The UDCC dispatcher ordered SSH to replace Bratsk in providing load frequency regulation. Staff at SSH switched the plant’s active and reactive power joint control systems (ARPJC) into regulating mode. From this moment, most of SSH’s units began to operate under the direct control of ALFCS located at the Siberian UDCC.
August 16, 2009 at 11:14 p.m. Dispatchers at the Siberian UDCC made the decision to start Unit 2 (which was kept in reserve) and immediately switch it to regulating mode. Unit 2 was selected to provide power regulating services as it was considered to be the most reliable unit at the plant given its recent servicing. Data on the Unit 2 start-up and operation through failure is summarized in Figure 4.
|4. Operating limits. This data from the operating log of Unit 2, shows that vibration increases when the turbine moves through or is operated in the “Not recommended” zone. The dotted line represents operation in the “Acceptable” zone, not a specific power load. Source: Sergey Popov|
August 17, 2009. Soon, Units 1, 2, 4, 5, 7, and 9 were all operating in regulating mode with Units 3, 8, and 10 generating baseload power. Unit 6 was undergoing scheduled maintenance and was not in operation. Instead of the normal 12 or so operators who staffed the turbine gallery on a typical day, on this day there were more than 100 workers present performing repair work.
All SSH units were equipped with RO-230/833-V-677 turbines, which have a very narrow recommended zone of power control, as shown in Figure 4. Operating with a power setting out of the recommended zone produces excessive vibration, as do transitions through the “not recommended” zone and back.
In addition, when Unit 2 was quickly pushed back into service, vibration levels in the turbine were extraordinarily high and quickly rising. Vibration data taken from the plant logs (Figure 5) show that the vibration levels had risen to dangerous levels.
|5. Increasing vibration. The Unit 2 operating log shows a steady increase in turbine vibration in the months preceding the accident. At the time of the accident, the turbine vibration was approximately four times acceptable levels. Source: Sergey Popov|
The authors were able to obtain and perform a detailed analysis of logs, written since April 21, 2009, until the time of the accident. Log data showed the Unit 2 turbine bearing vibration amplitude increased by a factor of four within this period. At the moment of the failure, vibration levels (double amplitude value) were as much as 840 µm, whereas the maximum acceptable vibration value is 160 µm. As is clearly seen in Figure 5, Unit 2 was operating for a long period of time with the vibration level above 600 µm. The vibration of other units during this period was no more than 200 µm.
The vibration control system for Unit 2, installed in 2009, was online and functioning but was not officially accepted for continuous operation, so the SSH operating staff were not allowed to rely on data it produced.
By this date, visible cracks showing the propagation of fatigue cracks in the attachment points of the Unit 2 cover were plainly visible at the locations where the mass of the turbine was secured to the plant’s structural foundation. Equipment fatigue, especially in the turbine anchor bolts, caused by the excessive vibration from turbine cycling and imbalance over an extended period of time, had finally reached the point of explosive failure.
August 17, 2009 at 8:13 a.m. The failure of the turbine anchoring system began the catastrophic sequence. First, the 1,860-ton turbine cover was blown off, leaving the Unit 2 turbine in its pit with no turbine mountings but with its wicket gate and head gate opened. In an instant, the 212-m water head pressure from the dam immediately ejected the turbine rotor from the pit. The rotor, continuing to spin as it flew across the gallery, destroyed everything and everyone in its path (Figure 6).
|6. Turbine ejection. When the turbine anchor bolts and support structures failed, water entered the Unit 2 pit, blowing the 1,860-ton turbine cover into the roof, and the turbine—while still spinning—across the gallery, destroying everything in its path. The turbine was operating at approximately 600 MW minutes before the accident. What remained of the turbine came to rest in the turbine pit. The turbine cover is not shown. Source: Reuters|
Within seconds of the turbine being completely ejected from the pit, a torrent of water began to fill the gallery because it was now connected to an open penstock. Water damage soon cut through the structural steel, and the turbine gallery roof collapsed. The water then started to flood pits of nearby units, causing further turbine failures. Electrical short circuits forced all the units to emergency shutdown. SSH power output tumbled from 4,100 MW to zero in an instant, while the in-house power supply system also shut down.
A cell phone camera video captured the short circuits and explosions in the area of Units 7 and 9 from outside the plant; that was followed by a deluge of water coming from the Unit 2 penstock breach that had flooded the turbine gallery and below. (You can see the video on YouTube at http:// tinyurl.com/3xd4rrf.)
These are the exact words (recorded by one of the authors, and translated) of a witness, Oleg Myakishev, a member of the Sayano-Shushenskaya staff: “First, I heard some kind of surge noise, and then I saw the unit cover swelling. The next moment, I saw a rotor breaking the cover and rising. It was spinning. I could not believe my eyes. Stones and armature began to scatter, and we had to turn away to avoid them. The rotor rose about three meters, its cover shot into the ceiling and destroyed it. I remembered that the water level was probably rising at a flow speed of about 380 m3/s [100,000 gallons per second (gal/s)], and ran to the Unit 10. I thought I can hardly be in time to avoid water. I reached Unit 10, ran upstairs, turned back, and saw everything getting destroyed. The water level was rising and some people were trying to swim. I realized that it is necessary to close the head gates… manually, because no power supply was available.”
Water was washing away people from the turbine gallery into the river. Some of them were later caught and rescued. Some were not. The staff later learned that the turbine gallery was not equipped with proper emergency exits. There were also some miraculous rescues. One worker was washed by the torrent of water filling the turbine gallery up into the ceiling, where he managed to grab some ventilation equipment and hold on until being rescued. Others led employees to safety using cell phones as flashlights.
Even as the turbine gallery filled with water, Unit 7 and 9 turbines continued to operate when their safety shutdown systems failed to trip. Eventually, the turbines were submerged, causing extensive damage to equipment and structures.
The only wicket gate that closed automatically was the Unit 5 gate—the remaining eight units’ wicket and head gates could not be closed remotely. Finally, the SSH chief engineer ordered the head gates closed manually. At 9:30 a.m., the heroic efforts of five men working in the dark with flashlights resulted in the manual closing of the remaining eight head gates (Unit 6 was closed before the accident). Complicating this work was the fact that there were no keys available for the room where manual head gate closing controls were located, so the metal doors had to be smashed.
In sum, the accident completely destroyed Units 2, 7, and 9. Units 5 and 6 were damaged but were repairable. The remaining five units were seriously damaged.
Call for Help
Within the first hours after the accident, the Ministry of Russian Federation for Civil Defense directed rescuers from across the country to SSH with the primary mission of finding and rescuing survivors as quickly as possible. Within 24 hours, more than 1,500 rescue workers arrived and began working. Fourteen survivors were rescued in this operation. Unfortunately, the final death toll had risen to 75 by the time rescue operations were suspended on August 29. Most of the fatalities were to workers who were working either on the turbine gallery or in the flooded rooms below (Figure 7).
|7. Rescue and rebuild. More than 1,500 rescue workers arrived at the plant within 24 hours and were able to locate 14 survivors. The death toll was 75 when rescue operations were halted on August 29. This photo, taken on August 18, shows the overall extent of the damage to the facility. Also, note the spillway in the background, where water must bypass the dam while the hydro plant is out of service. Currently, three of the 10 turbines are back in service, and a fourth (Unit 3) is expected to be back in service by the end of 2010. RusHydro says that the entire plant will return to service by 2014, although that estimate is ambitious. Source: Reuters|
The equipment destruction also caused damage to the surrounding environment from releases of industrial oils. More than 100 metric tons of oil were released into the river, which killed thousands of fish in downstream trout farms. Rescue teams used special chemicals spread by helicopters to congeal oil that was later manually removed from the water. By their quick action, rescue teams managed to minimize the environmental damage caused by these oil spills.
Investigations into the Cause of Failure
A number of forensic investigations where carried out following the incident. Turbine foundation bolts were subjected to thorough examination. The bolts found were broken, and fatigue damage covered an average of 65% of a bolt’s cross-sectional area. Some of the bolts had no traces of nut breaking, meaning that nuts had not even been installed on the bolts.
Unit 2 medium-level maintenance had been performed from January 14 to March 16, 2009, for the controls upgrades but also to eliminate the excessive vibration. Vibration remained after repairs were completed.
The logs and discussions with plant staff indicate that Unit 2 was again removed from service in early August due to continuing excessive vibration and was held in reserve. Moreover, Unit 2 was started and placed into baseload operation the evening of August 16 and then immediately placed into “regulating mode” in the early morning of August 17. That means it was subjected to more intense cycling service to follow loads, forcing the unit into multiple passes through “not recommended” zones of output power (see Figure 4). Those stresses were in addition to the long-term excessive rotating vibration already being experienced by the unit (see Figure 5).
The logs show that on August 17, 2009, at 8:13 a.m., ARPJC had set Unit 2 output power in the “not recommended” zone in order to meet grid demands, and the plant operations staff did not object. This was the final decision—following a long history of ignoring operating and maintenance deficiencies and obvious fatigue cracks in the turbine mounting structure—that caused the almost complete destruction of the SSH.
The accident required all the river water to pass through the spillway while repairs are under way. The spillway was not designed to conduct such large amounts of water in winter. In addition, last winter the spillway was covered with enormous amounts of ice, which was ineffectively removed by means of chain saws (Figure 8). This problem has been solved by running some of the restored units and finishing construction of in-rock bypass channels that became operational in June 2010.
|8. Winter freeze. Last winter, ice buildup on the spillway limited the amount of water that could pass downstream. Workers unsuccessfully attempted to use chain saws to break up the ice that formed on the spillway. Courtesy: Kirill Elesin, EKRA-Sibir Ltd.|
Spring flood surges in 2011 are again expected to add water to the Sayno basin in incredible amounts—up to 6,400 m3/s (1.7 million gps). Water discharge through the dam spillway is expected to be 3,850 m3/s (1 million gps). The long-term damage caused by excessive water flow through the spillway is unknown.
Plant reconstruction work continues at full speed, but the focus seems to be on repairs rather than on design improvements to ensure that a similar accident doesn’t occur. At first, the work focused on Units 5 and 6, which were not seriously damaged by the accident. The reassembly of those units was thoroughly scrutinized by the manufacturer, Power Machines Public Corp., and all the new engineering solutions were coordinated with Lenhydroproject Public Corp., which was the general designer of Sayano-Shushenskaya.
On December 30, 2009, operation of Unit 6 was restored, and it was successfully tested in open-circuit operating mode. On February 24, 2010, operation of Unit 6 was successfully demonstrated with Russian Prime Minister Vladimir Putin present. Unit 5 was completely restored and restarted on March 22, 2010, bringing the station’s capacity up to 1,280 MW.
Operation of Unit 4 was restored on August 2, adding another 640 MW to SSH’s capacity. At the commissioning ceremony, Deputy Prime Minister of the Russian Federation Igor Sechin promised that all the decisions concerning the restoration process will meet new technical requirements and standards set by governmental supervisory authorities for hydropower plant operation.
The restoration of Unit 3 is projected to be complete by the end of this year. If so, RusHydro has stated that this will “enable Sayano-Shushenskaya to operate during 2010–2011 fall-winter period without usage of operational spillway.” Perhaps not during the winter, but the expected spring floods will require SSH to bypass water via the spillway for many years to come. The RusHydro statement noted that “full restoration of the Sayano-Shushenskaya HPP is scheduled to be completed in 2014.” Most observers find that date optimistic and suggest that SSH won’t be fully restored for perhaps another four to six years.
Other positive plant improvements are under way. The wicket gate control systems have been redesigned. Now, the wicket gate will close automatically in case of power loss. Also, the unit head gates now can be directly controlled from the main control room. Backup power sources were also installed to provide power for head gate closing should the in-house power supply fail. In another significant procedural change, all the completed work will be accepted by primary equipment manufacturers, not by hydropower plant chiefs, as in past years.
What the Future Holds
It’s been over a year since the terrible accident at Russia’s Sayano-Shushenskaya Hydro Power Plant claimed 75 lives, caused significant destruction of primary hydropower plant equipment and buildings, and affected the region’s ecology. Even though the accident investigation report notes some of the causes and responsible persons, many questions about the design and safety of SSH remain unanswered.
There are many important technical and cultural improvements missing from the RusHydro list of plant design improvements. For example, the culture that places more value on the economics of power production than on human and plant equipment safety must change. Staff safety must be paramount and guide all operating decisions. The plant must also install modern real-time vibration monitoring systems on each unit and have unambiguous rules for their use that do not depend on operator interaction to shut down a malfunctioning unit. Finally, without a fully funded robust inspection (including random inspections by an independent organization), repair, and maintenance program, expect more part fatigue failures in the future, hopefully with less terrible results.
Finally, the human cost of the tragedy cannot be ignored. RusHydro has paid out about €4.8 million or about €25,000 to each bereaved family and will pay the parent’s salary each month to children until they reach “full age.” For families that lost both parents, the Russian government is paying an additional €25,000 to each family member. Sberbank, one of the largest banks in Russia, has cancelled the mortgages for families that lost a family member. Perhaps if this money had been invested in plant maintenance earlier, the Sayano-Shushenskaya would never have happened.
—Edited by Dr. Robert Peltier, PE. Alexander Boyko ([email protected]) is a relay protection engineer for EKRA-Sibir Ltd. located in Krasnoyarsk, Russian Federation and recently completed an assignment where he participated in commissioning the control system for the Naghlu Hydroplant Unit 4 in the Islamic Republic of Afghanistan. Sergey Popov ([email protected]), also a relay protection engineer for EKRA-Sibir Ltd., departed from the Sayano-Shushenskaya Hydro Power Plant two days before the accident and interviewed associates at the plant after the accident. Nemanja Krajisnik ([email protected]) is a power systems consultant for Siemens Transmission and Distribution Ltd., Manchester, UK.