Grid Security Gets Physical

Cybersecurity has grabbed the lion’s share of grid security attention, but last year’s attack on a substation in California served as a reminder that physical attacks are still a significant threat.

The attack began at 12:58 a.m. on April 16, 2013. Between then and 1:07 a.m., attackers cut telephone and telecommunications cables to Pacific Gas & Electric’s (PG&E’s) Metcalf substation near San Jose, Calif., a 500-kV facility on the strategic Path 15 in the California bulk power transmission system.

At 1:31 a.m., they opened fire on the substation, using high-powered assault-style rifles. The attackers fired from multiple directions. A surveillance video (Figure 1; viewable at https://www.youtube.com/watch?v=RQzAbKdLfW8) shows multiple muzzle flashes around 270 degrees during the attack.

1. Midnight blast. This screen capture from the Pacific Gas & Electric (PG&E) surveillance video of the attack on the Metcalf substation shows bullets striking the fence and equipment inside. Courtesy: PG&E and Santa Clara County Sheriff’s Office

The assault lasted some 19 minutes as the gunners knocked out 17 large transformers that supply power over the well-known constrained path to Silicon Valley. PG&E notified police by a 911 call at 1:41 a.m. By 1:45 a.m., transformers were cascading out of service.

The gunners apparently left the site by 1:50 a.m., just a minute before police arrived at the scene. But the officers were unable to immediately enter the substation, as the gates in the hurricane fence surrounding the facility were still locked.

PG&E’s grid operators were able to reroute power to avoid blackouts. But the damage was major, requiring nearly a month to bring the substation back into service. The cost was significant enough that both PG&E and AT&T, whose telecommunications lines were severed in the attack, offered $250,000 rewards leading the arrest and conviction of the attackers.

PG&E also announced that it would build “opaque” fences around important transmission substations, including at Metcalf, and also provide round-the-clock security shifts. According to the Wall Street Journal, the San Francisco–based utility eliminated 24-hour monitoring of substations in 2009.

What Is the Risk?

To date, more than a year-and-a-half after the Metcalf assault, no arrests have followed and no suspects have been publicly identified.

Yet the attack on the Metcalf substation has fundamentally changed the discussion about the security and resilience of the U.S. electrical grid, which for several years has been dominated by discussions about cyber security—how high-tech hackers could tap into power system controllers and computer software and cause enormous mischief. Now it’s gotten physical.

Jon Wellinghoff, the assertive Nevadan who chaired the Federal Energy Regulatory Commission (FERC) at the time of the Metcalf attack, had long been concerned about the focus on cybersecurity of the bulk power system (generating plants, switchgear, and transmission and distribution) at the expense of the old-fashioned guns and bombs approach to destruction and disruption. In November 2012, six months before the Metcalf attack, Wellinghoff said, “A coordinated physical attack is a very, very unsettling thing to me.” His remarks came shortly after a 2007 report by the National Academy of Sciences was declassified. That report (available online at http://bit.ly/1kV1y8J) found that a terrorist attack on the bulk power system could cost “hundreds of billions of dollars and result in thousands of deaths.”

Wellinghoff commissioned FERC’s grid security maven, Joe McClelland, to study the vulnerabilities of the grid to a conventional attack. That study soon became a center of controversy.

Shortly after the Metcalf attack, authorities alerted Wellinghoff. That apparently galvanized his thinking about the physical—as opposed to cyber—threats to the grid. But the federal officials had a disagreement. While Wellinghoff believed the Metcalf attack may have been a precursor or a training exercise for a terrorist attack, the FBI, which was also involved in the Metcalf investigation, disagreed, and classified the details of the attack, meaning they could not be discussed in public.

Wellinghoff then began sending out fairly clear signals about his views on the physical vulnerabilities of the bulk power system, without dwelling on the Metcalf assault. On April 24, 2013, just days after the Metcalf attack, he told a Bloomberg New Energy Finance public meeting in New York, as reported by SNL Energy, that “studies had shown that the nation’s whole electrical system would turn off if four substations in the Eastern Interconnection, three substations in the Western Interconnection and two substations in Texas were destroyed.” This was a reference to McClelland’s work that Wellinghoff had commissioned before the Metcalf attack.

The Wellinghoff warning drew no interest, in part because he was constrained by the FBI from mentioning the obvious connection to the Metcalf event. The FERC chair also began briefing industry officials on the findings of the FERC analysis of physical threats. None of that information was classified, according to Wellinghoff and FERC documents. Wellinghoff left FERC at the end of November 2013.

Belated Publicity

The first public mention of the Metcalf attack came at a December 2013 hearing of the House Energy and Commerce Committee. The committee was grilling FERC on its current performance, a fairly routine occurrence. Upsetting the conventional congressional snooze-fest, California Democrat Henry Waxman, who had been told of the Metcalf attack and was briefed by the FBI, revealed the incident during his questioning. He described it as “sophisticated” and employing “military-style weapons.”

Both Waxman and Cheryl LaFleur, acting FERC chairman, discussed the attack in generalities at the hearing but provided few details, which were still classified. Waxman said he had discussed the attack with the FBI, and the law-enforcement agency agreed to brief the House Energy and Commerce Committee. LaFleur also said she would permit FERC staff to discuss the attack with the committee staff, although she refused to reveal details because of the possibility of “copycat” attacks. LaFleur acknowledged that the April attack was the most sophisticated attempt to disrupt the electric grid that she had ever encountered.

This time, the Metcalf event got some public traction.

In early February, veteran electricity industry reporter Rebecca Smith of the Wall Street Journal outlined what had happened. A month later, she revealed in considerable detail the internal FERC report on the key vulnerabilities of the high-voltage transmission grid. She quoted Wellinghoff, now a private citizen, that it was “the most significant incident of domestic terrorism involving the grid that has ever occurred” in the U.S. Smith clearly had a copy of the FERC report, and the newspaper had been careful not to identify the specific interconnections that the agency’s analysis said could bring down the entire U.S. bulk power delivery system.

FERC’s regular monthly public meeting took place just days after Smith’s Wall Street Journal article. Two commissioners—Philip Moeller and John Norris—pushed back against the notion that physical grid security was a big problem and attacked the newspaper for revealing the FERC analysis. Moeller said the U.S. has the “world’s most advanced and robust electric transmission system that can respond instantly to planned and unplanned outages and even attacks. However, highlighting any real or perceived vulnerabilities and sharing specific security information or responsive actions may inadvertently promote the prospect of additional copycat attacks.” Norris said that “many people have jumped on this reaction train,” and that he feared a focus on the physical threats to the power grid would divert attention and syphon funding from smart grid technologies that he favors. (Still, such attacks have also occurred outside the U.S.; see sidebar.)

Pressure Mounts

Nonetheless, the pressure built on FERC to take some action on the prospects that old-fashioned bullets, bombs, and wire cutters could bring down the grid, not just hackers with laptops in China or Los Angeles. Senate Majority Leader Harry Reid (D-Nev.), Wellinghoff’s political godfather, wrote to FERC and the North American Electric Reliability Corp. (NERC), urging action to protect the grid, as did Sens. Mary Landrieu (D-La.), chairman of the Senate Energy and Natural Resources Committee, and Lisa Murkowski (R-Alaska), the ranking minority member.

In early March 2014, FERC ordered NERC, FERC’s private-sector reliability police, to prepare standards within 90 days on how to protect the grid from physical attacks. NERC has already been charged for many years with developing cybersecurity standards. LaFleur, the acting FERC chair, said, “Today’s order enhances the grid’s resilience by requiring physical security for the facilities most critical to the reliable operation of the bulk power system. It will complement the ongoing efforts of FERC and facility owners and operators to ensure the physical security of the grid.”

The order gave NERC and the electricity industry considerable flexibility in developing the standards. FERC told the grid operators to implement a three-step approach to physical protection.

First was an assessment to, in FERC’s words, “identify facilities that, if rendered inoperable or damaged, could have a critical impact on the operation of the interconnected grid through instability, uncontrolled separation, or cascading failures of the bulk power system.”

Second, grid operators must “evaluate potential threats and vulnerabilities to those facilities.”

Third, they must “develop and implement a security plan to address potential threats and vulnerabilities.”

In late May, NERC delivered a mammoth, 800-plus-page plan to FERC, providing the details of how the industry will respond to physical security threats. According to a NERC press release, NERC submitted the proposed response to FERC to a ballot among its stakeholders. It received “86 percent approval.”

But the other 14% had some words about it. E&E EnergyWire reported that the “sharpest critique” of the NERC plan came from the giant federally owned Bonneville Power Administration (BPA). BPA, which operates a multi-thousand-mile, multi-state high-voltage grid in the Pacific Northwest, commented, “It is virtually impossible to fully protect all critical [bulk electrical facilities] from attack by a determined foe.” BPA has absorbed attacks on its transmission facilities going back decades, though none that seriously damaged the flow of power on its massive system.

Portland, Ore.–based BPA commented to NERC that the opportunity to attack the transmission system “is already available… and implementable regardless of what physical hardening is implemented.” BPA said, “The biggest general question to answer is what will be considered adequate protection. Will we need a 24-hour on-site security force because the location is too remote to augment detection technology with fast response?”

The BPA response added, “Will we need security walls constructed to be as impervious as those of a maximum security prison? The list of potential risk mitigation barriers is endless, as is the cost of building and maintaining elaborate barriers for facilities that cover acres of ground.”

Much to Consider

In comments on FERC’s physical security order, the Battelle Memorial Institute, a private, nonprofit research institution in Columbus, Ohio, offered what it billed as a commonsense and integrated approach to both physical security and cybersecurity. Physical security assessments of the bulk power grid should include “regional studies that cross utility and system operator boundaries,” said a Battelle white paper. “Just as transmission expansions are now conducted regionally, so should physical security risk assessments.”

The Battelle report—“Recommendations for Implementing Comprehensive Bulk-Power System Security Standards”—said that risk assessments for physical security of the grid should look beyond the local and instead to regional analysis in order “to avoid creating seams issues.” The analysis also called for “sharing risk assessments between operators, consistent with” the NERC requirements. “For instance,” said Battelle, “a particular site may NOT be critical for maintaining service within the service territory of the utility that owns/operates it, but may significantly increase the potential for problems in external areas.”

The Battelle report pointed to the enormous 2003 blackout in the Northeast and Middle Atlantic states, where “loop flows through New York, Canada, and Michigan were triggered by lines tripping in Ohio.” That event created outages in the New York ISO, the Midcontinent ISO, and the PJM Interconnection.

The Battelle report also said that physical security “should incorporate all-hazards threat assessments to ensure that efforts to improve security and resiliency of the bulk power system do not unduly focus on a single threat vector.” It went on, “Environmental events, equipment failures, operational failures, and cyber attacks should all be considered alongside physical attacks for both their likelihood of occurrence, as well as quantification of the impacts each would create.”

Security plans “should include measures for dealing with events before, during, and after their occurrence, and not simply focus on preventing attacks. Since it is virtually impossible to prevent damage from all possible attacks or other all-hazard events, security plans should prioritize actions that will have the most impact on overall system resiliency across all threat scenarios.”

In addition to hardening possible targets of attack, the report said that it is “important that the standards allow for creating decoys and that critical assets are not obviously identified by a singular focus on applying measures only to them. For example, if sudden and/or significant improvements (fencing, barriers, lighting, cameras, etc.) are made to only a few assets, it would make them obvious targets for an attack.”

Tailoring security plans “to site-specific conditions” is also necessary. “Specific security measures, such as upgraded fences, barriers, and obscurants may be very effective in some locations, and nearly useless in others,” said the white paper. “The highest leverage investments in an urban environment will be very different than for remote locations.” The analysis adds that “differences in law enforcement and/or utility crew response times will determine what measures are most appropriate to deter attacks, but also to mitigate and recover from damages caused by attacks. For the most remote locations, remote detection, alarms, and redundant communications may be more important than the additional hardening measures which can be defeated with the additional time available for potential attackers.”

FERC is likely to act on the NERC filing and comments to its March 2014 order sometime this year, although the timing is unclear. ■

— Kennedy Maize is a POWER contributing editor.