Has Your ICS Been Breached? Are You Sure? How Do You Know?

Many known security breaches have been in enterprise networks, but why haven’t we heard about many industrial control system (ICS) breaches? Maybe because systems have been breached—but their owners are unaware because they’re lacking security instrumentation and personnel. 

Since 2010, the year Stuxnet was discovered, there has been an increase in industrial control system (ICS) vulnerability research and reported vulnerabilities, exploits, and ICS-specific malware (Figure 1). (Vulnerabilities are generally defined as flaws in a system or procedures that leave a system open to attack, exploits are programs that take advantage of a system’s vulnerability, and ICS-specific malware is code written specifically to compromise ICS systems.) Couple this with the fact that more and more control systems are being connected to the Internet (according to the Shodan website), and we have the ingredients for a bad recipe. All that’s needed is an attacker with a motive.

1. Publicly known ICS-specific vulnerabilities, exploits, and malware. This chart represents annual global data as of Dec. 31, 2014, as aggregated by the author. Sources: ics-cert.us-cert.gov, osvdb.org, rapid7.com, gleg.net, scadavulns.com

According to the NCCIC/ICS-CERT Year in Review 2014 (from The National Cybersecurity and Communications Integration Center and the U.S. Department of Homeland Security’s Industrial Control Systems Cyber Emergency Response Team), attackers do seem to be motivated: Last year, 245 cyber incidents were reported for ICS owners, 79 of which were in the energy sector. Attackers certainly don’t care if you are already North American Electric Reliability Corp./Critical Infrastructure Protection, Version 5 (NERC/CIP v5) compliant, they just want what they are after. They are professional, organized, and well-funded. If you kick them out, they will just come back.

If ICS are so vulnerable, why haven’t we seen more of these attacks? Perhaps it is because we aren’t looking!

There may be someone from IT looking at the enterprise side of the network, but most businesses don’t regularly look at network traffic patterns and logs on the control system network (if they are even available). If a company has a security operations center (SOC), most likely the ICS is not tied into it, which means that company is defending only part of its operations.

So what can ICS owners do in the face of more ICS vulnerabilities, malware, government regulations, ICS devices that are insecure, attacks on every sector (including energy), and the fact that breaches are inevitable? This article explains how network security monitoring (NSM) of industrial control systems will help you detect and respond to attacks, rather than learning about them only after law enforcement knocks on your door.

Network Security Monitoring

Network security monitoring is “the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. NSM is a way to find intruders on your network and do something about them before they damage your enterprise,” as described in The Practice of Network Security Monitoring, by Richard Bejtlich (chief security strategist of my employer, FireEye).

In 1986, Cliff Stoll, an astronomer-turned-systems administrator at Lawrence Berkeley National Laboratory, used monitoring techniques to hunt for a hacker who was stealing information from the national labs and selling it to the KGB. If you have not read his book, The Cuckoo’s Egg: Tracking a Spy Through the Maze of Computer Espionage, which is the first documented case of someone catching a hacker, I highly recommend it (see sidebar). Of the 80 systems that the hacker breached, only two system owners noticed! Many of the tools and techniques Stoll used—most importantly, a questioning attitude—are still used in defending corporate networks today. Stoll and a few others were a major influence that led to the invention of NSM by Todd Heberlein and his team at Lawrence Livermore National Laboratory in 1988. This article shows that these same tools and techniques can be applied to defending ICS.

Network Security Monitoring (NSM) Resources

■ Cliff Stoll, The Cuckoo’s Egg: Tracking a Spy Through the Maze of Computer Espionage (1989)■ One-hour PBS NOVA Special (1990) based on Cliff Stoll’s experience: https://www.youtube.com/watch?v=EcKxaq1FTac

Todd Heberlein et al., A Network Security Monitor (1990): seclab.cs.ucdavis.edu/papers/pdfs/th-gd-90.pdf

■ Richard Bejtlich, The Practice of Network Security Monitoring (2013): nostarch.com/nsm

■ Chris Sanders & Jason Smith, Applied Network Security Monitoring (2013):

■ The NSM Wiki: nsmwiki.org

■ The open source Linux distribution for NSM, Security Onion: securityonion.net

NSM needs two components to be successful: at least one person with a questioning attitude and the right tools for that person to collect and analyze the data while keeping a lookout for attacks. Collecting logs and data, detecting anomalies, and performing analysis on the anomalies (hunting for evil) is the NSM cycle. This is the daily job of the security analyst—to be on constant watch for strange things on a network that may be a breach. The goal is to make the time between the breach occurring and detection of the breach as short as possible.

For instance, when I was a new employee at Mandiant, I was issued a new laptop and was allowed to download programs I needed to do my job. I downloaded a free Modbus tool; two minutes later I received an email from our SOC analyst asking what the file I downloaded was and if I intended to install it. I was blown away at how fast they responded! If the downloaded tool had been malware, they would have started their response very quickly. Often, organizations don’t know they have been breached for many months or, in some cases, years.

What is important to know about NSM is that it uses passive collection, not active scanning of the ICS network, which we know can cause serious consequences. Active scanning uses software to probe the network to find devices, open ports, and even vulnerabilities. If not done properly, active scanning can cause control system devices to malfunction. Passive collection is monitoring traffic on Ethernet and serial networks, which will not affect control system devices.

To passively monitor a network for NSM, you need at least one network sensor and a way to get a copy of the network traffic to the sensor. Many organizations have network sensors already deployed on normal IT networks, but these sensors may not always be best suited for a control system environment.

If you are considering installing NSM sensors in your ICS, I strongly suggest that you begin in a lab environment and with all of the important ICS stakeholders, such as security operations, plant operations, engineering, and even the ICS vendors. NSM sensors for ICS environments must consider traffic volume and speed, the physical environment (whether or not the sensor needs to be industrially hardened), and what protocols the sensor needs to interpret.

The software platform of the sensor is important to meet these requirements. There are several new commercial ICS-specific platforms available on the market that can collect data and present it to a dashboard for the analyst, perform deep packet inspection of ICS protocols, and even draw a network diagram. There are also free tools available for NSM sensors, the most popular being Security Onion. Security Onion contains several tools such as Wireshark, Bro IDS, and Snort IDS that understand DNP3, Modbus, and other ICS protocols.

Placement of the sensors is important as well, so that the perimeter and important internal systems in the ICS network are visible to an analyst. (See sidebar, “So You Want to Be an NSM Analyst.”) The sensors should be connected to either an existing switch with port mirroring capability, or a dedicated network tap should be installed. It is even possible to passively monitor serial-based traffic. Lastly, don’t forget to secure the sensors and the data they collect.

So You Want to Be an NSM AnalystWhat type of person makes an ideal industrial control system (ICS) analyst for the purposes of network security monitoring (NSM)? As this article underscores, a questioning, vigilant attitude is paramount.Additionally, these general technical skills and abilities can be very helpful:


■ Understanding what the control system does, and its components.

■ Knowledge of the ICS protocols that are used.

■ Awareness of existing vulnerabilities of their control system components.

■ Ability to use NSM software and write custom intrusion detection system (IDS) rules.

■ Being able to communicate effectively.

■ Taking charge if an incident occurs.

The NSM sensor can collect several types of data. The richest type is the full packet capture (pcap), which is like a phone recording. It provides a record of exactly what happened. Because many ICS networks are low bandwidth compared to IT systems, capturing full pcaps is an advantage. The next type of data is called session data, which is like records on a phone bill showing who called whom when, and for how long. Session data has the advantage of taking less storage space than full pcaps. Be sure to size the sensor’s hard drive appropriately for how much data to store for how long. Other types of data include extracted files, intrusion detection system (IDS) alerts, and collected log files such as syslog and Windows event logs.

Advantages for Network Security Monitoring in ICS

In his book, Bejtlich talks about several disadvantages of implementing NSM in IT networks, such as high bandwidth, encryption, and mobile devices. It turns out that these disadvantages don’t exist in ICS networks, because ICS usually have lower bandwidth, the devices are static, and the traffic is unencrypted.

Plant owners are faced with aging control system components that have little or no security and cannot be readily patched or upgraded. NSM is the best way to monitor and protect legacy control system components. When the devices are eventually upgraded, NSM can still be used to monitor the newer, more secure devices, because we know that an unknown vulnerability may still be lurking.

Expecting that your ICS will be breached is a more powerful position than trying to prevent all breaches.

Figure 2 is an example of a typical plant network architecture. At the top is the corporate IT infrastructure consisting of business systems, personal computers, and laptops. Next is the plant demilitarized zone (DMZ) in which firewalls protect the plant and yet allow corporate access to certain types of plant data such as the plant historian. Deeper into the network are the plant distributed control systems (DCS) for turbine control, balance of plant, water plant, continuous emissions monitoring, and historian. At the lowest level are the controllers themselves, such as the programmable logic controllers (PLCs) for each of the turbines.

2. Example of a plant network architecture diagram. Courtesy: FireEye

An attacker may try to gain access to the plant DCS from the corporate network through a spearphishing email, compromising an engineering laptop that sometimes connects to DCS components, an infected USB drive, or via a third-party connection like a vendor. Once the attacker has gained access to the DCS, if you aren’t monitoring the network, then you may not know about the attack until it’s too late.

A Power Plant Example

Recently, I had the opportunity to work with a power utility on a security assessment. Their network team provided us with hour-long pcaps taken from the corporate switch of two of their power plants. By analyzing the pcaps, we were able to see what was happening on the plant corporate network. We were able to see connectivity from plant corporate PCs to the historians, other corporate machines, and the Internet. We used different software to extract data from the pcaps, including: IP addresses, MAC addresses, TCP and UDP ports, DNS requests, and even several different file types.

In the pcap from Plant 1, we saw traffic from a human-machine interface (HMI) inside the plant that wasn’t supposed to be connected to the corporate network. A walkdown verified that the HMI was indeed dual-homed. Also in the Plant 1 pcap, we saw several corporate PCs sending similar files to one IP address. This traffic appeared to match a known commodity malware, which was verified by the utility’s IT security and removed. In Plant 1 and Plant 2 pcaps, we saw the plant remote terminal units (RTUs) communicating with the utility’s energy management system using DNP3 protocol over the corporate network. The RTUs were not using DNP3 with secure authentication, nor were they using encryption. If an attacker who breached the corporate network found the RTU traffic, they could see the data and could also modify the values to and from the plant.

All of this information was gathered just from looking at a 1-hour pcap from the network! If you looked at a pcap on your network, would you find anything unexpected (Figure 3)?

3. Under the hat. Chris Sistrunk, shown here, and other security experts recommend using network security monitoring because it enables power plants to discover “hidden” digital traffic that can potentially harm industrial control systems. Courtesy: Chris Sistrunk

Gaining Visibility in the Plant Network

So you’ve selected an NSM sensor platform and tested it in a lab environment of your DCS components, and you have a dedicated person tasked with monitoring the network. What data will be collected and from where? Do you start monitoring NERC/CIP plants, or all of them? Let’s take our example network diagram and add NSM to it (Figure 4). You’ll have to choose whether the NSM data is sent to the existing corporate SOC or to a new SOC specifically for monitoring the control system (even if it’s as small as one analyst).

4.Example plant network architecture diagram instrumented with network security monitoring. Courtesy: FireEye

4. Example plant network architecture diagram instrumented with network security monitoring. Courtesy: FireEye

In the example DMZ, there is a web server and a historian, which is most likely running on a modern IT server that has the ability to run antivirus or whitelisting and even security agent endpoint protection software. All of these have been used in IT systems, so it will be important to collect all of those logs. Next we have the network sensors that capture and analyze traffic from each network segment. In the plant DCS segment of the network, the servers may only have Windows logs or syslog available. Because those operating system logs already exist, it is an advantage to monitor them. It may be possible to install endpoint protection on some devices, but that usually requires vendor approval. Lastly, if any logs exist on PLCs and other controllers, collect those as well.

As in the plant pcap example: What is normal traffic and what would be abnormal? Control systems are static and use a limited list of protocols, so it should be straightforward to determine the baseline of the ICS network traffic and behavior. This baseline may have already been created at the factory acceptance test or site acceptance test. Here are some of the things a security analyst will look for:


■ Exceptions from baseline (such as, A talks to B but never to C, or the DCS normally uses 30% of the available network bandwidth).

■ “Top Talkers” (A and B constantly talk to C; D talks to A once a day).

■ Unexpected connectivity (to Internet or the business network) that raises the question, “Why is the HMI talking to a news website?”

■ Known malicious IPs and domains (antivirus IP blacklists, ICS-CERT advisories).

■ Logins using default or shared accounts. (Monitor the PLC login that can’t be changed; attackers frequently use default and stolen credentials.)

■ Error messages that could correlate with vulnerabilities (such as 100% CPU usage).

■ Unusual system and firewall log entries.

■ Endpoint protection or other security system alerts.

■ Unexpected file and firmware updates.

■ Antivirus alerts.


You can implement network security monitoring in your networks today—without affecting your plant operations. There are free tools available to help you start looking at your ICS and hunting for anomalies.

People are the most important part of your defense: You can collect gigabytes of data and thousands of IDS alerts, but they are useless without someone interpreting them. Remember, adversaries are a “Who,” not a “What,” and they don’t care if you meet NERC/CIP or not. Don’t wait for law enforcement to knock on your door before you start monitoring your plant.

Chris Sistrunk (chris.sistrunk@ mandiant.com) is a senior ICS security consultant at Mandiant, a FireEye company. He discovered and helped fix many DNP3 protocol implementation vulnerabilities as part of Project Robus with Adam Crain. He is a registered PE in Louisiana; IEEE senior member; and member of Mississippi Infragard Board, Louisiana Tech EE Industrial Advisory Board, and the DNP Technical Committee. Previously, he was SCADA subject matter expert at Entergy. Chris also founded and organizes BSides—Jackson, Mississippi’s only cybersecurity conference.