Instrumentation & Controls

NERC CIPS Update: The Advantages of an Integrated Factory Acceptance Test

When adding, modifying, or upgrading a system, many critical infrastructures conduct a factory acceptance test (FAT). A FAT includes a customized testing procedure for systems and is completed before the final installation at the critical facility. Because it is difficult to predict the correct operation of the safety instrumented system or consequences due to failures in some parts of the system, a FAT provides a valuable check of these safety issues. Similarly, because cyber security can also impact the safety of critical systems if a system is compromised, it makes sense to integrate cyber security with the FAT.

An integrated factory acceptance test (IFAT) is a testing activity that brings together selected components of major control system vendors and plant personnel at a single place for validation and testing of a subset of the control system network and security application. An IFAT provides important benefits, including time savings, cost savings, improved ability to meet compliance requirements, and increased comfort level with integrated security solutions.

Integrated Controls Testing

Industrial control system (ICS) is a general term that encompasses several types of control systems used in industrial production, including the familiar supervisory control and data acquisition (SCADA) systems, distributed control systems, and other smaller control system configurations such as programmable logic controllers, which are often found in critical infrastructures such as electricity, water, and gas utility systems.

Over the past three decades, several hundreds of these protocols have been developed for both serial, local area network, and wide area network communication systems in industries including wastewater and electrical generation/distribution. Approximately 10 protocols currently dominate the industrial marketplace, including Modbus, Distributed Network Protocol (DNP3), EtherNet/IP, process fieldbus (Profibus), and Foundation fieldbus.

The choice of protocol is typically a function of the operating requirements, industry preference, vendor, and the system’s design history. For example, in a power utility’s SCADA system, a master located in a central facility could use the DNP3 protocol to query and control slave remote terminal units (RTUs) distributed in remote substations. SCADA systems and RTUs have published standards for communication between control centers, acceptance of alarms, issuance of controls, and polling of data objects such as Modbus. Over the past few years, these standards have moved toward a more open standard for SCADA systems and away from proprietary protocols, for example, TCP/IP Layer 3, and Layer 4. Other protocols, such as fieldbus and Profibus, are either analog or point-to-point, making them inherently difficult to secure.

SCADA applications are also very delay-sensitive, and newer protocols such as Frame Relay, Gigabit Ethernet, and Asynchronous Transfer Mode introduce data delay that can cause SCADA protocols to assume errors in the link. The traditional SCADA system was a closed serial network that contained only trusted devices with little or no connection to the outside world. As control networks evolved, the use of TCP/IP and Ethernet became commonplace, and interfacing to business systems became the norm. This connectivity increases the exposure to security risks and, as a result, increases vulnerabilities to process and SCADA networks.

Keeping Networks Secure

IT security has advanced more rapidly in the corporate market due to high security requirements typically mandated for federal and banking environments. For example, the wide use of computers in military and defense installations has long necessitated the application of security rules and regulations.

As with SCADA systems, the basic security premise was network isolation, which basically separates a system logically and physically. System downtime is tied to a percentage of uptime and a tolerance toward how long a system can maintain an outage.

Most IT systems are also subject to applications and programming issues that require frequent reboots. But over time, IT systems evolved from single-processor-type systems to faster processors that could run multiple applications at the same time. Private industry tends to drive software security requirements, which are then layered over existing hardware and software applications. Federal regulations and standards, such as NIST, ISO27001 and the Federal Information Security Management Act of 2002, also help drive security on IT-based systems. Now the idea of defense-in-depth is being applied in the design of IT systems and communication networks.

ICSs are very different from IT systems in that they are deterministic systems with very high reliability requirements. They follow the AIC model of Availability, Integrity, and Confidentiality. IT systems, by contrast, are CIA: Confidentiality, Integrity, and Availability. The key distinction is availability and the importance of availability to ICSs compared to IT systems.

Given the inherently isolated design of ICSs, security is believed to be automatic; however, that is far from the truth. We have learned that power line carrier and SCADA systems could be comprised. Simply placing a firewall as a logical perimeter defense mechanism and leaving all internal systems with very few or no security controls actually enables greater compromise, because a disgruntled employee could sabotage a system from the inside.

The other challenge for ICS security is the slow vendor adoption of security solutions within their applications or hardware solutions. An ICS has multiple interconnected systems, but each vendor utilizes its own unique solution, which is independent of other vendors’ solutions. This approach results in multiple vendors providing multiple solutions to solve the same problem of cyber security. There is no integration of cyber security solutions, which exacerbates the administration of cyber security within an ICS environment.

IFAT Benefits/Advantages

The purpose of an IFAT is to avoid costly redesign and troubleshooting during outage operations. Conducting an IFAT provides an opportunity to verify communications between systems and discover any potential issues prior to installation. It also enables redesign and troubleshooting to be completed in a more suitable environment, rather than attempting fixes under the pressures of an impending outage completion date. The IFAT, along with subsequent testing during installation, also provides oversight and verification to aid in meeting regulatory requirements for a site.

In addition to the time- and cost-saving benefits of an IFAT, integrated testing allows the customer and vendors to obtain knowledge of and comfort with the integrated security solution. The tests are designed to verify that systems and applications perform as required and do not negatively affect system operations. Upon completion of the IFAT, customers and vendors gain confidence that the integrated solution can be implemented without adverse effects on vendor systems and will function to meet the customer’s needs.

Because ICSs are now more integrated with IT systems and are largely digital, the number of vulnerabilities has increased. Therefore, the need for a more consistent cyber security implementation is extremely important. With the proliferation of cyber attacks, it is even more important to include cyber security in the initial design of an ICS, rather than later retrofitting multiple systems, which can be costly—and still not guarantee a reduction in cyber vulnerabilities.

Cyber security also affects vendors and integrators. If not adequately addressed, attacks—whether intentional or unintentional—can have impacts ranging from trivial to significant environmental discharges, serious equipment damage, and even deaths. The number of attacks will only increase as ICS environments become more interconnected with IT systems.

Plant operators are also at risk. With more ICS environments having mandatory regulatory compliance requirements addressing cyber security, significant fines can result from noncompliance with standards such as NERC Critical Infrastructure Protection (CIP), CFATS, and others. It can also reduce asset reliability, impacting neighboring plants or other regional entities.

In addition, without an IFAT, operation and maintenance problems with security controls may not be discovered until field installation, when they are expensive to fix. These issues often involve simple configuration changes, or multiple cyber assets not being captured by the vendor’s security solution. These issues are easy to fix if found early but require significant and expensive rework in the field during difficult outage operations.

Execution of an IFAT is intended to mitigate these issues before anticipated outage dates and allow redesign and troubleshooting to be completed in a more suitable environment. In addition, the IFAT (and subsequent testing during installation) can provide oversight and verification to aid in meeting the compliance requirements in an ICS environment.

A successful IFAT requires a team with representatives of the customer, the vendors (including control system vendors and security solution vendors), and a neutral third party. Customer representatives are necessary to provide oversight, decision-making, and technical knowledge of the corporate systems; they also need to learn about and be confident with the integrated solution. Representatives from each control system vendor and integrated security solution vendor must be present to verify that their systems are configured as required and to provide technical insight when troubleshooting the integrated security solution.

Finally, an impartial third party is fundamental to a successful IFAT. The third party defines and runs the testing activities and serves as the host for all customer and vendor representatives in order to create a neutral, productive work environment. The customer, vendors, and third-party host collectively ensure the success of the integrated solution and gain knowledge from one another throughout the IFAT process.

Conducting an IFAT

Any IFAT requires planning and preparation in order to be successful. Its scope has three basic areas:

  • Determination of systems and networks.
  • Testing to ensure that the equipment meets regulatory requirements and can be maintained by the customer.
  • Reporting IFAT results to vendors.

First, with the guidance of the neutral third party, the customer and vendors must determine the integrated network design and the systems required for testing at the IFAT. To provide the most benefit, the actual systems being installed at the customer site should be present at the IFAT for testing. When the actual systems are not available, equivalent systems should be used.

Once the integrated network has been designed and the necessary systems have been identified, a test plan is developed. The IFAT test plan, written by the neutral third party, based on customer and vendor requirements, describes the integrated network tests and security application tests to be conducted. This test plan is reviewed with all IFAT participants to ensure understanding and agreement. Upon agreement on a final test plan, detailed test plans are developed, including step-by-step guides and pass/fail criteria for each test. These test plans are followed during actual testing activities.

Upon completion of the IFAT testing activities, a final results package is distributed to all parties that includes the results of all tests, final baseline configurations for all systems, and an action item list for unresolved issues. This set of documentation provides the customer and vendors with the necessary tools for a smooth installation of system components.

With the current trend of more intelligent ICSs and increased regulatory compliance, the best approach to achieving ICS and IT integration is by conducting an IFAT. Performing an IFAT avoids costly redesign and troubleshooting during plant outages, saving time and money while ensuring a better security solution.

—Contributed by Jerome Farquharson, ([email protected]), critical infrastructure and compliance practice manager, and Alexandra Wiesehan, cyber security analyst, for Burns & McDonnell.

SHARE this article