Preparing for a NERC CIP Audit

Undergoing a NERC CIP audit is an ordeal, particularly when the ground rules are not clearly spelled out in advance. An experienced NERC CIP auditor lays out a comprehensive five-step plan that will show you how to prepare for a successful audit.

It’s no surprise that a North American Electric Reliability Corp. (NERC) Critical Infrastructure Protection (CIP) audit generates plenty of anxiety during the weeks preceding one. However, there is a preparation strategy that will allow you to get those emotions under control.

This article presents a detailed analysis of a five-step audit preparation strategy, followed by its application to a specific NERC CIP task. The strategy, based on real-world NERC CIP audit experiences, is a useful guide for those responsible for audit documentation preparation. Next, the entire five-step process is applied to a single NERC CIP sub-requirement as an example. You also will find some suggestions on where to seek additional help.

The NERC CIP audit environment is changing. Regional Entities (REs) are doing more work upfront and less on site. There are eight REs representing, according to NERC, “virtually all the electricity supplied in the United States, Canada, and a portion of Baja California Norte, Mexico.” If the Entity (essentially any company that generates or manages power flow on a transmission and distribution network) submits a clear Reliability Standard Audit Worksheet (RSAW) with good supporting evidence, most of the audit can be done before the Regional Auditors (RA) arrive at the Entity’s site. Every requirement the RAs check off before arriving on site means one less activity that you need to interrupt your subject matter expert’s (SME’s) schedule for, which is always a good thing.

Step 1: Understand the Requirements

The NERC CIP Reliability Standards (CIP Standards) were written by committee and, generally, without the benefit of a technical writer. The result is that they are not extremely clear and are often confusing. Although understanding the current version is vital, it is also useful to review the next version that is in draft form. Review your registration status against the requirements so that you are addressing the proper version and constraints.

The key sections in each of the eight CIP Standards are the two sections entitled Requirements and Measures. The Requirements describe what the Entity needs to do, and the Measures describe what evidence the auditors will be checking, although the section is written at a summary level.

The first step in understanding the Requirements within any of the CIP Standards is to review the language. Two especially interesting concepts are presented in the Requirements section that require close interpretation.

The first interpretation problem involves the use of technical and/or procedural controls. These require documentation, and when technical controls cannot be implemented, a technical feasibility exception (TFE) is required. Sometimes written with an “and” and sometimes with an “or,” the documentation requirements of the two cases are different. CIP-005 R2 uses the phrase “organizational processes and technical and procedural mechanisms,” whereas CIP-007 R5 uses the phrase “technical and procedural controls.” What is the difference between the two phrases? You need to understand this because it affects your procedures and the evidence you provide for the auditors. Note that Compliance Application Notice–0017 provides the interpretation that every technical control implies a procedural control, the process that implements the technical control. It also provides guidance on when either technical or procedural controls or both are applicable.

Another interesting interpretation problem concerns the collection of terms that show when documentation is required. The following words listed throughout the Requirements require the Entity to provide a document of some sort as evidence: policy, methods, processes, procedures, identify, maintain, document, list, approve, develop, review, implement, assign, assess, establish, controls, perform, update, retain, create, deploy, exercise, use, and test. Every one of these words indicates a document or task that is required by the standards. Eventually, you will need evidence (documents) each time one of these words appears.

Go through the Requirements one by one and read them carefully. Make a note of anything that you don’t understand. Create an outline of the significant statements or start a spreadsheet to track your information.

The Requirements themselves can be very confusing. As the example using CIP-005 R2 and CIP-007 R5 demonstrates, it can be daunting to untangle the true intent without help from those more experienced in the audit documentation process. Fortunately, a number of sources can help you understand the Requirements (see sidebar).

Step 2: Know What Auditors Are Looking For

Auditors are looking for just one thing—the opportunity to verify that you are “auditably compliant” with the CIP Standards. The auditors look for documentation of procedures, and verification that you are executing them (as referenced in Federal Energy Regulatory Commission [FERC] Order Docket No. RM06-22-008). This will include both the RSAWs and the associated evidence for each requirement.

The RSAWs used during the actual audit need to present two types of evidence, as referenced in FERC Order Docket No. RM06-22-008:

In critical infrastructure protection, and especially in the cyber security environment, the implementation of security measures is largely dependent on complex plans, policies and procedures that must be repeatable and verifiable. This necessitates documentation of both (1) the procedures to be followed and (2) verification that the procedures were followed as directed. These complex procedures require clear and consistent instructions (documentation) and consistent execution (implementation). Further, these procedures also require a method for reporting their completion.

To address this requirement, each Entity has processes, procedures, and policies that describe how the Entity meets the requirements of a RS, the means that the Entity uses to meet the requirement. The second type of evidence is used to show that the Entity is using these processes, procedures, and policies. This evidence is typically screen shots, lists, reports generated by applications, logs, or other evidence generated from the devices themselves.

Auditors also look for enforcement. If a policy says employees “should” follow a procedure, the auditor will want to know what action you took when an employee didn’t follow the procedure. Find out what the auditors consider “good” documentation versus “weak” or “poor” documentation. Note that your experience based on a “spot check” will most likely not be a good indicator of readiness for an audit because a spot check does not have the same requirements as an audit. A spot check is much less rigorous than an actual audit, so evidence that was sufficient for a spot check may not be adequate for an actual audit.

Usually, the auditors will not want to look at all of your evidence. Instead they will use a program called RAT-STATS (available for download, see sidebar) to select a statistically representative random sample. No one wants to look at 90 days of firewall logs (especially if there are thousands of pages in the logs). They will want to take a small sample, just enough to show that your process is retaining 90 days of logs. RAT-STATS will allow them to select a statistically accurate sample. This will also be used for personnel risk assessments, training, and access changes.

In summary, auditors want to see a procedure that meets the Requirement and evidence that you are following the procedure.

Step 3: Analyze Your Evidence

A documentation spreadsheet is very helpful for tracking your compliance. For each Requirement and sub-requirement, create a column, and then use a row for each piece of evidence. Each row should have a file name or document title. You can use a comments column to provide additional details (perhaps how to access the data). As each piece of evidence is analyzed, update the spreadsheet. Even if all pieces of evidence are not used for the RSAW, you may need them if the auditors ask for additional evidence during the audit. This spreadsheet will be very useful when packaging your evidence.

Analyzing your evidence is a three-part exercise:

  • First, determine what processes and procedures you have that demonstrate compliance. Use the documentation spreadsheet to track these documents. Your SMEs should be able to identify all these policies, procedures, and processes.
  • Second, for each process and procedure identified, find the supporting evidence that shows you are executing processes and procedures properly. This must be carefully done, or you will miss some of the evidence. Consider the size of the evidence. Typically, firewall logs contain huge amounts of data, but auditors will only want to see a sample. Have a plan for how to extract the sample.
  • Third, verify that your evidence is complete and sufficient. This is best done by a different person (not the one who created the evidence) to demonstrate a more objective analysis of the sufficiency of the evidence. If you are missing evidence, go back over the previous step and see if the evidence can be located or generated.

The third step can require a lot of work, as there may be dozens of documents to review. Also, associating documents with Requirements can be confusing, especially as a single document may be referenced in several different Requirements. The key is your document map that cross-references Requirements with the evidence.

When analyzing procedures and processes, you should also review them for compliance. Typically, one procedure will meet several sub-requirements. Note the specific sections that demonstrate compliance.

When you package your evidence, you will need this information for the bookmarks. Carefully consider the language in your procedures. Words like “should,” “generally,” and “typically” should be avoided. Use “must” or “shall” instead. Look for clear instructions and positive statements in your documents. See if reviewers can understand the document or if they need to ask for an explanation, especially for a process or procedure. If they need an explanation, then this is a good time to rewrite the document to provide clarity to show that it meets the requirement. Doing this early gives you time to update the document before your audit.

Review your definitions and look for any cases where you do not use industry references but redefine terms. If a term is in the NERC glossary and you define it differently, expect to be asked to defend your definition. It is best to simply reference the NERC glossary itself in your documents rather than repeat the definition.

Step 4: Package Your Evidence

The fourth step requires you to take the raw evidence (data) and package it into a format that makes it easy for the auditor to verify compliance. In most cases, the auditor will only want to see a sample of your evidence, so make sure you have internal procedures defined for extracting those samples. Remember that searchable PDFs are the form of evidence preferred by most auditors, but other formats (such as Excel spreadsheets) are acceptable.

The first step in packaging your evidence is to create a folder for each Requirement. You can also put the document reference in the folder. This gives you a good place to track your progress in preparing the evidence. In this folder, create a copy of your files and use the actual file names you will supply to the auditors. This will allow you to leave the originals untouched and available if you need them. The first file in the folder should be the RSAW. You will update the RSAW as you package your evidence and decide what should be directly referenced in the RSAW and what will be held back until it is needed or requested by the auditors.

You can also create a folder for common files. That way, you only need one copy of them. If you do this, be sure that all the bookmarks for each file include every Requirement that references the file.

Auditors greatly prefer bookmarked and searchable PDF files, especially if the document is referenced in several Requirements. Highlighting specific areas that address individual Requirements makes them stand out for the auditors. You may also want to put several smaller but related documents into one PDF with an explanation describing why they are related and what Requirement they address. Auditors also like Excel spreadsheets because they are easy to sort and manipulate.

Some evidence may not be obvious. For instance, if you are using SharePoint, you may have the document in one place, but the version history in another. Showing multiple versions that were effective for various date ranges during the audit period may require some manual work. Prepare before the audit for any versions that may be requested by auditors if the time to create each version is longer than you may have available while the auditors are on site.

A good practice is to put the policies, procedures, and plans in individual documents, including the revision history if that is kept in an individual file (SharePoint documents may be created this way).

Keep large documents in their own files. For smaller documents, selected related ones (usually by sub-requirement), put a summary at the top, and group them together in one PDF. Use links in the summary to point to the individual documents or RSAW references.

Regional Entities now provide secure electronic means to submit your evidence. Auditors prefer that each RSAW stand alone, so creating a folder for each Requirement with the RSAW and the associated evidence is the best way to submit it.

Presenting your evidence to the RE in RSAWs with a clear format and a good structure that is easy to understand and directly addresses each Requirement will allow the auditors to check off each Requirement one by one in their off-site review. Getting the evidence early allows them to review it at their location more efficiently. If the Entity submits a clear RSAW with good supporting evidence, most of the audit can be done before the RAs arrive at the Entity’s site. Every Requirement RAs check off before they are on site means one less activity that you need to interrupt your SME’s schedule for.

Step 5. Practice the Audit

The fifth and final step is to practice an audit. You can do this internally with your corporate audit team (if you have one), by using cross functionality between SME groups, or with an external consultant. Be certain to do this far enough in advance of the audit so that if you find serious weaknesses you can remedy them before the audit.

If you are using an internal mock audit, try to structure it so that the same people do not create the evidence and then audit it: You want a fresh set of eyes on the evidence. It helps to have a technically qualified audience, especially if they have not been involved in preparing the evidence. For example, you may have corporate IT staff who could act as the auditors.

Outside vendors can also be used for the practice audits. They generally provide a condensed version of the actual audit. Verify the credentials and experience of such auditors before selecting an outside vendor.

The goal of the practice audit is to verify that your evidence is sufficient to meet the Requirements and is clearly presented. Present only the evidence you have prepared that will be given to the actual auditors (and in same format that will be presented to the auditors).

Case Study: CIP-003 R6

This case study applies the five-step plan for preparing your evidence just described to the CIP-003 R6 requirement for Change Control and Configuration Management, particularly as its Requirements are complicated by its ties to CIP-005 R1.5, CIP-006 R2.2, and CIP-007 R1-R5.

CIP-003 R6: Understanding the Requirements. CIP-003 R6 requires that the Entity “shall establish and document a process of change control and configuration management for adding, modifying, replacing, or removing Critical Cyber Asset hardware or software, and implement supporting configuration management activities to identify, control and document all Entity or vendor-related changes to hardware and software components of Critical Cyber Assets pursuant to the change control process.”

The first question an Entity must answer is whether this is one process or two. There are compliant processes that are separate and also ones that are a single process. Let us assume for our example that there are two processes: one for change control and one for configuration management, but with common interaction between the two.

What did the authors mean by “change control,” as they did not provide a formal definition? Knowing that all the CIP standards are focused on protecting the bulk electric system, the inference is that the negative impact of changes will be minimized or eliminated. A review of the literature available for change control provides some clarity. As focused on the CIP standards, change control includes three components: documentation, approval, and testing.

CIP-007 R1 also provides guidance on “significant” changes and the testing required, ensuring that the change will “not adversely affect existing cyber security controls.” This will be documented as part of the change control process.

Though not a formal interpretation of the Requirement, all significant changes should have documentation of the change approval, test procedures, test results, and implementation status. This documentation should include the date of each activity, so the Entity can demonstrate that successful testing was completed before implementation.

Note that the testing must address the effect of the change on cyber security, especially cyber security controls. The CIP requirement does not address application testing. Testing must include the effect of the change on ports and services open on the asset. It should also include any other cyber security controls. For Windows-based PCs this does include some of the registry entries, because they directly affect cyber security. This should be documented as part of the change control process.

Configuration management is closely related to change control. CIP-007 R2 requires that “only those ports and services required for normal and emergency operations are enabled.” Configuration management is the process that provides the documentation that the auditors will use to review compliance with this requirement. A key to this documentation is the justification for each port and service active on the asset. Configuration management will also provide the configuration needed in CIP-009 to recover an asset.

Configuration management is typically documented using a baseline for each asset or a group of similar assets. This can take many forms. For routers, this may be a text file of the access controls lists. For Windows-based PCs, it may be a backup file plus a ports and services baseline. One key is that every asset should have a baseline configuration.

The configuration management baseline documentation must be updated anytime the baseline changes. For example, switching to a different anti-virus vendor may affect the ports needed for operation. This could either open or close ports in the baseline. The configuration management process should document this change with a new version of the baseline.

CIP-003 R6 Auditors’ Expectations. Auditors typically expect to see a change-tracking system for all significant changes (it may also track other changes). They will expect to see that every significant change is documented, tested, and approved before it is implemented. The only exceptions auditors expect are cases where there is no test system or backup system to test the change on. In those cases, auditors will expect to see specific procedures documented that will minimize any possible negative impacts.

Auditors will typically select a statistically accurate sample of all significant changes and review these changes in detail. This will include looking at the before and after test results and matching them to the baseline for the device. If there were any changes that affected the baseline, they will need to verify that the baseline was updated as required by the standards (within 30 days of the change).

Auditors will expect to see a documented process for configuration management. They expect to see baseline documentation for the ports and services for all critical cyber assets (CCAs) and cyber assets within an electronic security perimeter (except those with TFEs for CIP-007 R2). This documentation should include the justification for each port and service that is active on the asset. Typical justification is a manufacturer’s statement showing what is required, but if that includes a very large range of ports, an auditor will likely question it. Auditors may also ask if you have turned off any ports and then tested the asset to see if it will function without them. Simply doing a display of the current ports and services is not considered adequate justification.

Auditors will look for any cases where the baseline should have been changed as a result of a significant change to a CCA. All changes like that typically receive a detailed review.

CIP-003 R6: Analyze Your Evidence. At this point, review your change control and configuration management process documents. Are all the procedures accurately described? A good check is to take the procedure and see if someone who does not work in that area can successfully execute it. Alternatively, you can take someone who has implemented changes and walk them through everything they did to see if there are any steps they took that are not documented. You want to be confident that the procedures accurately represent what people really do.

Also verify the interaction between processes. Verify that if the ports and services baseline changes, part of the change process is to update the associated documents (this can include asset lists when decommissioning or adding an asset, configuration baselines, and even recovery procedures).

The next step is to determine what evidence you need to show that you are following the process. For change management, this will start with your change-tracking system. Assuming you are using a commercial change-tracking system (like Remedy, for example), you will need to gather any additional procedure documentation for the system if that is not covered in the previous documents you reviewed.

Export from the change-tracking system a report of all significant changes. If this is not easily done, it is a flag that you will need additional resources available during the audit to respond to data requests. The report should include a description of each change. If possible, determine which changes resulted in a change to the baseline documents. If that is not readily available from your tracking system, then find an alternate way to get that data.

From this report, you can run RAT-STATS (with confidence level of 95%, rate of occurrence of 0.5%, and precision range of 10%) to determine the sample size. For example, if there were 1,000 significant changes, you would need to sample 34 random changes. RAT-STATS can then be used to generate the 34 random samples. You can expect the auditors to do a similar calculation.

For the changes in this sample, pull the required documents and populate them. This will start with the change ticket, but it will also include the before and after test results. It may even include associated e-mail referenced in the ticket; an e-mail containing the justification for a change in ports and services is a good example.

Another concern for evidence is the organization of your Entity. Many Entities have completely different procedures and documentation for generation, transmission, and control centers. Often an IT group maintains the network connectivity and network devices, but it typically has different processes as well. Consider how to organize these different processes for presentation to the auditors.

CIP-003 R6: Package Your Evidence. When you analyzed your evidence, you should have been thinking about how to package it. As you documented each piece of evidence in your document reference spreadsheet, you should note how and where you want to package it. You should have created a folder for each CIP and then copied each document to that folder as it was analyzed, unless it was a large file that will be sampled instead. At this point, you can combine individual files into PDFs with bookmarks for the auditors to use.

Prepare each folder so that it contains only the files you will submit as evidence, the RSAW, procedure documents, and attributed evidence. Only keep the samples that you reference in the RSAW; however, you may want to keep other samples you have generated in an easily accessible backup folder.

CIP-003 R6: Practice the Audit. This can range from a simple exercise to a major effort. The primary objective is to verify the accuracy and adequacy of the evidence. A secondary objective is to prepare your SMEs for testifying during the audit.

The compliance team can make a first cut at this by reviewing the evidence. That can give them a feel for which areas are strongest and which areas may need more work. As this is solely in-house, it has the least cost. They can also select which SMEs will testify during the audit.

At this point, the Entity must decide whether to outsource the practice audit or keep it in house. This is generally a management decision. If there are sufficient trained and available resources within the company, an in-house practice audit is a good solution. In most cases, there are not enough internal resources to do a good mock audit, so an external consulting team with specific NERC CIP audit expertise is often used.

Set up a room similar to what the audit team will expect. Typically, they will be on one side of the room (with no one behind them). The room should have a projector and a computer with the data. During the actual audit, you will want to make sure that the computer you are using cannot get to live data, only the documents that you have prepared for the audit. This will limit what the auditors can see and will not allow them to see anything unexpected that could expand their audit.

There are three levels of mock audits that reflect the Entity’s commitment of resources:

  • An audit by one or two auditors of a single Requirement (or up to a maximum of three Requirements) can be done. This will limit the SME participation and typically only requires one to three days on site. This is a good option if you feel that most of your program is strong but you have one or two weak areas.
  • A mock audit with one audit team (typically, two people) from an external consultant working on site for four to five days can provide a high-level review of the evidence. As this is roughly one-eighth of a typical regional audit (two teams of three people each on site for one to two weeks), this will not include a deep dive into the evidence. It does require SME testimony during the mock audit.
  • A full-blown mock audit with two teams for two weeks can include an exhaustive, deep dive into the evidence. This will likely include considerable testimony by the SMEs. This provides the most detailed examination of the evidence and the greatest opportunity for SMEs to practice their presentation skills.

If you have different groups that will be presenting evidence, you should present an overview for the auditors. With different groups (or business units), you can either present it all together, point-by-point, for all groups or do each organizational unit independently, one at a time. Let us assume for the example that there is only one organizational unit and one set of procedures.

Next, your SME should give a high-level description of the change control process. For this example, assume change control will be presented first, followed by configuration management. Show the key steps for approval, testing, implementation, and documentation updates for change control. Demonstrate the list of all significant changes to the auditors. Pick a sample change and go through it, showing each key step in the process and the associated documentation. Describe how your process meets the requirements at each step in the process. Explain how the testing process ties to CIP-005 R1.5, CIP-006 R2.2, and CIP-007 R1-R5.

Once you have completed presenting the change control process, then introduce the configuration management process with a high-level description of the process. List the evidence you have prepared for the configuration management process. Introduce your evidence for the baseline used for comparison purposes. Pick an example of a baseline that was updated during the audit period, and show the details.

If you selected a third-party mock audit, you will receive an audit report with observations and recommendations (your internal audit could also create a similar report). Be sure to allow enough time before the actual audit to implement the suggestions and improvement so that your evidence will be complete and sufficient to satisfy the auditors.

James E. Chance (jchance@corprisk.net) is a senior security consultant with Corporate Risk Solutions. He is a NERC CIP consultant and a regional auditor contracted to multiple regions. This article is based on a paper presented at the 2012 ISA POWID Conference.