Existing nuclear power plants are increasingly facing the conversion to digital instrumentation and controls technology. Meanwhile, new nuclear designs have digital technology integrated throughout the plant. Digital controls will soon be inevitable, so how do we make the transition as smooth as possible? Without losing focus on the technical solutions, organizations have to pay attention to the nontechnical issues as well.
For many engineers involved in digital upgrades, their focus is on the technical issues of hardware and software design, the development and execution of test plans, or field implementation of communication networks. Unfortunately, a digital upgrade can be technically successful but doomed to failure because of issues on the nontechnical side. When a project is perceived as a failure by the management, then the next digital upgrade can be extremely difficult to get approved even if there are tremendous benefits for the organization. The opposite is also true. If a project is perceived as a success, then management is more likely to approve and support future digital upgrades.
Although many issues, both technical and nontechnical, can affect an upgrade, which ones should technical personnel address? Sometimes budget, scope, and schedule decisions are made by project managers and executives with input from technical leads. If you find yourself responsible for selling a digital upgrade to management, or managing its implementation, here are some important nontechnical aspects of the task to keep in mind.
Increase Management’s Knowledge of Digital Technology
For many nuclear power plant managers and executives, the word "digital" conjures up an image of their Blackberry or their personal computer. Engineering managers usually have a good understanding of all the major engineering disciplines and have a really strong background in either mechanical, civil, or electrical engineering. These managers understand the consequences of a pump cavitating or an electrical breaker tripping. Sadly, few managers have a strong background in instrumentation and controls (I&C), much less digital controls. Because of their background, it is hard for them to relate to issues that are unique to digital upgrades. If they do not understand, they will be unwilling to make decisions related to digital upgrades, unwilling to commit to large capital projects, and unwilling to commit resources. Thus, the first concern is educating managers who are not typically familiar or comfortable with digital upgrades.
Reduce Confusion. One way to improve the comfort level of managers and executives is to increase their knowledge about digital upgrades. It is important to describe issues in ways the listener can understand. Use simple, clear illustrations. As always, avoid using jargon, because your listeners will lose interest fast and may not understand what you are telling them. Also remember that acronyms can mean one thing when discussing a digital upgrade but mean something else to your listener. For example, an MOV at a nuclear power plant often means "motor operated valve." However, an MOV is also short for "metal oxide varistor," which is used for electrical protection in electronics.
Avoid Information Overload. One frequent type of mistake is overloading a person with information. Digital upgrades usually have a project length that ranges from several months to a couple of years. There will be plenty of time to enlighten management on digital issues and obstacles. As you discuss new concerns, it is important to allow people to digest what they have learned. When given the chance, pick only one or two of the most pressing issues and address them. Take advantage of informal opportunities when they occur.
Tailor Communication for the Person. Know your audience. Some people respond quickly if you leave them a voice mail, but it may take them a couple of days to respond to an e-mail. Other people are just the opposite. Each person is different. So when increasing managers’ familiarity with digital technology, choose a contact method that best suits each one’s personal preference. If their preferred method is not known, then ask them. People will tell you what works best for them.
Tailor Information for the Person’s Time Constraints and Interests. Higher-level managers or executives often will have less time than those in lower managerial levels. It is not that they believe that your concerns are unimportant; it is that they have many matters to deal with daily. Therefore, it is important to be able to tailor information for each person’s time constraints. A vice president may need to know only the part of the design that has a major impact on the organization. A mid-level or lower-level manager may need more information than is given to an executive. Remember that people will pay attention to and listen for what their role is or what their involvement may be. Be sure to highlight the decisions or actions that you need from the manager.
Here is an example related to cyber security. A vice president may only care that the digital upgrade does not meet the newly proposed cyber security standards. That person’s help is needed in contacting executives at other companies to help define the number of cyber security levels as well as develop a description of each of those cyber security levels. In this example, the scope of what is needed from the executive is narrow and focused. The amount of face-to-face time between the technical person and the executive could be less than one minute. As a follow-up, the technical lead may be asked to prepare a one-page write-up that provides concise talking points for the executive.
Now take the same issue, cyber security, and consider the needs of project manager or a first-line supervisor. These people need to coordinate resources from other departments and understand the impact of certain decisions. The face-to-face time may be considerably longer and the type of information may be considerably different with your direct supervisor than with the executive.
Developing Digital Skills
A second area of concern is the need to train a different type of instrument technician. For years, I&C technicians had a mechanical or electrical background. Now they also need to understand software and communication protocols. Today, plants need hybrid I&C technicians who incorporate all four skill sets.
Develop Hybrid Digital Experts. Utilities have begun to address the need for hybrid experts who have mechanical, electrical, software, and communications protocol skills. Unfortunately, companies cannot go to a university or a technical college to find people with these skills or knowledge because most educational institutions do not yet have programs that provide this type of cross-training. Therefore, companies have to develop these people internally with on-the-job training plus some specific vendor training.
Form Digital Groups. Now that more plants are transitioning to digital technology, more companies are beginning to create digital groups, while companies that had already started such groups are increasing their size and scope. With digital group formation, companies are initiating new training qualifications for their personnel. It is not unusual for these groups to focus only on the software portion or the design portion of the application. However, there is a need for digital groups to be formed in maintenance departments as well.
Select Topics for Training Digital Personnel. At a minimum, training for digital I&C personnel should include the following topics, which are not exhaustive but provide a basic foundation:
Software topics should include development, testing, and troubleshooting. Additional software topics should address quality assurance, storage practices, disaster recovery plans, and system response.
Communication topics should include networking, protocols, troubleshooting techniques, and cyber security.
The mechanical portion should involve measurement of flows, pressures, levels, and vibrations. Tube routing and process, fluid system design, and system response times would also be of interest.
Electrical topics would include measurement of temperatures, voltages, currents, and power. Electromagnetic interference (EMI) and basic circuit theory are needed as well.
A course in human factors engineering is a must if expectations for personnel include display development.
Understand the Regulatory Guidelines for Digital Technology. Even though the regulatory landscape for digital technology is changing rapidly, personnel have to be familiar with the current and proposed changes. Digital I&C engineers and technicians should read and understand the regulatory guides, branch technical positions, and interim staff guidance. In addition, the Institute for Electrical and Electronics Engineers (IEEE) 7-4.3.2 — Standard Criteria for Digital Computers in Safety Systems of Nuclear Power Plants, IEEE 603 — Standard Criteria for Safety Systems of Nuclear Power Generating Stations, and the report jointly issued by the Nuclear Energy Institute (NEI) and the Electric Power Research Institute (EPRI) — Guideline on Licensing Digital Upgrades (NEI 01-01 and EPRI TR-102348) provide a great deal of information about licensing digital upgrades.
Choose Sources of Training. Training should cover the different phases of an application’s life cycle such as design, testing, implementation, and maintenance. Though some on-the-job training is needed, that training should be supplemented with vendor-specific training, classes from industry groups such as the International Society of Automation (ISA), or EPRI- and company-developed classes. The ISA is a good source for basic measurement and communication classes. EPRI offers classes on EMI. Instruction on basic circuit theory, licensing issues, and quality assurance are often provided by utilities.
Creating Programs That Sustain Digital Applications
The nuclear industry has developed preventative maintenance programs and in-service inspection programs to support the continued operation of nuclear plants. In addition, the Institute for Nuclear Power Operations has created engineering program guidelines for 18 different areas. Some of these are setpoint control, fire protection, flow-accelerated corrosion, valves, heat exchangers, welding, and motors. Unfortunately, there is little definitive program guidance available to those responsible for nuclear plant digital control system upgrades. Lacking such guidance, the following are a few suggestions to help you bridge this digital information gap.
Software Development Programs. In the past, utilities have developed software without formal programs. IEEE Standard 7.4.3-2 described the various phases of software development for safety systems; however, it does not apply to systems that are not safety-related. This gap is a concern that needs to be addressed.
The nuclear power industry should look to the software industry in order to improve its software development process. The primary purpose of the nuclear power industry is to generate and market electricity in a safe and economical manner. In contrast, the primary purpose of software companies is to develop and market software. As such, the software industry has dedicated resources and funding for software development at a much higher level than the nuclear industry has. Because of the competitiveness of the software industry, however, it may be difficult to benchmark these type of companies. If benchmarking efforts are not effective, then another alternative is to hire people from the software industry.
No doubt, utilities would find it beneficial to develop more formal programs that provide guidance on writing specifications, requirements documents, and design description documents for both software and hardware. Also, organizations are beginning to see the need to integrate data from different control systems into their work processes, which means that data definition documents are being written. By using open protocols and defining data, companies are able to transfer information from one database to another database relatively easily.
Software Testing Programs. Traditionally, the design and licensing of nuclear plants have been very deterministic. Since the design was deterministic, the testing philosophy could also be very deterministic. Essentially, testing was very simple. If certain inputs are present, then a given outcome should be expected. With digital technology, the controllers can now handle more than one or two inputs. The possible combinations of inputs become hard to manage very quickly.
Therefore the testing philosophy has to change because of the complexity of the design. In the future, testing will have to be expanded beyond proving that two inputs will cause a specific output. Testing will have to prove that the processor can do 25 tasks at one time without any degradation in performance. Also, testing will have to look at degradation of functions if there is heavy network traffic.
Another issue with software testing is deciding how large to make the test platform or test bed. The test bed consists of processors, inputs, outputs, communication networks, and test equipment. In the past, test beds have been very small — perhaps only one processor along with a few inputs and outputs. This meant that personnel often could only test a portion of the code at one time. Now that applications are dependent upon much more robust networks, testing becomes more complex. To prevent a complete failure of an application due to a partial network failure, functions are partitioned across multiple processors. The test beds now have to be large enough to test the communications between multiple processors in order to test the partitioning portion of an application.
One possibility is for individual plants or utilities is to pool resources and develop a common test bed. This last option can result in a test bed with higher fidelity but at a reduced cost for companies. Therefore, each utility has to balance the cost of an enlarged test bed against the risk of a latent software flaw.
Software Quality Assurance Programs. A great deal of software at nuclear plants is outside the scope of the formal quality assurance plan for that plant. The reason is that most software is not safety-related. With the advent of safety-related digital controls, organizations have to revisit the issue of quality assurance for software.
There are other drivers for an expansion of the quality assurance program as it relates to software. As utilities increasingly use software to prove the design basis of a plant, or as an input to their operational decision-making, questions are being raised about the quality assurance levels of that software. Certain applications may have been developed for non-nuclear parts of a company but are now being used by nuclear organizations for licensing or for engineering justifications. Because this non-nuclear software was never formally verified and validated, the question becomes whether or not the results of this software should be accepted for use by nuclear power plants.
Another area where this is becoming an issue is the reactivity excursions caused by nonsafety-related equipment. In the past, many organizations have deemed software for nonsafety-related equipment to be a lower quality assurance level or possibly outside of the quality assurance program altogether.
Software Configuration Management Programs. How and where should organizations store backup copies of their software? The answers vary from embedding software in a document to storing disks in a document control safe. The same standard for software configuration management does not need to be implemented across the nuclear industry. However, each company needs to develop a standard way of handling, storing, and revising software.
Electromagnetic Interference and Radio Frequency Interference (EMI/RFI) Programs. Almost all of the U.S. nuclear operating fleet was designed and constructed prior to the issuance of the landmark EPRI Technical Report (TR-102323), Guidelines for Electromagnetic Interference Testing in Power Plants. Since that report, organizations have used these guidelines to assist them in qualifying equipment that is potentially susceptible to, or that could radiate, EMI/RFI. The strategy for plants has been to address qualification issues as they happen.
Now a shift is beginning to occur in the nuclear industry. As more and more digital equipment is added to plants, the need for a comprehensive strategy is more apparent. This includes the use of a spectrum or frequency management program for EMI/RFI. The program would review each room or area of the plant to determine which frequencies are used. When any anomalies are documented in a test report, they can be compared with the known frequencies for that room to determine if there is a problem or not. The result is a simpler and faster qualification process for new pieces of equipment. The use of a comprehensive strategy also makes it easier for companies to justify large communication modifications such as a replacement of the entire security radio system.
Cyber Security Programs. The poster child for programs that are needed for digital upgrades is cyber security. In 2009, the Nuclear Regulatory Commission (NRC) developed regulatory guidance on this issue, and each nuclear plant submitted a cyber security plan to the NRC. Recently, The U.S. Department of Energy (DOE) created the National Supervisory Control and Data Acquisition (SCADA) Test Bed, which enhances cyber security of control systems used throughout the electricity, gas, and oil industries. The National SCADA Test Bed is a joint effort between the Idaho National Laboratory and Sandia National Laboratory. With so many resources being diverted for cyber security, and given all the publicity that it has garnered, it can easily be seen that cyber security will be one of the programs that has to be in place to support a digital upgrade.
–James H. Flowers (firstname.lastname@example.org) is the I&C supervisor in nuclear development for the Southern Nuclear Operating Co. in Birmingham, Ala.