Software ranges from shrink-wrapped products available “off the shelf” to custom corporate implementations of enterprise systems that require sessions with shrinks to keep everyone sane. Regardless of its complexity, every piece of software a plant uses, or interfaces to, poses critical issues that require life-cycle management. Although functionality has always been the chief specification for software, plants must pay far more attention to long-term quality issues. These two concerns are often at odds with each other.

A recent survey of power plant owner/operators revealed that “version control” is one of the most prevalent issues with software applications—along with having the right team in place to manage the software after the vendor’s implementation team has left. (See “Integrated software platform eludes many owners/operators” in POWER, September 2007.) In fact, these are only two of the many life-cycle software issues facing plant managers. Others include:

• Cyber security—from mundane patches for the latest viruses to new code and logic to protect against criminal or terrorist activity.
• Personnel training.
• Tuning, calibration, and retraining of model-based software, collectively called configuration control.
• Consistency in software tools.

Although software life-cycle management may seem like yet another paper-trail-creation exercise or an otherwise additional nuisance for a plant, the consequences of not thinking in terms of the bigger picture are real. Here’s just one example: Twice in the same year at one utility plant, software systems were responsible for safety and reliability impacts during plant start-up simply because the staff could not find the current version of the system software.

Like any other component

One way to think about the software life cycle is to treat a piece of software just as you would a piece of equipment in the physical plant. Software should be purchased, designed, integrated (with the computer hardware and other software), and maintained to satisfy a specific set of functions.

During the “design” phase, you define the functionality required, write a specification, purchase or design the software, install and test it, and then turn it over to the plant. At this point, the software needs an “owner,” just like any other plant component. The long-term care and feeding of software involves operating and maintenance considerations, as do physical components. Economic decisions about the level of maintenance for software will depend on its expected useful life. As with physical component maintenance, you will need some mix of corrective, preventive, and predictive software maintenance strategies, which will also require reviews, validation, and upkeep on defined, regular schedules.

At nuclear plants, where change and innovation only invites increased regulatory scrutiny and paperwork, software is generally expected to have a long life cycle. Nuclear facilities also have great difficulty managing things without paper. Even when they create an electronic document, they typically manage the information as electronic paper—as a simple electronic file. But it is simply too cumbersome to manage the life cycle of a digital product like software with paper. The volume and complexity of the information is too great.

Fossil plants are used to dealing with more frequent intervention in their software systems, frequently released new versions and revisions, enhancements (purportedly for efficiency and productivity), and even wholesale replacement after only a few years. Software suppliers issue service packs and revisions to add functionality to a device, to respond to customer requests, or to fix “bugs” and flaws. Users also add or delete data points, change parameters, and reconfigure systems regularly. Plant safety, reliability, and performance are at stake if revisions and changes are not formally documented and managed (see sidebar).

Even when life-cycle management processes are implemented, owner/operators invariably underestimate the complexity of digital systems. The processes for dealing with software in the field either tend to be underdesigned or are overwhelmed and out of date shortly after rollout.

21st-century toolbox

Maintenance workers carry some tools that are recognized by everyone and others that are recognized by those with only limited mechanical work experience. Likewise, software maintenance and upkeep requires tools, and their use needs to be just as typical and self-explanatory. Ideally, these tools shouldn’t require training before being used by those responsible for the software. Usually, this means that they need to be provided on common platforms, like Microsoft-, or web-based interfaces.

One common tool for plant systems is a well-defined configuration management capability. (See “Digital technology spawns need for configuration management,” POWER, July 2008.) Plants must be able to compare a specific software configuration to an “as-built,” “as-required,” or “current” configuration. Some software vendors supply these configuration management tools and some don’t. Some tools supplied with an automation system work well; some don’t. Invariably, vendors focus on the initial design and development of their software, not on long-term life-cycle issues.

The data management toolset also needs to be consistent with the toolsets required of other software packages. For example, plants are typically designed using sophisticated computer-aided design software programs. “Virtual” plants are delivered along with the physical plant. If a virtual plant is going to have any “life-cycle” value to plant management, then the upkeep tools must be familiar and self-explanatory to those responsible for its maintenance. Owner/operators are demanding that the design data be provided in forms that plant management can use throughout the life of the plant. This impacts the tools selected by the engineering, procurement, and construction contractors to design the plant and “deliver” a virtual plant.

The service center model

One way to manage software life-cycle issues (and, for that matter, current software operations) is through the third-party service or partner model. With this approach, the software vendor essentially becomes the assigned “owner” of the plant’s software. Several power plant software suppliers now offer a monitoring and diagnostic service on a subscription fee basis.

One of the life-cycle benefits of this approach is that plant systems or components can be consistently benchmarked to those at other facilities of similar vintage, duty, operating life, and model number. Model-based software may be especially amenable to a services approach. Essentially, software design, engineering, revision, upkeep, calibration and retraining, and maintenance are managed by a third party, allowing the staff to focus on the power plant’s physical operation and maintenance, not software life-cycle issues.

Many owner/operators with large portfolios of plant assets have already converted to a centralized performance monitoring facility model where monitoring and diagnostic (M&D) responsibility either resides or is shared. (See this month’s Special Report for a description of Entergy’s Performance Monitoring and Diagnostic Center.) Independent power producers and merchant plant owner/operators, who manage many of the latest combined-cycle facilities, depend on the gas and steam turbine supplier for 24/7 monitoring and diagnostics; related services providers collect, analyze, and manage fleetwide data for such machines. Now, software vendors with even more specialized, and often more complex, M&D capabilities are beginning to deploy the services model to power plants.

Going forward

Below are some specific recommendations for managing software life-cycle issues.

Measure to manage. You can’t manage what you don’t measure. Some in the software business talk of software quality assurance (QA). The diagram provides a high-level overview of the metrics being contemplated for QA evaluations of software.

Inspect, don’t expect. Software quality assurance evaluations often include these metrics. You can’t control what you don’t measure. Source: www.dcs.qmw.ac.uk/~norman/papers /qa_metrics_article/section_3_metrics.html

Hold implementation post-mortems. Plenty of meetings are held to plan a successful software implementation, culminating in the “go live” date. Post-implementation meetings should be held to determine what went right, what went wrong, who retains ownership, how to meet continuous training requirements, who handles software storage, and the like.

Conduct usability evaluations. This is an emerging area of analysis, as indicated by the recent publication of books such as Maturing Usability: Quality in Software, Interaction, and Value, by B. Jefferies, et al. (October 2007). Most power plant owner/operators assess products through the experience of their peers in power generation; formal usability evaluations may prove to be excellent supplements.

Outsource and partner. If your plant is not the flagship of the fleet, if power generation is not the “core business” of your plant owner, or if you are owned by a small utility, resources may be scarce. Even contemplating the purchase of performance-based software may be daunting because of a lack of expertise. The emerging service center model is ideal for these situations.

—Timothy E. Hurst, PE ([email protected]) is president of Hurst Technologies (www.hursttech.com), a consulting engineering firm specializing in instrumentation and control systems for nuclear and fossil-fueled power stations. He also is a POWER contributing editor.