Focus on O&M (December 2007) 

TRAINING

 The web: Ideal for skills development

To power the operations of large industrial customers and keep appliances and equipment humming in homes and businesses, Minnesota Power (MP) employees must collectively perform like a well-oiled machine—24 hours a day, 365 days a year. Perhaps no one at the company knows that better than MP’s operations training administrator, John Batchelder (Figure 1).

1. Training guru. John Batchelder manages Minnesota Power’s web-based operations training program. Training opportunities are available to everyone, from apprentices to seasoned operators and technicians. Courtesy: Minnesota Power

A 31-year MP veteran based at the coal-fired Boswell Energy Center, Batchelder cut his teeth on power plant operations in the U.S. Navy. Since then, he’s realized that people naturally take a task-oriented approach to doing their job—sometimes at the expense of understanding how their work affects coworkers. Which is why, he emphasizes, training courses offered to MP employees must include the big-picture fundamentals, including how electricity is generated. “You can’t put the roof on the walls of your career home before you build the foundation,” Batchelder said.

In 2005, MP first offered workers the opportunity to sign up for one of 100 company-paid “seats” for using General Physics Corp.’s GPiLearn portal. Since then, interest in the GP module covering power generation fundamentals has increased substantially. To meet demand, MP doubled the supply of free training seats to 200.

Thanks to efforts led by Batchelder and MP’s training site coordinators—and lots of “customer” feedback—GP training has been augmented by MP-specific training developed in-house. Now employees can tap into 3,200 lesson and testing modules.

MP wrote the job description for Batchelder’s position in 1999. As soon as he got the job, Batchelder began working to improve training opportunities for all Boswell employees. For the next few years, his responsibilities were limited to the plant housing his office. But in 2004, Batchelder’s domain was expanded, to meet VP of Generation Al Hodnik’s goal of enhancing training for employees at all MP plants by December 2005.

Working with his supervisor—Madelen Schuemann, thermal business operations manager for Boswell Units 1 and 2—Batchelder hunkered down to analyze the offerings of web-based training providers. After reviewing the curricula of the vendors he found most impressive—in terms of the price, quality, and applicability of training modules to MP’s future needs—Batchelder recommended going with GP. He was already familiar with the company, having taken some of its training courses in 1998 to prepare for an upgrade of the Boswell plant to a distributed control system.

The three core modules of GP’s Power Plant Operators training curriculum are:

• Power Fundamentals (Figure 2), which gives all kinds of plant workers the theoretical and practical foundation they need. Among its topics are boilers, turbines, combustion basics, ash removal, electrical systems, water treatment, emissions control, and safety.
• Core Skills Development, designed to help generation companies qualify their craft personnel and develop a multiskilled workforce. This module targets several plant personnel categories with subject matter such as instrumentation and control (I&C), electrical and mechanical maintenance, and plant and coal yard operations.
• Advanced Operator Training, for senior technicians. It covers boiler water chemistry, heat rate calculations, and other advanced topics.

2. Training on demand. General Physics’ web-based training curriculum begins with the fundamentals and progresses to advanced, specific topics, such as steam turbine auxiliaries. Courtesy: Minnesota Power

To augment the core modules, Batchelder and MP’s training site coordinators worked with GP to develop several levels of training. As employees complete the courses at their own pace, they advance through basic topics like applied math and complex ones like instrumentation. “The training helps them handle their own jobs better,” Batchelder said, “but because they now have power generation basics under their belts, they also have that big-picture view. Instead of just recalibrating an instrument, they know why doing so is so important to overall plant performance.”

Another step that Batchelder took to augment the core modules was to create an I&C training development group, comprising senior I&C personnel from various MP plants. They traveled to each MP plant, spent time discussing its specific I&C training needs with managers, and then created lesson plans germane to each location.

Last year’s offering of free training sparked plenty of employee interest in continuing education. But so did MP’s creation of a new position: apprentice operator. According to Batchelder, “those of us leading the training initiatives noted how many entry-level employees became interested in fuels after that department introduced a fuels technician apprenticeship program. Fuels created the template for our effort to customize employee training levels.”

Batchelder explained that once someone successfully completes an apprenticeship, he or she is immediately certified by Minnesota’s Department of Labor and Industry as an expert at their craft. The certification is recognized by many other states (Figure 3).

3. Are you experienced? Completing an apprenticeship program at Minnesota Power is an excellent career move. The state’s certification is portable to other states. Courtesy: Minnesota Power

That’s the case because everyone who takes a web-based training course must pass a test to prove they’ve retained what they’ve learned. Batchelder said, “The industry-standard passing grade is 70% retention, and that’s MP’s cutoff, too.” If an employee doesn’t pass the test, he or she can take it again. Employees can monitor their progress through the various training levels at any time, as can their supervisors. “MP encourages its employees to avail themselves of free training to reach their short-term goal, whether that goal is to do their current job better or to be promoted,” Batchelder explained.

Do supervisors like the new web-based approach? Absolutely, for several reasons. One is the outpouring of employee interest in continuing education it has produced. For example, Boswell alone currently has 81 people enrolled in training courses. Companywide, 164 employees have registered for some sort of training, and 99% of them are operators or specialists in fuels or I&C.

Another reason the training programs have worked so well is that they are open to everyone. Interested employees don’t even have to work at an MP power plant; all they need is a desire to learn how electricity is generated. Several members of the company’s General Office Building department are in that category.

A third measure of the initiative’s success: Several supervisors have asked Batchelder whether additional training modules could be developed to help their employees get the licenses required for certain MP positions. “The options for reconfiguring training curricula to meet the needs and interests of craft and other workers are almost endless,” Batchelder said. He adds that their supervisors of new MP engineers specializing in various disciplines are very interested in making sure that they receive training in the fundamentals of power production.

Finally, the web-based training program has led to an expanded menu of apprenticeship opportunities, like those for operators and fuels specialists mentioned earlier. Another new apprenticeship that Batchelder helped create is in hydro plant operations, at the behest of hydro operations supervisor Tom Hughes. To get this program up and running, Batchelder and Hughes met with representatives of GP, visited all 11 of MP’s hydroelectric facilities, and then supervised the development of a web-based training curriculum specific to hydro plants.

According to Batchelder, MP is now looking to expand the web-based training initiative while ensuring that employees take courses at the appropriate level. As an example, the company has established a “credentials committee” whose members would include Batchelder, plant production coordinators, unit supervisors and managers, and representatives of Local 31 of the International Brotherhood of Electrical Workers.

The committee is responsible for “slotting” employees in courses at the correct level, taking into account their experience and expertise. Workers would have a say in their assignment. As Batchelder explains, “for example, someone who’s been here for 20 years could argue before the committee that they shouldn’t have to take an entry-level course. And if a younger worker brings plenty of computer savvy to the table, he or she would likewise get credit for that. We have a huge workforce transition to prepare for as experienced people retire; plant technology is a lot more complex. That’s another reason training is so important.”

Through GP’s “co-funding” option, MP and two other power companies—Ameren and Colorado Springs Utilities—are subsidizing the creation of more GP training modules. “It’s good for all parties involved,” Batchelder said. “GP can offer more training to all their clients, and the co-funding utilities can get the training they want to develop for their own employees at a lower cost.”

Taking stock of what’s happened over the past year, Batchelder says it’s gratifying to see employees responding to, and wanting to help develop more, training opportunities. “We dropped a pebble into a pond a year ago with GP,” he said, “and it’s just incredible how the ripples have spread.”

—Contributed by Julie Aho of Minnesota Power’s Public Affairs Department.

FAST-TRACKING PROJECTS

Upgrading a New Orleans pumping station

The start-up of a 2,300,000-gpm pumping station on the London Avenue Canal in New Orleans this August was cheered by all Crescent City residents. But few will ever appreciate how quickly the project was completed—nine months from awarding a contract for the pumps to their commissioning.

A key part of the city’s flood-control system, the London Avenue Canal collects drainage and pumps it into Lake Pontchartrain. In August 2005, neighborhoods along the canal suffered some of the worst damage from Hurricane Katrina and the subsequent floods caused by levee failures.

On December 22, 2006, the U.S. Army Corps of Engineers (USACE) changed its flood and storm water control strategy by calling for the installation of high-capacity direct-drive pumps at London Avenue Canal, where Hurricane Katrina caused two major breaches. The problem was that the USACE said the new pumps were needed within nine months—by mid-August 2007, the beginning of hurricane season. At the time, several pump vendors were quoting delivery times of 18 to 24 months; they considered the timeframe impossible.

Within days of the USACE appraisal, Michael Pittman of M.R. Pittman Group LLC—a local general contractor specializing in infrastructure projects—was on the phone with Al Huber, president of Patterson Pump Co. (www.pattersonpumps.com), a subsidiary of Gorman-Rupp Co. Pittman impressed upon Huber the huge boost his pumps could give to the ongoing New Orleans recovery effort. Pittman also emphasized that the direct-drive pumps were the critical missing pieces of the proposed project; its foundation and surrounding infrastructure were already in place, thanks to the heroic efforts of the federal team responsible for recovery of the Greater New Orleans area.

Answering the call, Huber committed to manufacture, deliver, and test eight 68-inch x 72-inch semi-axial vertical flow (SAFV) Patterson pumps within 150 days. A contract was awarded on December 22, 2006. While Patterson was building and specially fitting the pumps, the recovery team was installing new surge gates, discharge pipes, and other foundational units of the pumping station to prepare for the pump deliveries.

Huber made good on his commitment by delivering the pumps to New Orleans on schedule. Lengthy dry tests conducted during the second week of August 2007 verified the pumps’ ability to run in water. After equally successful wet tests, performed the following week, all eight pumps were officially accepted by the USACE. The pumps were then delivered to the London Avenue Canal, where they were installed (Figure 4) and passed final tests.

4. Eight is enough, for now. One of the eight 68-inch x 72-inch SAFV mixed-flow pumps being installed at the London Avenue Canal in New Orleans. Courtesy: Patterson Pump Co.

Col. Jeffrey Bedey, commander of the USACE’s Hurricane Protection Office, recently reflected on Patterson Pump Co.’s contribution to the recovery effort: “On August 15, we completed testing of all eight pumps here at London Avenue Canal. It was an unbelievable effort by a team that made history. I’m proud to be part of the team, just as Patterson Pump should be proud of itself. They were there all along the way.”

Bedey summed up the project succinctly: “Between December 22 and August 15 we had the award of a contract, the delivery of eight pumps, and their installation and acceptance and final testing. I challenge the engineering community to find a better example of making things happen quickly.

“What we are doing here in New Orleans is about more than just building pumps. It’s about restoring the faith and confidence in the people. Not in the U.S. Army Corps of Engineers, but that inner faith and confidence such that people want to come back to this great community.”

As of September 2007, the population of metropolitan New Orleans was 70% of pre-Katrina levels—up from 50% in August 2006.

PLANT MANAGEMENT

Turn plant failures into successes

When a mishap occurs at your power plant, do you care to fully understand why? Some people don’t. Others care, but only to the extent required by regulations or corporate policy. Still others care passionately, and see immense value in understanding the root causes of failure.

Root-cause analysis is not just for catastrophic accidents; it’s for day-to-day problem-solving, too. A troublesome induced-draft boiler fan, for example, may require routine replacement of bearings. Your mechanic might be content to go on changing bearings every month, but if you conduct a root-cause analysis and find the bearing failures are due to an out-of-balance rotor, then you can fix the problem and stop wasting the mechanic’s time and your precious O&M money. This is what root-cause analysis is all about: getting to the heart of a problem. It’s not just a one-time exercise, but a way of thinking.

A lot of people don’t like to talk about plant failures. In fact, many people—usually those at the highest levels of management—try to cover them up. But by not talking about failures, by changing the focus, people miss the point. Failure is perhaps the single most valuable experience in anyone’s lifetime. In fact, root-cause specialists like to say that the secret to success is failure.

If you believe in root-cause analysis and in learning from mistakes, you probably already have an incident reporting program at your plant. You’ve also probably already learned that it can be tough to make the program accurate, impartial, and effective. A big impediment to accuracy is cost, particularly when trying to determine the cause of catastrophic equipment failures.

Three levels of investigation can be undertaken: (1) determining the mode of failure, (2) recognizing contributing causes, and (3) finding the true root cause. If the first level costs X, then the second level probably costs 3 to 5X. Reaching the third level, experts estimate, can cost 10 to 50X. Contributing to cost run-up are such potential land mines as undocumented repair work, inadequate knowledge of metallurgy—the processes of original equipment manufacturers (OEMs) and rework facilities are often proprietary—and secondary damage that masks the triggering event.

Other impediments to impartiality are commercial, social, and political goals. For example, investigators may skew the findings toward equipment design in order to recoup money from OEMs and insurance providers. Or they may point the finger at faulty instrumentation to avoid harming a power company’s reputation.

Even if incident reports rise above the bias and deliver accurate, impartial findings, they still may not be effective if the results are incorrectly applied. Results are often applied in either of two ways. At one end of the spectrum, written reports are simply circulated via intra-office mail and employees are directed to “read and initial” them. This approach, according to training specialists, fails to demonstrate management commitment and tells employees that the incident report holds little importance.

At the other extreme, conducting excessive instruction and ramming the results into employees’ heads could be equally ineffective. The nuclear-powered Navy, for example, laid its cornerstone on accurate, hard-hitting incident reports. But many veterans believe the program’s value was diminished by marathon training sessions and overzealous disciplinary action.

Training specialists say that the most effective way to apply incident reports lies somewhere between the two extremes, and that corporate culture plays a key role. The culture, they say, must encourage the following four types of staff behavior that lead to effective incident-reporting programs.

First step: Freeze!

Criminal investigators say that it is literally impossible for someone to walk into a room without leaving behind some evidence of their presence. Similarly, when a power plant mishap occurs, there is always some evidence. But the trail fades as time passes, so the evidence must be captured—or frozen—as soon after the incident as possible.

Consider this true story: An engineering team was called in to investigate an unexplained steam-turbine trip and its subsequent catastrophic failure, which sent flames leaping through the plant and metal flying into the control room. Two months into the investigation, the team was at a loss to explain the event.

Then, one day, a junior member of the team noticed that the turbine’s local emergency-stop button was pushed in, and jammed so it could not be released. The team then realized that the lone on-duty operator, who claimed he had not manually tripped the unit, was hiding the truth. He was petrified of being blamed for the destruction.

As it turned out, pushing the emergency-stop button was the right action, and the catastrophic failure was not the operator’s fault. After feeling a sudden eruption of horrendous vibrations, he had pushed the button, as he should have. Doing so was supposed to slam shut all steam-supply valves to the turbine and electrically disconnect the generator. But every one of the valves was rusted in its sleeve, and hung open. The generator disconnect, however, worked flawlessly, so the turbine failed from severe overspeed. That jammed emergency-stop button shows the importance of freezing evidence. When an incident occurs, note the positions of everything—every pushbutton, valve, indicator, person, piece of equipment, mark, or blemish.

Next, introduce ignorance

When searching for the cause of a failure, we often overlook the obvious. This is especially true for formally educated technologists—we relish the thought of digging into nitty-gritty details and exploring all kinds of complex possibilities. But what we should be doing is purposely introducing ignorance—asking the simple questions first. One failure specialist tells this story:

  I accompanied a fellow worker to one of our sulfuric acid plants to help rebuild one of its waste-heat boilers. Barney was much more experienced than I was, and had been coming to the site annually for a long time—these waste-heat boilers had been rebuilt once a year for the last 20 years because of sulfuric acid corrosion. I respected Barney greatly, as did all site personnel.

  The next morning, we met with an equally respected team of corporate metallurgists who had also been coming to the site annually, as well as the plant’s reliability engineering staff. I learned that once per year, these experts would get together to try to get more life out of the boiler tubes. They had tried just about every metal they could think of.

  I looked around the room and noted that while individual members of the group had changed from time to time, the collective team of experts had inherited all the folklore and tradition from their predecessors. Their biases were obvious to me, so I started asking some simple, straightforward questions. They answered my questions politely, but they were clearly annoyed.

  That night, I came right out with it and asked Barney, “What causes the boiler tubes to corrode?” He flashed me a disgusted look and said, “This is a sulfuric acid plant. Acid is made in the boilers. Boiler tubes and sulfuric acid don’t mix well. So don’t ask such a stupid question.”

  The next morning, we met with the plant manager. Still troubled, I blurted out, “I just don’t understand why anyone would design a boiler with tubes that will corrode in one year!” Barney could have shot me. But the plant manager was wide-eyed. “You know,” he said, “I’ve been wondering the same thing, but I didn’t want to sound stupid. Our other sulfuric acid plant isn’t having these problems. Sure, we begin the acid-making process in the boilers, but there’s not supposed to be acid until after those tubes.”

  Encouraged by this exchange, I called my boss at the home office and suggested that we conduct a full-blown root-cause analysis. He liked the idea so much he flew to the plant site to personally propose the project. When he was explaining the proposal, my boss said that “only the truly ignorant can ask the questions that lead to discovery because they don’t know enough to be ashamed of themselves.” Then, he looked at me and said, “We’ve got one of the most ignorant people imaginable right here!”

  The proposal was accepted, and the study pinpointed as the cause of failure a dew point problem, not a metallurgical one. When we verified our hypotheses, we moved the affected tubes one foot behind their original location. It worked! In fact, we eventually changed the tube materials back to their original, less-expensive specification.

The team of experts had looked at this problem for 20 years! They were brilliant in their field, but they made two of the most common mistakes in problem-solving. First, they tried to solve the problem before they looked at the evidence (some have described this as a medical problem called “premature evaluation”). Second, they failed to introduce ignorance. To help break what often can be a thin shell of understanding, you must intentionally pit ignorance against expertise.

Third, grow a “why tree”

The root causes of all industrial incidents lie in the little things that present themselves on a day-to-day, moment-by-moment basis. To uncover them, we must sincerely ask, Why?

Experts recommend using a unique device called the “why tree.” It’s similar to the fault tree used in design industries, but it’s much easier to use. What you must remember, however, is that the why tree is not probabilistic in nature, and therefore does not predict potential failure rates. Instead, it uses evidence, along with ignorance and expertise, to dig logically into an actual problem.

When incidents happen in our physical world, we can always find a physical problem. But all physical problems are caused by people who do something inappropriate. In other words, people are triggering mechanisms. Unfortunately, the usual reaction is to find the person who triggered the problem and blame him or her. Our eagerness to “point the finger” and discipline the scoundrel implies that the cause lies with people—not what’s in their minds, not the way they make decisions, but the physical person.

Granted, sometimes discipline makes sense. But only when we define specifically what we want it to accomplish. Our intentions should be precise, fair, and appropriate. More important, we must continue down the why tree and understand why people did what they did. The following is a good test of when to stop asking why: “Do I understand the causes well enough that I’m convinced I would do the same thing under the same circumstances?” When you can answer “yes” to that question, you’re at the point where you can identify the root causes and make lasting, constructive improvements.

Make the findings visible

When a root-cause investigation is finally complete, the team lets out a collective sigh of relief. “Finally, we have cracked the shell. We know what caused the failure. Victory is ours.”

“BILGEWATER,” as Marmaduke Surfaceblow might say.

Who cares if the investigating team understands the causes of failure, if nobody else understands? Who cares if the principal investigator has been enlightened, but not the plant manager? What is the point of root-cause analysis if every single individual in the company does not learn the lessons? Read the sage advice of a failure specialist:

  Several years ago, I was investigating a large electrical system failure at an industrial facility. The entire complex had lost power for many hours. When I arrived on the scene, I felt almost overwhelmed by the complexity of the electrical system and its cascading series of failures—I was an electrical novice. Our team eventually completed its work and produced a final report. I knew what was in it because I was part of the team. But I didn’t write it. In fact, I couldn’t have written it because it was too complicated. The report was five inches thick—page after page of detailed, technical writing.

  Because I was contractually bound to contribute my own narrative, I decided to write a different report—one that people could read. I limited it to a few pages, and, because most people prefer looking at visual images, I included numerous drawings. I even hired an artist, which meant I had to translate the causes of a complex, technical catastrophe to a person trained only in fine arts.

  At times, I found myself talking to the artist as I would to a 10-year-old. But in the process, I found areas that even I didn’t fully understand. I realized that I was falling into the same trap as other so-called experts—when you stripped away the technical jargon, there was little substance to what I was saying. It was then that I realized the importance of translating a study’s findings. When my final report was ready, the artist fully understood the causes of the multimillion-dollar, technological failure. So did others with no technical training—including my secretary and my wife.

This specialist’s final report was only 30 pages—15 of text and 15 of supporting graphics. Most of the graphics were cartoons that exaggerated the phenomenon being explained. The ability to exaggerate added immensely to the report’s clarity—it was a way of “raising the volume” to get the reader’s attention.

—POWER editors