Long-time POWER readers may remember Marmaduke Surfaceblow, a fictional character whose engineering escapades were brilliantly portrayed in hundreds of stories published within POWER magazine’s pages over more than 30 years beginning in 1948. Today, the fictional series continues through Marmy’s granddaughter, Marnie, who is an engineering wiz in her own right.
________
Maya and Marnie troubleshoot air-cooled condenser performance issues at a New Mexico power plant. Bad sensor data contributes to poor decisions, making the problem more complex.
“Air-cooled condensers [ACCs] are amazing, Maya. You only find them at 1% of power plants, and each one has a unique design for the installation, but all of them are big. This one covers more surface area than the rest of the power plant!” exclaimed Marnie Surfaceblow, vice president of Surfaceblow & Associates International.
Nodding in agreement, Maya Sharma, Marnie’s lead field engineer, replied, “I have seen them described in technical papers, but until now I did not understand their magnitude. Ma’am, does this ACC resemble others you have seen?”
“This ACC is unusual,” Marnie noted. “The site here in New Mexico is 3,650 feet above sea level, so on average the air is about 13% less dense. Normally that might require an even larger ACC to get the heat transfer they need, but instead, they installed two banks of variable-frequency-drive fans—25 ‘push’ fans in the inlet duct and 25 ‘pull’ fans in the outlet duct. It’s an interesting concept, and yet, … ah, there’s our host waving at us, so the mystery of the uncanny ACC must be left unsolved—for now.”
In This Issue
Full issueMeeting the Gang
The first part of the morning meeting focused on site safety and project goals, then went down a rabbit hole as plant engineers tried to fill in the numerous gaps in the plant performance data revealed by Maya’s initial survey. “At this juncture,” noted Maya, “your ACC has almost twice the measured data points of the entire power station.”
Santiago Martinez, the chief engineer of the Pecos Power Plant, nodded agreement. “The plant was built in 1944 to burn the local coal and help supply power for the atomic weapons program. Retrofitting sensors and wiring is complicated and expensive because we have a lot of hazardous materials in the plant. Lead paint, asbestos insulation, mercury manometers—shoot, half the clocks in the plant used to have radium-painted faces.”
“Didn’t you do any upgrades when you converted from coal to pulp and paper waste?” asked Marnie.
Shaking his head, Santiago answered, “Not many to speak of. Having a vibrating grate stoker furnace gives us a lot of fuel flexibility, so most of our modifications were to the fuel unloading, conveyors, and boiler fuel feed systems.”
It was Marnie’s turn to shake her head. “If you’re that strapped for cash, then help me wrap my mind around why so much money was spent on a fancy new ACC,” she said. “Then, there’s the plant heat rate consideration. Your stoker is a very old and simple design, and given that your furnace isn’t the most efficient, you’re burning a fuel with a large amount of latent heat loss, and let’s not forget the fact your turbine is so old that for its first 15 years of operation, there were only 48 states.”
Plant Manager Sheila Bell cleared her throat and replied, “Lots of people ask that same question. The plant was designed to provide reliable power for the Manhattan Project, so it was designed with a lot of redundancy. A preventive maintenance culture was established from startup. We operate well below our design, and our cycling is gradual and planned so far in advance that you can set your watch to our schedule, not the other way around.”
Pecos Power’s plant accountant, Arturo Milagro, chimed in. “About 10 years ago the local landfill told the paper mill they couldn’t accept their waste. The next-closest landfill is an 80-mile round-trip. So, one day the mill owners asked if we were willing to do a swap.”
“I see,” said Maya. “They provide you with fuel, and you provide them with process steam.”
“Correct. We get free fuel, and they take up to 400,000 lbm/hr of steam from our intermediate-pressure turbine,” Arturo explained. After a short pause, he asked the engineers in the room, “How many equivalent megawatts is that?”
“About 5 to 5.5 MW, depending on your turbine heat rate,” replied Marnie with unearthly speed. “Of course, we aren’t sure what your turbine heat rate is, which is part of the problem.”
ACC Performance Issues
“So, we now come to the ACC,” Santiago pronounced. “I assume you’ve seen the Pecos River by the plant border. We once used that water for heat rejection. The plant was built with a wet mechanical draft cooling tower. Then, in the 1980s, we moved to a semi-dry system due to concerns over water use. Then, regulations came, and the regulators had no mercy for us. ‘Use dry cooling or shut down,’ they demanded.”
“And we almost did shut down,” said Sheila. “But Arturo ran the numbers hundreds of times …”
“Thousands,” muttered Arturo.
“Thousands of times,” Sheila continued, “and it looked like we’d have a 10-year payback period. But here we are—barely past two years later—and our plant is hurting.”
“I found something in your records that looks strange. I don’t want to be suspicious, but … it’s my nature,” Marnie said. “The ACC was released for commissioning testing in September, but testing didn’t start until December. Was that so the manufacturer could take advantage of the cooler air to help them pass the tests?”
“That would make sense,” Sheila said with a laugh, “but there was a site subsidence concern. There are old mine shafts and natural caverns all around here, and the geotechnical folks missed that. We field-modified the inlet and outlet cooling air ductwork to reduce steel weight. The cooling air outlet ductwork went from 40 feet to 20 feet, and the inlet air ductwork went from 30 feet to just 10 feet.”
Marnie and Maya shared a look that communicated a mutual “Aha!” Marnie continued the discussion while flipping through a printout of the test report. “The report says the ACC passed commissioning on the first test run that December, but by late May the following year, your backpressure started rising and stayed well above design the whole summer and fall. It improved in the winter, but then by spring it got worse?”
“We had our first forced outage in years,” affirmed Santiago. “Turns out the heat transfer surfaces were coated with fluff from the cottonwood trees along the river. We used high-pressure air lances to blow it clean, and we recovered most of our pressure loss. Yet, barely five months later, our backpressure was borderline right as we entered winter.”
“And this March you experienced your second unplanned outage,” Maya noted.
Santiago asked Vasquez Toya, the heat rejection system lead, to discuss the outage. “Right. There was some residual cottonwood fluff—it’s hard to get everything clean, the ACC is cramped, and with no cooling air flowing, it can turn into an oven in the sun. We limit inspection and cleaning time to just two hours a day per person for health reasons. Anyhow, instead of cottonwood fluff, there was sand and dust all over the tubes. We blew it out again with air lances, but it was a bit harder to clean this time.”
“Is your online air cleaning system operational?” asked Maya.
“Yeah, but it just doesn’t seem to work well in practice,” replied Santiago. “We ran them on continuous cycle mode when we had the cottonwood fluff problem, and we ended up eroding some tubes halfway through the metal by sandblasting them.”
Day Operator Cory Flintlock added, “We’ve also locked out the push fans because we heard something that sounds like bearing rattle, and our vibration monitors were giving us strange readings. The pull fans alone can get us about three-quarters the air flow we need, and that’s alright until the ambient temperature breaks 90F.”
“Those bearings are rated for 20 years. I can’t believe they’re failing after barely more than two,” said Santiago with resignation. “After that outage our backpressure was almost at design. Then, that first hot day came and the system stalled out,” continued Santiago. “It’s funny. We couldn’t get the air flow or pressure rise we needed, our fan power was less than normal, and our temperature probe readings were all over the map. We limped along for a week. Then, shut down last Friday and called you folks.”
“Is it safe to assume the paper mill had a penalty in your contract based on the number and length of unplanned outages?” asked Marnie.
Sheila put her face in her hands briefly, then replied, “The mill has natural gas boilers to meet their steam requirements. Gas prices are high because the main pipeline is badly constrained. We can be offline for seven days. After that, we’re responsible for their fuel gas charges for up to two months. We called you on day three, and today is day six.”
Checking her phone, Marnie’s face became grim. “In other words, we have about 12 hours to get you back online,” she presumed.
Data Problems
“At least we sent you data ahead of time so you could be ready as soon as you got here,” Santiago said, but he didn’t sound hopeful. “Unfortunately, I heard some of the data might be bad.”
Trying to sound positive (but failing), Maya replied, “Sir, much of your data has problems. Of 7,511 tags in the distributed control system [DCS], 3,944 are missing, bad, incorrect, uncalibrated, or displaying strange or even random values. An example is the flue gas temperature exiting your induced draft fan, a critical control parameter for the ACC, which has been stored as values between 0 and 150 ppm.”
Trying to lighten the mood (and failing), Marnie added, “Did you know your hot reheat steam pressure is about 25 psi higher than your cold reheat steam pressure? Yes, the steam flows backwards to heat your flue gas.”
Jumping in to prevent her boss from making light of more DCS problems, Maya quickly said, “I noticed alarming vibration trends for three of your push fans, sir, but I do not believe they are real. I ran many statistical analyses on the data and discovered there was no correlation to anything. Hence, I believe they are random noise from a sensor that has either failed or is badly calibrated.”
The thought that they had been making operations decisions based on random data alarmed several in the conference room, who all started talking at once. Marnie and Maya ran over to huddle with Sheila and Santiago. Then, as a team, they called the room to order.
“Who did you hire to help you?” asked Marnie. “Maya and me, that’s who. Everyone remember Pulp Fiction? OK … about half the room. Remember Winston Wolf, the fixer, portrayed by Harvey Keitel? Maya and I are Winston Wolves. We solve problems, and we work for you. We have a plan, and we’ll stay here today, tonight, and beyond if we must. Our goal is getting you online by midnight and keeping you online for at least one full day. Who’s with us!?”
Much Revealed in ACC Inspection
Marnie walked the length and breadth of the ACC with Santiago as her escort. Taking numerous photos as they walked, she used her black Sharpie quite often to highlight bad gaskets, loose seals, and occasional small holes. A reflection caught her eye, which was found to be a large area on the inlet ductwork that was seemingly stripped of paint and abraded to base metal. Stepping back to survey the entire cooling air system, she thought, “Wow … that’s a really short flow path!” As she approached the entrance to the pull fans, she saw two open inspection hatches near the bottom of the ductwork.
“Santiago, can we close these hatches? If nothing else, they’re a confined space hazard,” she noted.
 |
|
1. Workers try to shut an open hatch, but find it too corroded to get fully closed. Source: POWER |
Santiago called two of the maintenance crew over. Despite several minutes of effort, only one door could be closed, with the other seemingly rusted permanently open (Figure 1). Upon questioning the staff, Marnie learned the hatches were left open to allow for potential online cleaning with the air lances.
Leaving the maintenance crew to work on the doors, Marnie asked, “Santiago, can you have the operators turn the fans on and off for me please?”
Santiago called the control room, then passed his phone to Marnie. After giving the operators instructions and the crew performing the appropriate safety checks, the operators ran both fan banks through cycles from idle to full capacity, and with different numbers of fans in service. As the fans approached full capacity, a noise was indeed heard coming from the push fans.
“Hear those bearings?” yelled Santiago over the din. Marnie gave no answer, instead walking to the inlet to the pull-fan bank and clipping her safety harness to a hardpoint to resist the tug of the air flow. Producing a green laser strobe light from somewhere in her purse, she signed to Santiago to take video as she played the laser over the outer opening to the cooling air ductwork, and then inside over the fan banks themselves.
 |
|
2. Marnie finds sand and small rocks inside a pull-fan bank inspection hatch. Source: POWER |
A minute or so later, Marnie thanked the operators, then replied to Santiago’s earlier query as the fan noise quieted. “That sound isn’t bad bearings, but it is bad,” she said. After confirming the fans were locked out, Marnie reached inside a small inspection hatch just upstream of the pull-fan bank and removed a handful of sand and small rocks (Figure 2). “That’s the source of your noise—most of it, anyhow.”
“Oh, good,” replied Santiago with relief. “There’s no vibration problem.”
“Oh, I didn’t say that,” Marnie replied coyly, “but let’s see what Maya was able to discover.”
Fixing ‘Crazy Data’
Maya had managed to find the technical manuals for the vibration monitors. Then, using an optical character recognition tool and translation software, she converted them from Mandarin to Hindi. With her confidence renewed, she was able to successfully calibrate the instruments, which ceased the semi-random number generation.
Her next task focused on weeding out thousands of dead, obsolete, duplicate, and phantom variables from the plant historian. Her efforts made it easier to assess the accuracy of the remaining thermocouples, manometers, ammeters, and other sensors. Yet, her assessment was not encouraging.
“I see,” said Marnie, sitting down beside her after washing up from the plant inspection. “You’re asking yourself: ‘Why is the crazy data crazy? Is it nature in the form of a faulty sensor or nurture in the form of being exposed to crazy data?’ ”
“In a general manner, ma’am,” Maya responded. “I discovered the readings for temperature and pressure at each fan inlet and exit are highly erratic and not correlated with ambient temperature or humidity.”
“Do you know why?” asked Marnie.
“I have strong suspicions, ma’am, but to verify my assumptions requires inspecting the ACC as you have,” Maya said.
Marnie smiled. “Proving, again, we make a great team,” she said. “Thank you for debugging the fan vibration monitors. I’ve done that once, and if I had to do it twice, I’d quit engineering and join an Amish community.”
“Black really is your color, ma’am,” Maya said slyly. “I’ve emailed the team so we can present our findings, but … do we know all our findings yet?”
“We’ll talk the problem out as we go,” Marnie replied. “My tools are logical leaps, engineering expositions, and science savvy.”
Maya added, “And carafes of coffee.”
“That too,” her boss agreed.
The Big Reveal
Maya called the conference room to order and began the meeting. “The first task was to re-calibrate the fan block vibration monitors, as severe fan oscillations can be catastrophic. Upon doing that, small vibrations were detected on nearly all fans in the push- and pull-fan blocks. None were urgent, but all were undesirable,” she said.
Marnie picked up the story. “The fan vibrations are caused by the same agent fouling your heat transfer surfaces—the cottonwood fluff. The Fremont cottonwood trees along the river produce catkins each spring with especially sticky sap. Measures like air lances will help at first, but eventually the sap will bake onto hot equipment. Your ACC heat exchanger will steadily foul and lose efficiency until you come offline long enough for a deep cleaning,” she said.
“While your fans have sap buildup on them that created mild vibration, this was unknown until today,” said Maya. “Your fan vibration monitors gave continuous false positives, and Marnie informed me of excessive sand and debris in the fan housings, creating a noise similar to bearing failure. Thus, you turned off the push-fan blocks, removing almost half of your air flow capability.”
“That all makes sense, given how the cottonwood fluff has impacted your plant,” Marnie noted, pausing to pick her next words carefully. “But one critical mistake was actually behind nearly all of the problems. Siting the ACC on unstable ground mandated shortening the inlet and outlet cooling air ducts, allowing significant hot air recirculation into the inlet duct. This recirculation was so severe that sand and other debris have been swept into the inlet duct, even eroding one edge of it smooth. Sand accumulated in the bottom of the push-fan housing, causing that dreadful racket whenever the push-fan block was used.”
Shaking her head in disbelief, Marnie continued, “We may never know why someone left two large inspection hatches open just upstream of the pull-fan block. Perhaps they had a revelation in a dream that this would improve the plight of the overworked pull-fan block. Ironically, it made everything much worse, including the hot-air recirculation, which now had an even shorter circuit back to the pull fans.”
The room was abuzz as the plant staff soaked in what Marnie and Maya had found. Santiago was the first to pose a question, and his was straight to the point. “I’m stunned at both the complexity and simplicity of the problem, and how quickly you found it. But tell us—how can we return to full generation, safely, but as fast as possible?”
Maya quickly responded with a sound plan. “First, try finding a weak solvent to wash the tree sap from the fan blades in your push block, as it is likely more affected being closest to the outside air. Try also removing some from the first 10% of the heat exchanger. Secure those open inspection hatches and start the plant during the coldest time of the night. Tomorrow, survey the area carefully and see if some baffling or a guide may be made from aluminium sheet to divert hot air exiting the pull-fan block so less is recirculated.”
“Temporary solutions for now, but we’re happy to discuss permanent fixes that will last as long as your plant does,” Marnie concluded.
—Una Nowling, PE is an adjunct professor of mechanical engineering at the University of Missouri-Kansas City.
