Tag Archives: generator

More From Minter Hatchery Generator Failure Report, Discipline For 3 Staffers

Lord knows that someone who has difficulties running a toaster in the morning probably isn’t qualified to weigh in on matters such as restarting a backup generator at a state salmon hatchery.

But the root cause analysis for the failure of one at WDFW’s Minter Creek facility last winter was released yesterday and it makes for some aggravating reading.

A PAGE FROM THE 2018 OPERATIONS MANUAL FOR THE MINTER CREEK HATCHERY BACKUP GENERATOR STATES IT “SHOULD BE RUN BI-WEEKLY TO EXERCISE IT AND TO ENSURE EVERYTHING IS WORKING AS IT SHOULD.” (WDFW)

We’ve reported on this before here, here and here, but to review, the backup power source wouldn’t start after 45-plus-mile-an-hour gusts cut electricity to the South Sound hatchery around 5:30 p.m. on Dec. 14, 2018, essentially giving millions of Chinook fry there just 20 to 30 minutes to live before the lack of dissolved-oxygen-rich fresh water running through their incubation trays led to their death.

Crews scrambled to fix the generator, but it was out for around three and a half hours and was an entirely preventable disaster if it had been checked as routinely as the operating manual’s recommendation — every two weeks.

As a result of the investigation, three WDFW staffers described as “Minter Creek employees” have now been disciplined.

“One employee was demoted; another was suspended; and the third received a written reprimand,” said agency spokeswoman Michelle Dunlop late Monday afternoon.

She said that the punishments were based on the level of responsibility of the individuals.

Dunlop said the report, which is marked confidential and appears to have been finalized in March, was not released until the disciplinary process had been completed.

THE 97-PAGE-LONG DOCUMENT SHOWS that the last time the 22-year-old, 350 kW generator had been tested before the outage was Oct. 8, near the start of the fall storm season and 67 days beforehand.

The manufacturer’s operations manual “specifies bi-weekly testing of the generator system to maintain readiness,” WDFW-contracted independent investigators Ron Carper and Frank Sebastian write.

THE BACKUP GENERATOR’S CONTROL PANEL. (WDFW)

Prior to the October test, it had been 70 days since one was performed in July, their report shows.

They found that in the 21 months leading up to the disaster, the generator was only tested 12 times, on average once every 54 days, though with 16- and 17-day intervals in early 2018 too.

But it apparently wasn’t tested at all during one 195-day period — more than half a year — between early March and mid-September 2017.

Comments associated with that September test say “Tree on line (wind).”

As you can imagine, the frequency of testing went up following December’s disaster.

“After 12/14/2018, the average days between generator runs was 5.2 days,” their report states.

“Although hatchery staff responded promptly when the incident occurred and worked tirelessly to save fish at the hatchery, there were missteps along the way,” WDFW Director Kelly Susewind said in a press release yesterday that accompanied the report. “We’ve learned a lot from this and are addressing shortcomings identified by the contractors.”

THE REPORT ALSO INCLUDES INTERVIEWS and details how the two staffers stationed at Minter as well as another from a nearby facility who responded that evening as power outage alarms went off worked their butts off to try and get the generator going again.

As one drove over to the Purdy Napa to get new batteries and cables, another worked to get water flowing other ways to the trays holding young salmon, which succeeded in one area.

As the news also traveled up the chain of command, two firemen even arrived with a gas-powered pump but “did not have any hoses for the pump which would connect to the fittings at the hatchery,” and while it wasn’t their job to know, nor could they figure out why the generator wouldn’t fire up.

The report states that the initial failure to start was most likely caused by “a loose or cracked starting battery connection,” but there was only so much anyone could have done to get the generator going again after the “Emergency Stop” was activated to replace the batteries and cables.

That’s because a device known as an “air damper” first had to be reset.

The device is a rare one, with no generators elsewhere in the state hatchery system having one, according to the report.

“The generator would not start and displayed an ‘Air Damper’ fault light. [Name blacked out but a hatchery specialist 3] did not know the significance of this fault or how to correct it,” the report says.

It was only after a second person from the state’s Capital Assets Management Program in Lacey, a diesel mechanic, was apprised of the situation by phone at around 8 p.m. that they realized that that needed to be done and how to do it.

“These instructions resulted in the generator system becoming operational and suppling (sic) electrical power to the Minter Creek Hatchery,” the report states.

It’s believed that the delay in figuring out the air damper issue didn’t result in the loss of more fish.

But it is emblematic of wider communications failures at the facility, beginning with what to do in that critical first half hour.

The report states:

“The employees at Minter Creek Hatchery did not have a pre-established plan or instructions on how to maintain water flow and overcome the failure of electrical power in sustaining fish viability. The employees realized maintaining oxygenation in the water flow was critical and initially worked at various unguided solutions. This included activation and utilization of non-electric powered pumps and gravity flow of water.”

It also says:

“The employees present at Minter Creek Hatchery did not have the knowledge, training or reference materials on how to fully reset the Emergency Stop control, following activation of the Emergency Stop control.

“The employees present at Minter Creek Hatchery did not have knowledge, training or reference materials on how to reset the Air Damper fault indication, following activation of the Emergency Stop control.”

A “Considerations for the future” section of the report lays out an improved set of responsibilities for the facility manager and their hatchery specialists.

About half an hour after the generator was started, local utility crews restored the electricity.

Several days later the root cause analysis was contracted.

The report states that the backup power source was serviced and tested in March 2018 and found to be in “good overall condition,” though it apparently had been at least six years since the starter batteries had been replaced.

Batteries are more likely to fail the older they get.

Carper’s and Sebastian’s conclusion states:

The best explanation of the root cause generator failure is the result from a loose or cracked starting battery connection. Due to potential for corrosion, vibration, and what can be fairly significant temperature cycles, generator manufacturers typically recommend maintenance procedures for regular inspections of all wiring. Battery cable connections may become loose or damaged and should be inspected at every
generator exercise cycle by gently tugging and wiggling the cables.

It is inherent that electrical terminations on hot lead battery posts that are on vibrating engine frames will tend to loosen over time, so regular checks are recommended.

According to the generator log book, it had not been run for 67 days prior to the failure. With more frequent testing and inspections, this issue with the battery cable could have been discovered and repaired under normal conditions.

WDFW initially estimated that up to 6.2 million Chinook fry might have died without fresh water circulating through their rearing trays, but have now revised that downwards to 4.1 million following “a more robust inventory.”

More than half have been replaced with “excess” fish from six state, tribal — including 750,000 from the Suquamish and Nisqually Tribes — and tech college hatcheries.

The facility also raises chum and coho salmon for state and tribal fisheries. Those fish return and spawn later in the year than Chinook, and their eggs were still in a stage where they could survive if the water was drained off but were still kept moist.

A SCHEMATIC DRAWING SHOWS THE LAYOUT OF MINTER CREEK HATCHERY. (WDFW)

AGAIN, I’M NOT TRYING TO COME OFF AS SOME EXPERT in these matters and I readily acknowledge that s*** is just gonna happen at the worst possible times.

Been there, done that.

I do appreciate that WDFW apparently published the report without being prompted to get ahead of some reporter’s public disclosure request.

It harkens back to agency leaders’ vow before state lawmakers last winter to “to hold ourselves accountable for the tragic loss of the fish.”

WDFW DIRECTOR KELLY SUSEWIND, FISH PROGRAM DEPUTY ASSISTANT DIRECTOR KELLY CUNNINGHAM AND HATCHERY DIVISION MANAGER ERIC KINNE SPEAK BEFORE A STATE SENATE COMMITTEE ON WHAT HAPPENED AT MINTER CREEK. (TVW)

“We’re committed to making improvements to safeguard against another incident like the one at Minter Creek,” Kelly Cunningham, WDFW acting assistant Fish Program director said in the news release.

But even as the problem is being addressed, the report reveals an operations negligence that I find unacceptable.

That the two-decade-old generator wasn’t tested more frequently or ahead of the National Weather Service’s forecasted high wind warning that day, which put the fish at risk, boggles my mind.

That there wasn’t an established backup plan to the backup power source is puzzling.

As wild salmon continue to struggle and their recovery is decades — centuries? — down the road, the future of our seasons and starving orcas depend ever more highly on producing fish.

Minter is a key hatchery, putting out Chinook and chum, which southern resident killer whales depend on year-round and in fall, respectively.

Again, WDFW is obviously working on improving their operation, but they need to do better, and show us they are.

This was an entirely preventable, manmade disaster that didn’t need to happen if regular testing had been undertaken and it wasn’t assumed that the generator would just fire up whenever needed.

A harsh lesson, but one that’s been learned hopefully.

Anglers Unhappy With Another Salmon Hatchery Disaster

Washington salmon fishermen and others were furious to learn that as many as 6.2 million Chinook fry died last Friday when a state hatchery lost power during a windstorm and a backup generator failed to start up.

CHINOOK FRY. (C. ANDERSON, USFWS)

Wow. So stupid Doesn’t anyone test their generators?” posted guide Brian Oldfield in the comments below a link to our website and a WDFW news release that came out very late yesterday afternoon.

The words that are coming out of my mouth right now …,” fellow guide Mark Ervig posted.

There goes Area 13 again. Sounds like some hatchery management adjustments are in order. This is unacceptable, Director Susewind,” wrote Dave Wright.

Will there be consequences for not being prepared? Nah. Just to the salmon,” posited Michael Rodriguez.

Seems these things happen quite often. I’d like to see a total number of lost fish such instances over the last five years for each West Coast state. I’d read that story,” added Chad Price.

OK,” posted longtime Washington hook-and-bullet-world observer Dave Workman. “The department has 24 hours to replace the fish. How’s that for a solution?”

REACTION TO NEWS THAT MILLIONS OF CHINOOK HAD BEEN LOST DUE TO A POWER OUTAGE AND GENERATOR FAILURE AT A WDFW HATCHERY WAS NEGATIVE.

WDFW is in fact searching for replacement salmon for those that were being reared at Minter Creek Hatchery, at the north end of Carr Inlet near Purdy, according to agency hatchery manager Eric Kinne.

“Right now we’re looking at other facilities, even tribal facilities, for excess capacity,” he said this afternoon.

Kinne says that come next May and June, some level of release will occur, possibly with Chinook from as far away as the state’s Samish Hatchery, where a million-plus extra eggs were collected this fall primarily to help feed southern resident killer whales.

That could help backfill the “dent” Deep South Sound might otherwise see during 2022’s, 2023’s and 2024’s fisheries, not to mention the availability of the salmon for orcas.

But in the meanwhile, Kinne wasn’t ducking questions about what he termed a “devastating loss” in Monday’s press release, the latest in a recent series of disasters and mishaps that have struck WDFW hatcheries, which provide most of the fuel for state salmon and steelhead fisheries.

“It’s tough to explain this one,” he acknowledges. “I understand it for sure. We’re not happy either.”

Kinne says he spent part of today writing up a contract for a “root cause analysis” to get to the bottom of the failure.

He says that the hatchery’s backup 350 kVA diesel generator is tested monthly and its last annual service was 10 months ago.

When Friday’s high winds cut the power to the facility around 5:30 last Friday evening, the generator should have automatically turned on, keeping cool water flowing into 900 trays with 6,800 tiny Chinook each.

But it didn’t.

SALMON INCUBATION TRAYS IN ONE OF EIGHT IDENTICAL ROOMS AT MINTER CREEK HATCHERY WHERE CHINOOK, COHO AND CHUM EGGS HATCH. ADULT CHINOOK RETURN EARLIER IN THE YEAR AND THEIR EGGS HAD BECOME FRY, MAKING THEM MORE VULNERABLE TO A LOSS OF CIRCULATING WATER THAN THE EGGS OF LATER RETURNING COHO AND CHUMS. (WDFW)

A WDFW staffer is at Minter around the clock and so when the motor didn’t start, they began working on it as well as calling nearby workers back in for help.

Kinne, who lives an hour and a half away in Rochester and actually worked at the facility for several years two decades ago, received a call and made his way north through the storm to help deal with the situation.

“There were branches all over the roads,” he says.

While some worked on the generator, others checked rearing ponds, which are fed by gravity.

That’s not the case for the water pouring through the Chinook trays, however. Spawned in late August and September, the fish are now about an inch long and while they still have their egg sacs, they need running water to bring oxygen to their gills.

Crews attempted to use gas-powered pumps to get creek water into the building and the trays but were largely unsuccessful, though they had better luck with later-arriving salmon stocks.

“We saved all of the coho that were in incubators, the chums that were in the incubators,” Kinne says.

The eggs of both species can survive for several hours if the water is drained away but the eggs are still kept moist, he says.

“Some the coho were outside in the rearing ponds and were not affected,” he adds.

Crews were ultimately able to start the generator after nearly three hours — right before the local public utility district finally got the power back on.

Kinne says the outage occurred at the “most inopportune time” because it came with the “highest number” of young salmon in incubation.

The majority of trays that held the Chinook are being poured into a holding pond and the true magnitude of the loss will become clearer when any survivors reach the “swim up” stage between now and May and June, when they were otherwise scheduled for release.

Kinne said the 5.7 million falls believed lost represents the “majority” of production for release into the lower Deschutes River and from Minter Creek for sport fisheries.

When you include the half million White River spring kings fry that also died and were being reared for orcas, the 6.2 million lost Chinook is roughly 9 percent of WDFW’s overall statewide production, and certainly a higher percentage of Puget Sound’s.

It wasn’t immediately clear what sort of dollar loss it all represented.

“Not a lot spent to date because they were still in incubation,” Kinne says. “The majority of the cost comes for feeding and marking and tagging.”

The bigger loss could be down the road when the returning adults would have attracted angler effort.

Deep South Sound, or Marine Area 13 as it is also known, has become a good spot to target Chinook, especially in Budd Inlet’s Big and Little Fish Traps for Deschutes-bound kings.

Even as the cost of redundancy for backups to backup systems would be high considering the 80-odd hatcheries WDFW operates across the state, the predictable nature of fall, winter and early spring windstorms, and the increasing importance of salmon to fisheries and orcas makes testing generators at the facilities much more often a good idea.