Lord knows that someone who has difficulties running a toaster in the morning probably isn’t qualified to weigh in on matters such as restarting a backup generator at a state salmon hatchery.
But the root cause analysis for the failure of one at WDFW’s Minter Creek facility last winter was released yesterday and it makes for some aggravating reading.
A PAGE FROM THE 2018 OPERATIONS MANUAL FOR THE MINTER CREEK HATCHERY BACKUP GENERATOR STATES IT “SHOULD BE RUN BI-WEEKLY TO EXERCISE IT AND TO ENSURE EVERYTHING IS WORKING AS IT SHOULD.” (WDFW)
We’ve reported on this before here, here and here, but to review, the backup power source wouldn’t start after 45-plus-mile-an-hour gusts cut electricity to the South Sound hatchery around 5:30 p.m. on Dec. 14, 2018, essentially giving millions of Chinook fry there just 20 to 30 minutes to live before the lack of dissolved-oxygen-rich fresh water running through their incubation trays led to their death.
Crews scrambled to fix the generator, but it was out for around three and a half hours and was an entirely preventable disaster if it had been checked as routinely as the operating manual’s recommendation — every two weeks.
As a result of the investigation, three WDFW staffers described as “Minter Creek employees” have now been disciplined.
“One employee was demoted; another was suspended; and the third received a written reprimand,” said agency spokeswoman Michelle Dunlop late Monday afternoon.
She said that the punishments were based on the level of responsibility of the individuals.
Dunlop said the report, which is marked confidential and appears to have been finalized in March, was not released until the disciplinary process had been completed.
THE 97-PAGE-LONG DOCUMENT SHOWS that the last time the 22-year-old, 350 kW generator had been tested before the outage was Oct. 8, near the start of the fall storm season and 67 days beforehand.
The manufacturer’s operations manual “specifies bi-weekly testing of the generator system to maintain readiness,” WDFW-contracted independent investigators Ron Carper and Frank Sebastian write.
THE BACKUP GENERATOR’S CONTROL PANEL. (WDFW)
Prior to the October test, it had been 70 days since one was performed in July, their report shows.
They found that in the 21 months leading up to the disaster, the generator was only tested 12 times, on average once every 54 days, though with 16- and 17-day intervals in early 2018 too.
But it apparently wasn’t tested at all during one 195-day period — more than half a year — between early March and mid-September 2017.
Comments associated with that September test say “Tree on line (wind).”
As you can imagine, the frequency of testing went up following December’s disaster.
“After 12/14/2018, the average days between generator runs was 5.2 days,” their report states.
“Although hatchery staff responded promptly when the incident occurred and worked tirelessly to save fish at the hatchery, there were missteps along the way,” WDFW Director Kelly Susewind said in a press release yesterday that accompanied the report. “We’ve learned a lot from this and are addressing shortcomings identified by the contractors.”
THE REPORT ALSO INCLUDES INTERVIEWS and details how the two staffers stationed at Minter as well as another from a nearby facility who responded that evening as power outage alarms went off worked their butts off to try and get the generator going again.
As one drove over to the Purdy Napa to get new batteries and cables, another worked to get water flowing other ways to the trays holding young salmon, which succeeded in one area.
As the news also traveled up the chain of command, two firemen even arrived with a gas-powered pump but “did not have any hoses for the pump which would connect to the fittings at the hatchery,” and while it wasn’t their job to know, nor could they figure out why the generator wouldn’t fire up.
The report states that the initial failure to start was most likely caused by “a loose or cracked starting battery connection,” but there was only so much anyone could have done to get the generator going again after the “Emergency Stop” was activated to replace the batteries and cables.
That’s because a device known as an “air damper” first had to be reset.
The device is a rare one, with no generators elsewhere in the state hatchery system having one, according to the report.
“The generator would not start and displayed an ‘Air Damper’ fault light. [Name blacked out but a hatchery specialist 3] did not know the significance of this fault or how to correct it,” the report says.
It was only after a second person from the state’s Capital Assets Management Program in Lacey, a diesel mechanic, was apprised of the situation by phone at around 8 p.m. that they realized that that needed to be done and how to do it.
“These instructions resulted in the generator system becoming operational and suppling (sic) electrical power to the Minter Creek Hatchery,” the report states.
It’s believed that the delay in figuring out the air damper issue didn’t result in the loss of more fish.
But it is emblematic of wider communications failures at the facility, beginning with what to do in that critical first half hour.
The report states:
“The employees at Minter Creek Hatchery did not have a pre-established plan or instructions on how to maintain water flow and overcome the failure of electrical power in sustaining fish viability. The employees realized maintaining oxygenation in the water flow was critical and initially worked at various unguided solutions. This included activation and utilization of non-electric powered pumps and gravity flow of water.”
It also says:
“The employees present at Minter Creek Hatchery did not have the knowledge, training or reference materials on how to fully reset the Emergency Stop control, following activation of the Emergency Stop control.
“The employees present at Minter Creek Hatchery did not have knowledge, training or reference materials on how to reset the Air Damper fault indication, following activation of the Emergency Stop control.”
A “Considerations for the future” section of the report lays out an improved set of responsibilities for the facility manager and their hatchery specialists.
About half an hour after the generator was started, local utility crews restored the electricity.
Several days later the root cause analysis was contracted.
The report states that the backup power source was serviced and tested in March 2018 and found to be in “good overall condition,” though it apparently had been at least six years since the starter batteries had been replaced.
Batteries are more likely to fail the older they get.
Carper’s and Sebastian’s conclusion states:
The best explanation of the root cause generator failure is the result from a loose or cracked starting battery connection. Due to potential for corrosion, vibration, and what can be fairly significant temperature cycles, generator manufacturers typically recommend maintenance procedures for regular inspections of all wiring. Battery cable connections may become loose or damaged and should be inspected at every
generator exercise cycle by gently tugging and wiggling the cables.
It is inherent that electrical terminations on hot lead battery posts that are on vibrating engine frames will tend to loosen over time, so regular checks are recommended.
According to the generator log book, it had not been run for 67 days prior to the failure. With more frequent testing and inspections, this issue with the battery cable could have been discovered and repaired under normal conditions.
WDFW initially estimated that up to 6.2 million Chinook fry might have died without fresh water circulating through their rearing trays, but have now revised that downwards to 4.1 million following “a more robust inventory.”
More than half have been replaced with “excess” fish from six state, tribal — including 750,000 from the Suquamish and Nisqually Tribes — and tech college hatcheries.
The facility also raises chum and coho salmon for state and tribal fisheries. Those fish return and spawn later in the year than Chinook, and their eggs were still in a stage where they could survive if the water was drained off but were still kept moist.
A SCHEMATIC DRAWING SHOWS THE LAYOUT OF MINTER CREEK HATCHERY. (WDFW)
AGAIN, I’M NOT TRYING TO COME OFF AS SOME EXPERT in these matters and I readily acknowledge that s*** is just gonna happen at the worst possible times.
Been there, done that.
I do appreciate that WDFW apparently published the report without being prompted to get ahead of some reporter’s public disclosure request.
It harkens back to agency leaders’ vow before state lawmakers last winter to “to hold ourselves accountable for the tragic loss of the fish.”
WDFW DIRECTOR KELLY SUSEWIND, FISH PROGRAM DEPUTY ASSISTANT DIRECTOR KELLY CUNNINGHAM AND HATCHERY DIVISION MANAGER ERIC KINNE SPEAK BEFORE A STATE SENATE COMMITTEE ON WHAT HAPPENED AT MINTER CREEK. (TVW)
“We’re committed to making improvements to safeguard against another incident like the one at Minter Creek,” Kelly Cunningham, WDFW acting assistant Fish Program director said in the news release.
But even as the problem is being addressed, the report reveals an operations negligence that I find unacceptable.
That the two-decade-old generator wasn’t tested more frequently or ahead of the National Weather Service’s forecasted high wind warning that day, which put the fish at risk, boggles my mind.
That there wasn’t an established backup plan to the backup power source is puzzling.
As wild salmon continue to struggle and their recovery is decades — centuries? — down the road, the future of our seasons and starving orcas depend ever more highly on producing fish.
Minter is a key hatchery, putting out Chinook and chum, which southern resident killer whales depend on year-round and in fall, respectively.
Again, WDFW is obviously working on improving their operation, but they need to do better, and show us they are.
This was an entirely preventable, manmade disaster that didn’t need to happen if regular testing had been undertaken and it wasn’t assumed that the generator would just fire up whenever needed.
A harsh lesson, but one that’s been learned hopefully.