Reservoir failure linked to software

An internal report from the city of Lewiston places blame for the reservoir failure on a software problem that happened after work done by an Idaho Falls firm.

The document, obtained by the Lewiston Tribune through a public records request, covers activities the day before and the day of the breach of the city’s High Reservoir, which was constructed in the 1920s. The failure released a flood of more than 3 million gallons of water that damaged homes, streets and businesses.

A telephone message and email to the Idaho Falls firm, Control System Technology (CST) on Thursday were not immediately returned.

The city’s look at the reservoir failure was in collaboration with CST and third-party investigators that are not named, according to the April 27 report.

“After thorough examination, we found no direct evidence of … (hackers) or malicious intent,” according to the report.

Sections of the report were redacted, citing exemptions in Idaho code.

When the breach happened Jan. 18, the city’s water treatment plant that normally feeds the reservoir was undergoing a $30 million upgrade and was disconnected from the reservoir. The contractor for the upgrade was IMCO.

During that project, an issue surfaced Jan. 17 that “required the involvement of a subcontractor,” CST, according to the report by two city of Lewiston employees: Dustin Spooner, information systems director, and Danny Santiago, systems administrator.

A CST employee gained remote access to the city’s supervisory control and data acquisition system at 11:10 a.m. for five minutes on Jan. 17 to troubleshoot and resolve the issue, according to the report, which relied on staff testimony and logs.

The system gathers and analyzes real-time data to monitor and control water and wastewater treatment equipment.

The CST employee with remote access instructed a CST employee on site to “disconnect and reconnect the network cable from the (supervisory control and data acquisition) machine in an attempt to resolve the issue they were facing,” according to the report. “This action led to the loss of network connectivity.”

A value in the supervisory control and data acquisition system became stuck while network connectivity was lost leading to an inaccurate reading of the water level at High Reservoir, according to the report.

“This stuck value caused the system to continually call for water, leading to the overfilling of the reservoir,” according to the report. “The excessive pressure on the reservoir eventually led to its failure.”

No alarms were triggered “as the system didn’t detect abnormal water levels or prolonged pumping due to the stuck value,” according to the report.

The next day, Jan. 18, police were alerted about a large volume of water on a road at 3 a.m., according to the report.

Law enforcement contacted water staff “who discovered the (supervisory control and data acquisition) system unresponsive and unable to control the water flow at High Reservoir,” according the report.

Water flow prevented personnel from shutting off the water manually and High Reservoir ruptured, according to the report.

“The disconnection of the network from the (supervisory control and data acquisition) machine by CST employees caused the software to capture and maintain the High Reservoir sensor’s value at that specific moment, resulting in a stuck value,” according to the summary and conclusion in the report.

“This stuck value unintentionally set off a chain of events that ultimately led to the rupture of High Reservoir,” according to the summary and conclusion in the report.

The report makes recommendations to strengthen the city’s supervisory control and data acquisition system to prevent similar incidents in the future.

City personnel have identified a cloud-based supervisory control and data acquisition vendor that would improve accessibility, security, reliability and functionality of the system at a lower cost, according to the report.

Discussions were underway with the vendor when the report was issued, but it is unclear what the result of the talks were.

Lewiston Mayor Dan Johnson said in an email Thursday a public records request would be required to obtain information about what recommendations in the report have been followed or not followed.

The rupture of the reservoir, one of the city’s largest, has impacted all of Lewiston. A total of 65 claims have been filed with the city by individuals and businesses seeking compensation for property damage, loss of belongings in the flood and reduction in income.

Most of the city’s water users were under a boil water order for about a week while the damaged reservoir was disconnected from the rest of the city’s water infrastructure.

A majority of the city’s water customers were banned from using automated irrigation from early May until July 31 because of reduced capacity in the city’s water system while the reservoir was repaired. The project to fix the reservoir cost about $3 million.

Williams may be contacted at ewilliam@lmtribune.com or (208) 848-2261.