Partial Outage of the “Waterfall” Node

Resolved31/01/2023 at 7:20 AMUTC
The issue has been resolved — the node is replying to requests normally.

To remove any potential future occurrence of that issue, a notification system for anomalies has been added and tested to the infrastructure stack.
Monitoring31/01/2023 at 7:05 AMUTC
The fix has been implemented (reloading the node stack).
Identified31/01/2023 at 7:02 AMUTC
The abnormal CPU and disk usage has been identified at the Waterfall node.

The root cause is declared to be a MariaDB process that leaked memory bit-by-bit over the past days leading to no RAM being left for other processes.
Investigating31/01/2023 at 6:47 AMUTC
We are currently investigating this incident.
As of now, seems like the Waterfall node is unresponsive to some requests.

cmld. - Partial Outage of the “Waterfall” Node – Incident details