Major Firewall Issue at RU-OMS-CHRNSH

Affected

Cumlord Mail (mail.cumlord.ru)

Appwrite API (api.cumlord.ru)

GitLab Instance (gl.vprw.ru)

LanguageTool Server (languagetool.supplemental-services.vapronva.pw)

Linx (ln.vprw.ru)

Mastodon (mastodon.vapronva.pw)

Resolved28/05/2023 at 5:46 AMUTC
The bho-core node is back online now too.
That marks all the services being back up online.
Update28/05/2023 at 5:40 AMUTC
data-ligma-gaming-v2 and should-be-fine-dobh are back online.
Monitoring28/05/2023 at 5:38 AMUTC
I cannot believe this, but apparently the culprit all this time was a power outage. The behaviour that was observed was as expected: firewall was the last thing to shut down after 2 hours and 10 minutes of working off the battery.
All the systems are being automatically turned on.
Update27/05/2023 at 7:43 PMUTC
Unfortunately, there are no "remote hands" available there at the moment, so the expected downtime will exceed 36 hours, until mid-day Monday.
Identified27/05/2023 at 6:14 PMUTC
Due to a faulty hard drive on the entry firewall, the whole location seems to be unavailable.
Strangely enough, there were none health issues from ZFS (as well as scrubbing). Maybe something else is at fault.
Investigating27/05/2023 at 6:04 PMUTC
We are currently investigating this incident.

cmld. - Major Firewall Issue at RU-OMS-CHRNSH – Incident details