Major Firewall Issue at RU-OMS-CHRNSH

Resolved

Major outage

Started about 3 years ago27/05/2023Lasted about 12 hours27/05/20236:04 PM28/05/20235:46 AMUTC

Affected

Cumlord Mail (mail.cumlord.ru)

Appwrite API (api.cumlord.ru)

GitLab Instance (gl.vprw.ru)

LanguageTool Server (languagetool.supplemental-services.vapronva.pw)

Linx (ln.vprw.ru)

Mastodon (mastodon.vapronva.pw)

Updates

Resolved
28/05/2023 at 5:46 AMUTC
Resolved
28/05/2023 at 5:46 AMUTC
The bho-core node is back online now too.
That marks all the services being back up online.
Update
28/05/2023 at 5:40 AMUTC
Update
28/05/2023 at 5:40 AMUTC
data-ligma-gaming-v2 and should-be-fine-dobh are back online.
Monitoring
28/05/2023 at 5:38 AMUTC
Monitoring
28/05/2023 at 5:38 AMUTC
I cannot believe this, but apparently the culprit all this time was a power outage. The behaviour that was observed was as expected: firewall was the last thing to shut down after 2 hours and 10 minutes of working off the battery.
All the systems are being automatically turned on.
Update
27/05/2023 at 7:43 PMUTC
Update
27/05/2023 at 7:43 PMUTC
Unfortunately, there are no "remote hands" available there at the moment, so the expected downtime will exceed 36 hours, until mid-day Monday.
Identified
27/05/2023 at 6:14 PMUTC
Identified
27/05/2023 at 6:14 PMUTC
Due to a faulty hard drive on the entry firewall, the whole location seems to be unavailable.
Strangely enough, there were none health issues from ZFS (as well as scrubbing). Maybe something else is at fault.
Investigating
27/05/2023 at 6:04 PMUTC
Investigating
27/05/2023 at 6:04 PMUTC
We are currently investigating this incident.

cmld. - Major Firewall Issue at RU-OMS-CHRNSH – Incident details