Major Firewall Issue at RU-OMS-CHRNSH

Resolved

Major outage

Started over 2 years agoLasted about 12 hours

Affected

Cumlord Mail (mail.cumlord.ru)

Appwrite API (api.cumlord.ru)

GitLab Instance (gl.vprw.ru)

LanguageTool Server (languagetool.supplemental-services.vapronva.pw)

Linx (ln.vprw.ru)

Mastodon (mastodon.vapronva.pw)

Updates

Resolved
28/05/2023 at 5:46 AM
Resolved
28/05/2023 at 5:46 AM
The bho-core node is back online now too.
That marks all the services being back up online.
Update
28/05/2023 at 5:40 AM
Update
28/05/2023 at 5:40 AM
data-ligma-gaming-v2 and should-be-fine-dobh are back online.
Monitoring
28/05/2023 at 5:38 AM
Monitoring
28/05/2023 at 5:38 AM
I cannot believe this, but apparently the culprit all this time was a power outage. The behaviour that was observed was as expected: firewall was the last thing to shut down after 2 hours and 10 minutes of working off the battery.
All the systems are being automatically turned on.
Update
27/05/2023 at 7:43 PM
Update
27/05/2023 at 7:43 PM
Unfortunately, there are no "remote hands" available there at the moment, so the expected downtime will exceed 36 hours, until mid-day Monday.
Identified
27/05/2023 at 6:14 PM
Identified
27/05/2023 at 6:14 PM
Due to a faulty hard drive on the entry firewall, the whole location seems to be unavailable.
Strangely enough, there were none health issues from ZFS (as well as scrubbing). Maybe something else is at fault.
Investigating
27/05/2023 at 6:04 PM
Investigating
27/05/2023 at 6:04 PM
We are currently investigating this incident.

cmld. - Major Firewall Issue at RU-OMS-CHRNSH – Incident details