cmld. Status - Incident history https://cumlord.instatus.com cmld. Fri, 17 Nov 2023 14:00:00 +0000 SEBP — Tinkoff Returns Errored Requests Type: Incident Affected Components: Sosanie Ebla Premium (api.sosanie-ebla-bot-premium.vapronva.pw) Nov 17, 14:00:00 GMT+0 - Investigating - I am currently investigating this incident. Some providers of the TTS are not working as expected. Nov 17, 16:36:00 GMT+0 - Identified - Apparently Tinkoff now requires to sign the agreement "on use of services", which only a legal entity can sign. Oh, and it also restricts any third-party usage and/or reselling of the services. Which is a bummer, but understandable in the light of their recent controversy nevertheless. Type: Incident

Affected Components:

<p><small>Nov <var data-var='date'> 17</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Investigating</strong> - I am currently investigating this incident. Some providers of the TTS are not working as expected..</p> <p><small>Nov <var data-var='date'> 17</var>, <var data-var='time'>16:36:00</var> GMT+0</small><br><strong>Identified</strong> - Apparently Tinkoff now requires to sign the agreement "on use of services", which only a legal entity can sign. Oh, and it also restricts any third-party usage and/or reselling of the services. Which is a bummer, but understandable in the light of their recent controversy nevertheless..</p> ]]> Fri, 17 Nov 2023 14:00:00 +0000 https://cumlord.instatus.com/incident/clp50oq8u24169vootxu4mlg6 https://cumlord.instatus.com/incident/clp50oq8u24169vootxu4mlg6 Regular Software Update Triage on Tueday! (W44) Type: Maintenance Duration: 1 hour and 8 minutes Affected Components: Cumlord DNS (dns.cumlord.ru), Paperless (paperless.vapronva.pw), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Linx (ln.vprw.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), Mozilla Send (send.vapronva.pw), Shlink (shlink.vapronva.pw), Umami (analytics.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Wakapi (wakapi.vapronva.pw), Obsidian Sync Database, WriteFreely (wf.vprw.ru), Cumlord Certificate Authority (ca.cumlord.ru), Infrared Minecraft Proxy (bebracraft.net), Sentry (sentry.cumlord.ru), Privateering Archival Storage (privateering.vapronva.pw), Planka (pl.vprw.ru), ISU Calendar Schedule (schedule.university.vapronva.pw), Outline (wiki.cumlord.ru), genji.likes-sniffing-f.art, ChatGPT UI (chatgpt.vapronva.pw), Nextcloud (nc.vprw.ru), Nextcloud (nf.vprw.ru), Weblate (wb.vprw.ru), Flagsmith (feature.api.cumlord.ru), ILLA Builder (illa.vapronva.pw), BetterScan (scanmycode.vapronva.pw), Sonarqube (sonarqube.vapronva.pw), Vikunja (vi.vprw.ru), Mattermost (mm.vprw.ru), AIBooru (ab.vprw.ru), BeatTogether (bt.vprw.ru), Immich (im.vprw.ru), Immich for VR (vrc.vprw.ru), Jitsi (ji.vprw.ru), Zoom Recording Service, PhotoPrism Instance, thats-a-nice-argument-unfortunately.com, Sosanie Ebla Premium (api.sosanie-ebla-bot-premium.vapronva.pw), фредж.рф, Personal Seedbox, HedgeDoc (hd.vprw.ru), Cumlord SSO (sso.cumlord.ru) Nov 13, 23:32:00 GMT+0 - Identified - Maintenance is now in progress. 1. `lamentable-examination` will be scaled down. 2. `distrubing-lamp`, `voracious-speech`, `impeccable-argument` will upgraded and restarted one at a time. Nov 13, 23:40:00 GMT+0 - Identified - App updates are now in progress. Nov 14, 00:18:53 GMT+0 - Identified - All apps have been upgraded. Nov 14, 00:40:00 GMT+0 - Completed - Maintenance has completed successfully. Nov 13, 23:30:00 GMT+0 - Identified - I am planning for a scheduled maintenance to update firmware, software packages, critical apps on all nodes. This, however, will result in a temporary unavailability of certain services (including critical infrastructure) up to 3 minutes at a time. Type: Maintenance

Duration: 1 hour and 8 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

<p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>23:32:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress. 1. `lamentable-examination` will be scaled down. 2. `distrubing-lamp`, `voracious-speech`, `impeccable-argument` will upgraded and restarted one at a time..</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>23:40:00</var> GMT+0</small><br><strong>Identified</strong> - App updates are now in progress..</p> <p><small>Nov <var data-var='date'> 14</var>, <var data-var='time'>00:18:53</var> GMT+0</small><br><strong>Identified</strong> - All apps have been upgraded..</p> <p><small>Nov <var data-var='date'> 14</var>, <var data-var='time'>00:40:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>23:30:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance to update firmware, software packages, critical apps on all nodes. This, however, will result in a temporary unavailability of certain services (including critical infrastructure) up to 3 minutes at a time..</p> ]]> Mon, 13 Nov 2023 23:30:00 +0000 https://cumlord.instatus.com/maintenance/cloxjz32y120049wn816ik825p https://cumlord.instatus.com/maintenance/cloxjz32y120049wn816ik825p Hypervisors and Firewall Updates + Networking Upgrade + Rack Cleaning Type: Maintenance Duration: 1 hour and 56 minutes Affected Components: Sosanie Ebla Premium (api.sosanie-ebla-bot-premium.vapronva.pw), HedgeDoc (hd.vprw.ru), фредж.рф, Shlink (shlink.vapronva.pw), Immich for VR (vrc.vprw.ru), Infrared Minecraft Proxy (bebracraft.net), Privateering Archival Storage (privateering.vapronva.pw), Zoom Recording Service, BetterScan (scanmycode.vapronva.pw), Immich (im.vprw.ru), Cumlord DNS (dns.cumlord.ru), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Linx (ln.vprw.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), Mozilla Send (send.vapronva.pw), Umami (analytics.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Wakapi (wakapi.vapronva.pw), Obsidian Sync Database, Paperless (paperless.vapronva.pw), WriteFreely (wf.vprw.ru), Cumlord Certificate Authority (ca.cumlord.ru), Sentry (sentry.cumlord.ru), Planka (pl.vprw.ru), ISU Calendar Schedule (schedule.university.vapronva.pw), Outline (wiki.cumlord.ru), genji.likes-sniffing-f.art, ChatGPT UI (chatgpt.vapronva.pw), Nextcloud (nc.vprw.ru), Nextcloud (nf.vprw.ru), AIBooru (ab.vprw.ru), BeatTogether (bt.vprw.ru), Jitsi (ji.vprw.ru), PhotoPrism Instance, thats-a-nice-argument-unfortunately.com, Cumlord SSO (sso.cumlord.ru), Weblate (wb.vprw.ru), Flagsmith (feature.api.cumlord.ru), ILLA Builder (illa.vapronva.pw), Sonarqube (sonarqube.vapronva.pw), Vikunja (vi.vprw.ru), Mattermost (mm.vprw.ru) Nov 3, 06:30:00 GMT+0 - Completed - Maintenance has completed successfully. Proxmox host, pfSense, and TrueNAS systems have been upgraded successfully. Unfortunately, installing a 10gig network card on `BHO Core` did not work. Nov 3, 05:07:12 GMT+0 - Identified - `pfs` and `fire-utiliser` have been upgraded successfully. Nov 3, 04:00:00 GMT+0 - Identified - I am planning for a scheduled maintenance during that time: 1. Upgrading Proxmox hosts to the latest version; 2. Upgrading pfSense to 23.09; 3. Installing 10gig card in `bho-core`; 4. Cleaning up the rack. Nov 3, 04:34:02 GMT+0 - Identified - Maintenance is now in progress. The `fire-utiliser` host and `pfs` firewall are now being updated. Type: Maintenance

Duration: 1 hour and 56 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. Proxmox host, pfSense, and TrueNAS systems have been upgraded successfully. Unfortunately, installing a 10gig network card on `BHO Core` did not work..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>05:07:12</var> GMT+0</small><br><strong>Identified</strong> - `pfs` and `fire-utiliser` have been upgraded successfully..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance during that time: 1. Upgrading Proxmox hosts to the latest version; 2. Upgrading pfSense to 23.09; 3. Installing 10gig card in `bho-core`; 4. Cleaning up the rack..</p> <p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>04:34:02</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress. The `fire-utiliser` host and `pfs` firewall are now being updated..</p> ]]> Fri, 3 Nov 2023 04:00:00 +0000 https://cumlord.instatus.com/maintenance/clohju8wa0636a3olu9w1hs05 https://cumlord.instatus.com/maintenance/clohju8wa0636a3olu9w1hs05 Regular Software Update Triage on Tueday! Type: Maintenance Duration: 1 hour and 59 minutes Affected Components: Umami (analytics.vapronva.pw), Obsidian Sync Database, Sentry (sentry.cumlord.ru), ISU Calendar Schedule (schedule.university.vapronva.pw), Wakapi (wakapi.vapronva.pw), PhotoPrism Instance, фредж.рф, thats-a-nice-argument-unfortunately.com, Cumlord DNS (dns.cumlord.ru), Vikunja (vi.vprw.ru), Matrix (matrix.vapronva.pw), Appwrite API (api.cumlord.ru), Sosanie Ebla Premium (api.sosanie-ebla-bot-premium.vapronva.pw), Nextcloud (nc.vprw.ru), Pixelfed (pixelfed.vapronva.pw), GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Cumlord Mail (mail.cumlord.ru), WriteFreely (wf.vprw.ru), Paperless (paperless.vapronva.pw), Shlink (shlink.vapronva.pw), Linx (ln.vprw.ru), Cumlord SSO (sso.cumlord.ru), Bitwarden (bitwarden.cumlord.ru), Mastodon (mastodon.vapronva.pw), Immich for VR (vrc.vprw.ru), Cumlord Certificate Authority (ca.cumlord.ru), Mozilla Send (send.vapronva.pw), Personal Seedbox, Infrared Minecraft Proxy (bebracraft.net), Jitsi (ji.vprw.ru), PeerTube (peertube.vapronva.pw), Sonarqube (sonarqube.vapronva.pw), Privateering Archival Storage (privateering.vapronva.pw), Zoom Recording Service, Planka (pl.vprw.ru), AIBooru (ab.vprw.ru), BeatTogether (bt.vprw.ru), Outline (wiki.cumlord.ru), Flagsmith (feature.api.cumlord.ru), genji.likes-sniffing-f.art, ChatGPT UI (chatgpt.vapronva.pw), Nextcloud (nf.vprw.ru), Mattermost (mm.vprw.ru), Weblate (wb.vprw.ru), ILLA Builder (illa.vapronva.pw), BetterScan (scanmycode.vapronva.pw), Immich (im.vprw.ru) Oct 17, 05:26:33 GMT+0 - Completed - Maintenance has completed successfully. Oct 17, 04:02:46 GMT+0 - Identified - `doka1-ts`, `impeccable-argument`, `disturbing-lamp` nodes have been upgraded as well as any services on them. Oct 17, 03:30:00 GMT+0 - Identified - I am planning for a scheduled maintenance during that time: performing regular software updates (both apps, software, and firmware). This, however, will affect critical infrastructure such as reverse proxies, runners for CIs, etc. Additionally, kernel updates will be performed. That means some services might be not working or have a degraded performance for up to 1 minute at a time. Oct 17, 03:41:00 GMT+0 - Identified - `Should Be Fine` machine will be promptly restarted. This follows a 1-minute downtime across all web-related services. Oct 17, 03:28:00 GMT+0 - Identified - Maintenance is now in progress. `Should Be Fine` is being updated. This will include a close-to-zero downtime deployment of Caddy. Oct 17, 03:42:00 GMT+0 - Identified - `Should Be Fine` has been restarted successfully and is up and running will all the services on it hopping smoothly. Oct 17, 04:28:00 GMT+0 - Identified - `bho-core`, `vapor-scaler`, `ts@ek` nodes have been upgraded. Type: Maintenance

Duration: 1 hour and 59 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

<p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>05:26:33</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>04:02:46</var> GMT+0</small><br><strong>Identified</strong> - `doka1-ts`, `impeccable-argument`, `disturbing-lamp` nodes have been upgraded as well as any services on them..</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>03:30:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance during that time: performing regular software updates (both apps, software, and firmware). This, however, will affect critical infrastructure such as reverse proxies, runners for CIs, etc. Additionally, kernel updates will be performed. That means some services might be not working or have a degraded performance for up to 1 minute at a time. .</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>03:41:00</var> GMT+0</small><br><strong>Identified</strong> - `Should Be Fine` machine will be promptly restarted. This follows a 1-minute downtime across all web-related services..</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>03:28:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress. `Should Be Fine` is being updated. This will include a close-to-zero downtime deployment of Caddy..</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>03:42:00</var> GMT+0</small><br><strong>Identified</strong> - `Should Be Fine` has been restarted successfully and is up and running will all the services on it hopping smoothly..</p> <p><small>Oct <var data-var='date'> 17</var>, <var data-var='time'>04:28:00</var> GMT+0</small><br><strong>Identified</strong> - `bho-core`, `vapor-scaler`, `ts@ek` nodes have been upgraded..</p> ]]> Tue, 17 Oct 2023 03:30:00 +0000 https://cumlord.instatus.com/maintenance/clntprk399694xioeq2nk7t98 https://cumlord.instatus.com/maintenance/clntprk399694xioeq2nk7t98 Mail Storage Migration & Upgrades Type: Maintenance Duration: 47 minutes Affected Components: Cumlord Mail (mail.cumlord.ru) Oct 16, 22:34:12 GMT+0 - Identified - All mailcow services are back up and running. Oct 16, 22:43:04 GMT+0 - Completed - Maintenance has completed successfully. Mail is operating without any issues. Oct 16, 22:13:49 GMT+0 - Identified - Storage migration has been completed. Oct 16, 21:55:44 GMT+0 - Identified - Maintenance is now in progress. Oct 16, 22:00:00 GMT+0 - Identified - I am planning for a scheduled maintenance on Cumlord Mail during that time. The storage (files, databases, etc) will be migrated to a new dataset structure, as it is one of the leftover legacy projects that have not been yet migrated. Unfortunately, due to the nature of the maintenance, the service might be unavailable for up to 5–10 minutes at a time. Oct 16, 22:29:01 GMT+0 - Identified - Mailcow is now being updated and configs are being migrated. Type: Maintenance

Duration: 47 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>22:34:12</var> GMT+0</small><br><strong>Identified</strong> - All mailcow services are back up and running..</p> <p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>22:43:04</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. Mail is operating without any issues..</p> <p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>22:13:49</var> GMT+0</small><br><strong>Identified</strong> - Storage migration has been completed..</p> <p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>21:55:44</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance on Cumlord Mail during that time. The storage (files, databases, etc) will be migrated to a new dataset structure, as it is one of the leftover legacy projects that have not been yet migrated. Unfortunately, due to the nature of the maintenance, the service might be unavailable for up to 5–10 minutes at a time..</p> <p><small>Oct <var data-var='date'> 16</var>, <var data-var='time'>22:29:01</var> GMT+0</small><br><strong>Identified</strong> - Mailcow is now being updated and configs are being migrated..</p> ]]> Mon, 16 Oct 2023 22:00:00 +0000 https://cumlord.instatus.com/maintenance/clnte6ei31221xioecmg0qif4 https://cumlord.instatus.com/maintenance/clnte6ei31221xioecmg0qif4 Regular Software Update Triage on Tueday! (W41) Type: Maintenance Duration: 35 minutes Affected Components: Wakapi (wakapi.vapronva.pw), PhotoPrism Instance, Vikunja (vi.vprw.ru), Matrix (matrix.vapronva.pw), Appwrite API (api.cumlord.ru), Pixelfed (pixelfed.vapronva.pw), GitLab Instance (gl.vprw.ru), WriteFreely (wf.vprw.ru), Paperless (paperless.vapronva.pw), Shlink (shlink.vapronva.pw), Cumlord SSO (sso.cumlord.ru), Mastodon (mastodon.vapronva.pw), Immich for VR (vrc.vprw.ru), Cumlord Certificate Authority (ca.cumlord.ru), Mozilla Send (send.vapronva.pw), Personal Seedbox, Infrared Minecraft Proxy (bebracraft.net), PeerTube (peertube.vapronva.pw), Umami (analytics.vapronva.pw), Sonarqube (sonarqube.vapronva.pw), Privateering Archival Storage (privateering.vapronva.pw), Planka (pl.vprw.ru), AIBooru (ab.vprw.ru), BeatTogether (bt.vprw.ru), Outline (wiki.cumlord.ru), Flagsmith (feature.api.cumlord.ru), Nextcloud (nc.vprw.ru), ChatGPT UI (chatgpt.vapronva.pw), Nextcloud (nf.vprw.ru), Mattermost (mm.vprw.ru), Weblate (wb.vprw.ru), ILLA Builder (illa.vapronva.pw), BetterScan (scanmycode.vapronva.pw), Immich (im.vprw.ru) Oct 10, 18:45:00 GMT+0 - Identified - Maintenance is now in progress. Oct 10, 18:40:00 GMT+0 - Identified - I am planning for a scheduled maintenance during that time. This includes only a regular update of the hosted services (app updates), not critical apps (like, Traefik, proxies, DNS, etc). Oct 10, 19:20:00 GMT+0 - Completed - Maintenance has completed successfully. Type: Maintenance

Duration: 35 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

<p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>18:45:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>18:40:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance during that time. This includes only a regular update of the hosted services (app updates), not critical apps (like, Traefik, proxies, DNS, etc)..</p> <p><small>Oct <var data-var='date'> 10</var>, <var data-var='time'>19:20:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> Tue, 10 Oct 2023 18:40:00 +0000 https://cumlord.instatus.com/maintenance/clnknugyu12719tog7s18mfdg https://cumlord.instatus.com/maintenance/clnknugyu12719tog7s18mfdg Upgrading Nextcloud to 27.1 + Setting Up Elasticsearch Type: Maintenance Duration: 36 minutes Affected Components: Nextcloud (nc.vprw.ru) Sep 30, 14:36:13 GMT+0 - Identified - Okay, I will try to force Nextcloud to reinstall its internal folders since holy frick, I can't get over this random error and can't find it anywhere else... Sep 30, 15:45:20 GMT+0 - Completed - I was not able to install Elasticsearch... It just does not work. It works fine on new instances, but refuses to work on mine. THE SAME FRICKING ERROR! Oh well... Sep 30, 14:00:00 GMT+0 - Identified - I am planning for a scheduled maintenance during that time to: 1. Reinstall `3rdparty` Nextcloud dependencies 2. Properly upgrade to the 27.1.0 3. Set up Elasticsearch and index all files Sep 30, 15:09:34 GMT+0 - Identified - PHP and the dependencies there... are stupid. Sep 30, 14:01:00 GMT+0 - Identified - Maintenance is now in progress. Sep 30, 14:13:00 GMT+0 - Identified - Compose has been merged with the upstream changes. Elasticsearch still does not work. Nextcloud, on the other hand, is up and running without any other issues. Type: Maintenance

Duration: 36 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>14:36:13</var> GMT+0</small><br><strong>Identified</strong> - Okay, I will try to force Nextcloud to reinstall its internal folders since holy frick, I can't get over this random error and can't find it anywhere else....</p> <p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>15:45:20</var> GMT+0</small><br><strong>Completed</strong> - I was not able to install Elasticsearch... It just does not work. It works fine on new instances, but refuses to work on mine. THE SAME FRICKING ERROR! Oh well....</p> <p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance during that time to: 1. Reinstall `3rdparty` Nextcloud dependencies 2. Properly upgrade to the 27.1.0 3. Set up Elasticsearch and index all files.</p> <p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>15:09:34</var> GMT+0</small><br><strong>Identified</strong> - PHP and the dependencies there... are stupid..</p> <p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>14:01:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>14:13:00</var> GMT+0</small><br><strong>Identified</strong> - Compose has been merged with the upstream changes. Elasticsearch still does not work. Nextcloud, on the other hand, is up and running without any other issues..</p> ]]> Sat, 30 Sep 2023 14:00:00 +0000 https://cumlord.instatus.com/maintenance/cln63sg8v0559a3oi1bdhk80p https://cumlord.instatus.com/maintenance/cln63sg8v0559a3oi1bdhk80p Upgrading the Appwrite API to the 1.4.4 Version Type: Maintenance Duration: 2 minutes Affected Components: Appwrite API (api.cumlord.ru) Sep 29, 22:42:00 GMT+0 - Identified - Maintenance is now in progress. Sep 29, 22:40:00 GMT+0 - Identified - I am planning for a scheduled maintenance aimed to upgrade and migrate Appwrite to the latest commit at `1.4.x` branch that includes `1.4.4` version. Sep 29, 22:44:00 GMT+0 - Completed - Maintenance has completed successfully; Appwrite is up and running. Type: Maintenance

Duration: 2 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>22:42:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>22:40:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance aimed to upgrade and migrate Appwrite to the latest commit at `1.4.x` branch that includes `1.4.4` version..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>22:44:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully; Appwrite is up and running..</p> ]]> Fri, 29 Sep 2023 22:40:00 +0000 https://cumlord.instatus.com/maintenance/cln56smxa1015qslskualm8br https://cumlord.instatus.com/maintenance/cln56smxa1015qslskualm8br Upgrading the Storage Server (DLG2) to Scale 23.10 Type: Maintenance Duration: 12 minutes Affected Components: Obsidian Sync Database, Wakapi (wakapi.vapronva.pw), фредж.рф, thats-a-nice-argument-unfortunately.com, Umami (analytics.vapronva.pw), Vikunja (vi.vprw.ru), Matrix (matrix.vapronva.pw), Appwrite API (api.cumlord.ru), Pixelfed (pixelfed.vapronva.pw), Sosanie Ebla Premium (api.sosanie-ebla-bot-premium.vapronva.pw), Mozilla Send (send.vapronva.pw), Cumlord Mail (mail.cumlord.ru), GitLab Instance (gl.vprw.ru), PeerTube (peertube.vapronva.pw), WriteFreely (wf.vprw.ru), Paperless (paperless.vapronva.pw), Shlink (shlink.vapronva.pw), Linx (ln.vprw.ru), Mastodon (mastodon.vapronva.pw), Immich for VR (vrc.vprw.ru), Sonarqube (sonarqube.vapronva.pw), Jitsi (ji.vprw.ru), Zoom Recording Service, Planka (pl.vprw.ru), AIBooru (ab.vprw.ru), ISU Calendar Schedule (schedule.university.vapronva.pw), BeatTogether (bt.vprw.ru), Outline (wiki.cumlord.ru), Flagsmith (feature.api.cumlord.ru), genji.likes-sniffing-f.art, Nextcloud (nc.vprw.ru), ChatGPT UI (chatgpt.vapronva.pw), Nextcloud (nf.vprw.ru), Mattermost (mm.vprw.ru), Weblate (wb.vprw.ru), ILLA Builder (illa.vapronva.pw), BetterScan (scanmycode.vapronva.pw), Immich (im.vprw.ru) Sep 26, 23:52:00 GMT+0 - Identified - Maintenance is now in progress. Sep 26, 23:50:00 GMT+0 - Identified - I am planning for a scheduled maintenance during this time. As the main storage server will be briefly unavailable, many services hosted by the BHO-C will be down. Sep 27, 00:04:00 GMT+0 - Completed - Maintenance has completed successfully. The storage server has migrated its ZFS pools successfully. Type: Maintenance

Duration: 12 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

<p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>23:52:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>23:50:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance during this time. As the main storage server will be briefly unavailable, many services hosted by the BHO-C will be down..</p> <p><small>Sep <var data-var='date'> 27</var>, <var data-var='time'>00:04:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. The storage server has migrated its ZFS pools successfully..</p> ]]> Tue, 26 Sep 2023 23:50:00 +0000 https://cumlord.instatus.com/maintenance/cln0zcmaq49787arohtlzy2uld https://cumlord.instatus.com/maintenance/cln0zcmaq49787arohtlzy2uld Regular Software Update Triage on Tueday! Type: Maintenance Duration: 29 minutes Affected Components: Obsidian Sync Database, PhotoPrism Instance, Umami (analytics.vapronva.pw), Matrix (matrix.vapronva.pw), Appwrite API (api.cumlord.ru), PeerTube (peertube.vapronva.pw), Paperless (paperless.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Immich for VR (vrc.vprw.ru), Jitsi (ji.vprw.ru), Planka (pl.vprw.ru), AIBooru (ab.vprw.ru), BeatTogether (bt.vprw.ru), Outline (wiki.cumlord.ru), Flagsmith (feature.api.cumlord.ru), Nextcloud (nc.vprw.ru), Mattermost (mm.vprw.ru), Weblate (wb.vprw.ru), ILLA Builder (illa.vapronva.pw), BetterScan (scanmycode.vapronva.pw), Immich (im.vprw.ru) Sep 26, 20:21:46 GMT+0 - Identified - Maintenance is now in progress. Sep 26, 20:20:00 GMT+0 - Identified - I am planning for a scheduled maintenance during that time. I will updating any services that have new versions available. Most of them contain a basic database, however some require a more complex migration structure. Sep 26, 20:32:00 GMT+0 - Identified - Most of the services have been successfully update and are now up and running. ...except for the Umami (requires a complicated migration?) and PeerTube (cannot install themes). Sep 26, 20:51:00 GMT+0 - Identified - Umami has been successfully migrated. Well... It required to edit a few rows in the `_prisma_migations` to at least `2023-04-17` since I migrated it during the beta. Sep 26, 20:57:51 GMT+0 - Identified - Ok, PeerTube plugins and themes can now only be installed via the CLI tool... Sep 26, 21:26:57 GMT+0 - Completed - Maintenance has completed successfully. All the processing nodes at RU-OMS-CHRNSH have their software being successfully upgraded. Type: Maintenance

Duration: 29 minutes

Affected Components: , , , , , , , , , , , , , , , , , , , ,

<p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>20:21:46</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>20:20:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance during that time. I will updating any services that have new versions available. Most of them contain a basic database, however some require a more complex migration structure..</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>20:32:00</var> GMT+0</small><br><strong>Identified</strong> - Most of the services have been successfully update and are now up and running. ...except for the Umami (requires a complicated migration?) and PeerTube (cannot install themes)..</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>20:51:00</var> GMT+0</small><br><strong>Identified</strong> - Umami has been successfully migrated. Well... It required to edit a few rows in the `_prisma_migations` to at least `2023-04-17` since I migrated it during the beta..</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>20:57:51</var> GMT+0</small><br><strong>Identified</strong> - Ok, PeerTube plugins and themes can now only be installed via the CLI tool... .</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>21:26:57</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. All the processing nodes at RU-OMS-CHRNSH have their software being successfully upgraded..</p> ]]> Tue, 26 Sep 2023 20:20:00 +0000 https://cumlord.instatus.com/maintenance/cln0qmsap34829ayohud1y681w https://cumlord.instatus.com/maintenance/cln0qmsap34829ayohud1y681w Major Outage of Power at RU-OMS-CHRNSH Type: Incident Duration: 16 days and 3 hours Affected Components: Obsidian Sync Database, Wakapi (wakapi.vapronva.pw), фредж.рф, thats-a-nice-argument-unfortunately.com, Umami (analytics.vapronva.pw), Vikunja (vi.vprw.ru), Matrix (matrix.vapronva.pw), Appwrite API (api.cumlord.ru), Pixelfed (pixelfed.vapronva.pw), Sosanie Ebla Premium (api.sosanie-ebla-bot-premium.vapronva.pw), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Mozilla Send (send.vapronva.pw), Cumlord Mail (mail.cumlord.ru), GitLab Instance (gl.vprw.ru), PeerTube (peertube.vapronva.pw), WriteFreely (wf.vprw.ru), Paperless (paperless.vapronva.pw), Shlink (shlink.vapronva.pw), Linx (ln.vprw.ru), Cumlord SSO (sso.cumlord.ru), Bitwarden (bitwarden.cumlord.ru), Mastodon (mastodon.vapronva.pw), Immich for VR (vrc.vprw.ru), Cumlord Certificate Authority (ca.cumlord.ru), Sentry (sentry.cumlord.ru), Sonarqube (sonarqube.vapronva.pw), Infrared Minecraft Proxy (bebracraft.net), Jitsi (ji.vprw.ru), Zoom Recording Service, Planka (pl.vprw.ru), AIBooru (ab.vprw.ru), ISU Calendar Schedule (schedule.university.vapronva.pw), BeatTogether (bt.vprw.ru), Outline (wiki.cumlord.ru), Flagsmith (feature.api.cumlord.ru), genji.likes-sniffing-f.art, Nextcloud (nc.vprw.ru), ChatGPT UI (chatgpt.vapronva.pw), Nextcloud (nf.vprw.ru), Mattermost (mm.vprw.ru), Weblate (wb.vprw.ru), ILLA Builder (illa.vapronva.pw), BetterScan (scanmycode.vapronva.pw), Immich (im.vprw.ru) Sep 26, 02:30:00 GMT+0 - Monitoring - Hosts have successfully started up. I am monitoring for all the services to turn on without any issues. Sep 25, 20:58:00 GMT+0 - Identified - Yep, I have the confirmation that the power was out at around 23:00 (GMT+6). UPS provided the power for additional 3.5 hours. I am waiting for the power to come up at around 8:30 (GMT+6). Sep 26, 02:10:00 GMT+0 - Identified - The power has been fully restored. The UPS will turn on in a couple of minutes and the systems will begin a gradual start up sequence. Sep 26, 03:10:00 GMT+0 - Resolved - This incident has been resolved. All the services are up and running, everything is operating at a normal level. Sep 25, 20:43:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 16 days and 3 hours

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

<p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>02:30:00</var> GMT+0</small><br><strong>Monitoring</strong> - Hosts have successfully started up. I am monitoring for all the services to turn on without any issues..</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>20:58:00</var> GMT+0</small><br><strong>Identified</strong> - Yep, I have the confirmation that the power was out at around 23:00 (GMT+6). UPS provided the power for additional 3.5 hours. I am waiting for the power to come up at around 8:30 (GMT+6)..</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>02:10:00</var> GMT+0</small><br><strong>Identified</strong> - The power has been fully restored. The UPS will turn on in a couple of minutes and the systems will begin a gradual start up sequence..</p> <p><small>Sep <var data-var='date'> 26</var>, <var data-var='time'>03:10:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. All the services are up and running, everything is operating at a normal level..</p> <p><small>Sep <var data-var='date'> 25</var>, <var data-var='time'>20:43:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Mon, 25 Sep 2023 20:43:00 +0000 https://cumlord.instatus.com/incident/cln0oweff17300arohm1px6n7e https://cumlord.instatus.com/incident/cln0oweff17300arohm1px6n7e Critical Update to the C-Mail Type: Maintenance Duration: 4 minutes Affected Components: Cumlord Mail (mail.cumlord.ru) May 31, 03:12:00 GMT+0 - Identified - Maintenance is now in progress. May 31, 03:10:00 GMT+0 - Identified - I am planning for a maintenance during that time. It will include a [critical update](https://github.com/mailcow/mailcow-dockerized/releases/tag/2023-05a) to the Mailcow software. During this time, the mail services will be restarted briefly. May 31, 03:14:00 GMT+0 - Completed - Maintenance has completed successfully. Type: Maintenance

Duration: 4 minutes

Affected Components:

<p><small>May <var data-var='date'> 31</var>, <var data-var='time'>03:12:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>May <var data-var='date'> 31</var>, <var data-var='time'>03:10:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a maintenance during that time. It will include a [critical update](https://github.com/mailcow/mailcow-dockerized/releases/tag/2023-05a) to the Mailcow software. During this time, the mail services will be restarted briefly..</p> <p><small>May <var data-var='date'> 31</var>, <var data-var='time'>03:14:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> Wed, 31 May 2023 03:12:00 +0000 https://cumlord.instatus.com/maintenance/clib4ox6v09477wofzrcgf2ji https://cumlord.instatus.com/maintenance/clib4ox6v09477wofzrcgf2ji Major Firewall Issue at RU-OMS-CHRNSH Type: Incident Duration: 29 days, 6 hours and 26 minutes Affected Components: Umami (analytics.vapronva.pw), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Linx (ln.vprw.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), Mozilla Send (send.vapronva.pw), Shlink (shlink.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Wakapi (wakapi.vapronva.pw), Obsidian Sync Database, Paperless (paperless.vapronva.pw), WriteFreely (wf.vprw.ru) May 27, 18:14:00 GMT+0 - Identified - Due to a faulty hard drive on the entry firewall, the whole location seems to be unavailable. Strangely enough, there were none health issues from ZFS (as well as scrubbing). Maybe something else is at fault. May 28, 05:38:09 GMT+0 - Monitoring - I cannot believe this, but apparently the culprit all this time was a power outage. The behaviour that was observed was as expected: firewall was the last thing to shut down after 2 hours and 10 minutes of working off the battery. All the systems are being automatically turned on. May 28, 05:40:10 GMT+0 - Monitoring - `data-ligma-gaming-v2` and `should-be-fine-dobh` are back online. May 28, 05:46:25 GMT+0 - Resolved - The `bho-core` node is back online now too. That marks all the services being back up online. May 27, 19:43:00 GMT+0 - Identified - Unfortunately, there are no "remote hands" available there at the moment, so the expected downtime will exceed 36 hours, until mid-day Monday. May 27, 18:04:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 29 days, 6 hours and 26 minutes

Affected Components: , , , , , , , , , , , , , , , ,

<p><small>May <var data-var='date'> 27</var>, <var data-var='time'>18:14:00</var> GMT+0</small><br><strong>Identified</strong> - Due to a faulty hard drive on the entry firewall, the whole location seems to be unavailable. Strangely enough, there were none health issues from ZFS (as well as scrubbing). Maybe something else is at fault..</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>05:38:09</var> GMT+0</small><br><strong>Monitoring</strong> - I cannot believe this, but apparently the culprit all this time was a power outage. The behaviour that was observed was as expected: firewall was the last thing to shut down after 2 hours and 10 minutes of working off the battery. All the systems are being automatically turned on..</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>05:40:10</var> GMT+0</small><br><strong>Monitoring</strong> - `data-ligma-gaming-v2` and `should-be-fine-dobh` are back online..</p> <p><small>May <var data-var='date'> 28</var>, <var data-var='time'>05:46:25</var> GMT+0</small><br><strong>Resolved</strong> - The `bho-core` node is back online now too. That marks all the services being back up online..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>19:43:00</var> GMT+0</small><br><strong>Identified</strong> - Unfortunately, there are no "remote hands" available there at the moment, so the expected downtime will exceed 36 hours, until mid-day Monday..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>18:04:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Sat, 27 May 2023 18:04:00 +0000 https://cumlord.instatus.com/incident/cli6el0w9751850bamwrhiwspur https://cumlord.instatus.com/incident/cli6el0w9751850bamwrhiwspur Firmware Updates to the C-DNS Nodes Type: Maintenance Duration: 10 minutes May 24, 11:56:50 GMT+0 - Identified - Maintenance is now in progress. At all times there will be at least 2 nodes responding to the DNS requests. May 24, 12:04:32 GMT+0 - Completed - Maintenance has completed successfully. May 24, 11:55:00 GMT+0 - Identified - I am planning for a scheduled maintenance to update kernel, PowerDNS authoritative servers, and web panel. Type: Maintenance

Duration: 10 minutes

<p><small>May <var data-var='date'> 24</var>, <var data-var='time'>11:56:50</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress. At all times there will be at least 2 nodes responding to the DNS requests..</p> <p><small>May <var data-var='date'> 24</var>, <var data-var='time'>12:04:32</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>May <var data-var='date'> 24</var>, <var data-var='time'>11:55:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance to update kernel, PowerDNS authoritative servers, and web panel..</p> ]]> Wed, 24 May 2023 11:55:00 +0000 https://cumlord.instatus.com/maintenance/cli1nabwp6871sdof3uwb4m4c https://cumlord.instatus.com/maintenance/cli1nabwp6871sdof3uwb4m4c Installation of the UPS to the RU-OMS-CHRNSH Type: Maintenance Duration: 7 days and 22 hours Affected Components: Paperless (paperless.vapronva.pw), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), Obsidian Sync Database, GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Linx (ln.vprw.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), WriteFreely (wf.vprw.ru), Mozilla Send (send.vapronva.pw), Shlink (shlink.vapronva.pw), Umami (analytics.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Wakapi (wakapi.vapronva.pw) May 12, 14:10:00 GMT+0 - Completed - Maintenance has completed successfully. May 12, 11:26:00 GMT+0 - Identified - Maintenance is now in progress. May 12, 11:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Type: Maintenance

Duration: 7 days and 22 hours

Affected Components: , , , , , , , , , , , , , , , ,

<p><small>May <var data-var='date'> 12</var>, <var data-var='time'>14:10:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>May <var data-var='date'> 12</var>, <var data-var='time'>11:26:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>May <var data-var='date'> 12</var>, <var data-var='time'>11:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> ]]> Fri, 12 May 2023 11:00:00 +0000 https://cumlord.instatus.com/maintenance/clhv05yv41563198oktp7ehwh2 https://cumlord.instatus.com/maintenance/clhv05yv41563198oktp7ehwh2 Docker Died and Broke Down — I Broke Down Too Type: Incident Duration: 2 hours and 10 minutes Affected Components: Paperless (paperless.vapronva.pw), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), Obsidian Sync Database, GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Linx (ln.vprw.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), WriteFreely (wf.vprw.ru), Mozilla Send (send.vapronva.pw), Shlink (shlink.vapronva.pw), Umami (analytics.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Wakapi (wakapi.vapronva.pw) Apr 6, 13:20:00 GMT+0 - Investigating - `bho-core` server is behaving weird as hell. I have no idea what is going on. Apr 6, 15:30:00 GMT+0 - Resolved - This incident has been resolved. I will not provide any comments. I am broken. Type: Incident

Duration: 2 hours and 10 minutes

Affected Components: , , , , , , , , , , , , , , , ,

<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>13:20:00</var> GMT+0</small><br><strong>Investigating</strong> - `bho-core` server is behaving weird as hell. I have no idea what is going on..</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>15:30:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. I will not provide any comments. I am broken..</p> ]]> Thu, 6 Apr 2023 13:20:00 +0000 https://cumlord.instatus.com/incident/clg59j3410030w7onov924pl2 https://cumlord.instatus.com/incident/clg59j3410030w7onov924pl2 Major Software Updates to Core Infrastrucutre at RU-OMS Type: Maintenance Duration: 1 hour and 9 minutes Affected Components: Paperless (paperless.vapronva.pw), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), Obsidian Sync Database, GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Linx (ln.vprw.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), WriteFreely (wf.vprw.ru), Mozilla Send (send.vapronva.pw), Shlink (shlink.vapronva.pw), Umami (analytics.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Wakapi (wakapi.vapronva.pw) Apr 6, 12:39:26 GMT+0 - Completed - Maintenance has completed successfully. All the services are reported and checked to be running without any issues. Apr 6, 11:32:53 GMT+0 - Identified - Maintenance is now in progress. Core servers are being turned off as I await for the `DLGv2` to finish updating. Apr 6, 12:36:40 GMT+0 - Identified - A few issues with Kubernetes setup on `DLG-v2` have been fixed. Core servers are now up and running, as I await for all the services to start up. Apr 6, 12:19:54 GMT+0 - Identified - The `DLG-v2` storage server has been successfully updated to the `Bluefin` TrueNAS train. Apr 6, 11:30:00 GMT+0 - Identified - I will be performing major software updates to the core infrastructure at `RU-OMS`. The updates will include the following: - Updating TrueNAS storage (`data-ligma-gaming-v2`) to Bluefin - Updating Linux kernel and others (for instance, Docker) on the core servers (`bho-core` and `vapor-scaler`) These updates are necessary to ensure that systems are up-to-date with the latest software versions and to improve the overall performance and stability of the services I provide. During the maintenance period, many services *will* experience temporary disruption. I apologize for any inconvenience this may cause and will strive to minimize any downtime. Type: Maintenance

Duration: 1 hour and 9 minutes

Affected Components: , , , , , , , , , , , , , , , ,

<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>12:39:26</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully. All the services are reported and checked to be running without any issues..</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>11:32:53</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress. Core servers are being turned off as I await for the `DLGv2` to finish updating..</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>12:36:40</var> GMT+0</small><br><strong>Identified</strong> - A few issues with Kubernetes setup on `DLG-v2` have been fixed. Core servers are now up and running, as I await for all the services to start up..</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>12:19:54</var> GMT+0</small><br><strong>Identified</strong> - The `DLG-v2` storage server has been successfully updated to the `Bluefin` TrueNAS train..</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>11:30:00</var> GMT+0</small><br><strong>Identified</strong> - I will be performing major software updates to the core infrastructure at `RU-OMS`. The updates will include the following: - Updating TrueNAS storage (`data-ligma-gaming-v2`) to Bluefin - Updating Linux kernel and others (for instance, Docker) on the core servers (`bho-core` and `vapor-scaler`) These updates are necessary to ensure that systems are up-to-date with the latest software versions and to improve the overall performance and stability of the services I provide. During the maintenance period, many services *will* experience temporary disruption. I apologize for any inconvenience this may cause and will strive to minimize any downtime..</p> ]]> Thu, 6 Apr 2023 11:30:00 +0000 https://cumlord.instatus.com/maintenance/clg51ch9i312007b3ondjrkph6o https://cumlord.instatus.com/maintenance/clg51ch9i312007b3ondjrkph6o Software Updates to C-DNS Type: Maintenance Duration: 2 days and 20 hours Mar 7, 12:10:00 GMT+0 - Completed - Pre-upgrade operations have been done successfully. Mar 7, 11:02:00 GMT+0 - Identified - Pre-upgrade operations have started. This includes: - zero-downtime update to `Waterfall`, `Lavender`, and `Kettle` nodes; - update to the dashboard/panel; - zero-downtime update to the database. Mar 7, 11:24:04 GMT+0 - Identified - Domains under C-DNS operation have been upgraded to the new SOA that includes looser timings for TTL and a new management email. Type: Maintenance

Duration: 2 days and 20 hours

<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>12:10:00</var> GMT+0</small><br><strong>Completed</strong> - Pre-upgrade operations have been done successfully..</p> <p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>11:02:00</var> GMT+0</small><br><strong>Identified</strong> - Pre-upgrade operations have started. This includes: - zero-downtime update to `Waterfall`, `Lavender`, and `Kettle` nodes; - update to the dashboard/panel; - zero-downtime update to the database..</p> <p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>11:24:04</var> GMT+0</small><br><strong>Identified</strong> - Domains under C-DNS operation have been upgraded to the new SOA that includes looser timings for TTL and a new management email..</p> ]]> Tue, 7 Mar 2023 11:02:00 +0000 https://cumlord.instatus.com/maintenance/cley1i6xs56890haoje31x8g3j https://cumlord.instatus.com/maintenance/cley1i6xs56890haoje31x8g3j Updating and Optimizing PeerTube Instance Type: Maintenance Duration: 1 day, 13 hours and 38 minutes Affected Components: PeerTube (peertube.vapronva.pw) Mar 4, 20:37:37 GMT+0 - Completed - Maintenance has completed successfully: the instance has been updated, and all the migration issues have been resolved. Mar 4, 20:00:00 GMT+0 - Identified - I am planning for a scheduled maintenance on the [PeerTube](https://peertube.vapronva.pw). The plans are: - move to a new deployment strategy; - update to the latest `5.1.0-rc1` version; - optimize instance for livestreaming. Mar 4, 20:17:47 GMT+0 - Identified - A few deep configuration changes are required for the new version. Additionally, assets regeneration is required. Mar 4, 20:02:33 GMT+0 - Identified - Maintenance is now in progress. Mar 4, 20:07:47 GMT+0 - Identified - The instance has been successfully deployed and updated to a new version. Type: Maintenance

Duration: 1 day, 13 hours and 38 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:37:37</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully: the instance has been updated, and all the migration issues have been resolved..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:00:00</var> GMT+0</small><br><strong>Identified</strong> - I am planning for a scheduled maintenance on the [PeerTube](https://peertube.vapronva.pw). The plans are: - move to a new deployment strategy; - update to the latest `5.1.0-rc1` version; - optimize instance for livestreaming..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:17:47</var> GMT+0</small><br><strong>Identified</strong> - A few deep configuration changes are required for the new version. Additionally, assets regeneration is required..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:02:33</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:07:47</var> GMT+0</small><br><strong>Identified</strong> - The instance has been successfully deployed and updated to a new version..</p> ]]> Sat, 4 Mar 2023 20:00:00 +0000 https://cumlord.instatus.com/maintenance/cleudkdat159012hpok8egwqy1y https://cumlord.instatus.com/maintenance/cleudkdat159012hpok8egwqy1y Updating Firmware on Nodes at RU-OMS-CHRNSH Type: Maintenance Duration: 16 minutes Affected Components: Paperless (paperless.vapronva.pw), WriteFreely (wf.vprw.ru), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), Mozilla Send (send.vapronva.pw), Umami (analytics.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Wakapi (wakapi.vapronva.pw), Obsidian Sync Database, Linx (ln.vprw.ru), GitLab Instance (gl.vprw.ru), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), Shlink (shlink.vapronva.pw) Feb 14, 08:31:19 GMT+0 - Completed - Maintenance has completed successfully. Feb 14, 08:19:17 GMT+0 - Identified - Maintenance is now in progress. Feb 14, 08:15:00 GMT+0 - Identified - Packages and other firmware will be updated on `BHO Core`, `Should Be Fine DOVH` and `Vapor Scaler` are going to be updated. Type: Maintenance

Duration: 16 minutes

Affected Components: , , , , , , , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 14</var>, <var data-var='time'>08:31:19</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Feb <var data-var='date'> 14</var>, <var data-var='time'>08:19:17</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress..</p> <p><small>Feb <var data-var='date'> 14</var>, <var data-var='time'>08:15:00</var> GMT+0</small><br><strong>Identified</strong> - Packages and other firmware will be updated on `BHO Core`, `Should Be Fine DOVH` and `Vapor Scaler` are going to be updated..</p> ]]> Tue, 14 Feb 2023 08:15:00 +0000 https://cumlord.instatus.com/maintenance/cle3z1hvo922713kamxf77n2qgx https://cumlord.instatus.com/maintenance/cle3z1hvo922713kamxf77n2qgx Major Outage on RU-OMS-CHRNSH Location Type: Incident Duration: 22 hours and 8 minutes Affected Components: Paperless (paperless.vapronva.pw), Wakapi (wakapi.vapronva.pw), Cumlord Mail (mail.cumlord.ru), Appwrite API (api.cumlord.ru), GitLab Instance (gl.vprw.ru), Mastodon (mastodon.vapronva.pw), Matrix (matrix.vapronva.pw), LanguageTool Server (languagetool.supplemental-services.vapronva.pw), PeerTube (peertube.vapronva.pw), Pixelfed (pixelfed.vapronva.pw), Mozilla Send (send.vapronva.pw), Shlink (shlink.vapronva.pw), Umami (analytics.vapronva.pw), Bitwarden (bitwarden.cumlord.ru), Obsidian Sync Database, WriteFreely (wf.vprw.ru), Linx (ln.vprw.ru) Feb 10, 07:15:00 GMT+0 - Investigating - We are currently investigating this incident. The location is completely unavilable. Feb 10, 07:25:00 GMT+0 - Identified - The provider (JSC Smartkom; `AS42676`) is experiencing problems with their BGP prefix announcement. Feb 10, 07:30:22 GMT+0 - Identified - According to several looking glasses, the AS completely lost its upstreams. I am staying in communication with the ISP. Feb 10, 07:32:00 GMT+0 - Monitoring - The AS' prefixes are being announced correctly. A few regional ISPs are routing successfully but via incredibly long routes. Feb 10, 07:37:07 GMT+0 - Resolved - All routes are being announced correctly, routing is done without any issues (tested from several looking glasses). Type: Incident

Duration: 22 hours and 8 minutes

Affected Components: , , , , , , , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>07:15:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. The location is completely unavilable..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>07:25:00</var> GMT+0</small><br><strong>Identified</strong> - The provider (JSC Smartkom; `AS42676`) is experiencing problems with their BGP prefix announcement..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>07:30:22</var> GMT+0</small><br><strong>Identified</strong> - According to several looking glasses, the AS completely lost its upstreams. I am staying in communication with the ISP..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>07:32:00</var> GMT+0</small><br><strong>Monitoring</strong> - The AS' prefixes are being announced correctly. A few regional ISPs are routing successfully but via incredibly long routes..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>07:37:07</var> GMT+0</small><br><strong>Resolved</strong> - All routes are being announced correctly, routing is done without any issues (tested from several looking glasses)..</p> ]]> Fri, 10 Feb 2023 07:15:00 +0000 https://cumlord.instatus.com/incident/cldy7bbrh215968icmx3wxlvzmd https://cumlord.instatus.com/incident/cldy7bbrh215968icmx3wxlvzmd Updating Firmware on C-DNS Nodes Type: Maintenance Duration: 7 minutes Feb 10, 06:31:00 GMT+0 - Identified - `Waterfall` node is being updated. Feb 10, 06:34:00 GMT+0 - Identified - `Waterfall` node is back online. `Kettle` node is now being updated. Feb 10, 06:36:46 GMT+0 - Completed - Maintenance has completed successfully. Feb 10, 06:30:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. `Waterfall` and `Kettle` nodes are going to be enrolled to the Ubuntu Pro program, Docker and other packages will be updated, as well as new Linux kernel, resulting in temporary outages. The update process will be done one-by-one to not cause any unresolvable DNS requests. Type: Maintenance

Duration: 7 minutes

<p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>06:31:00</var> GMT+0</small><br><strong>Identified</strong> - `Waterfall` node is being updated..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>06:34:00</var> GMT+0</small><br><strong>Identified</strong> - `Waterfall` node is back online. `Kettle` node is now being updated..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>06:36:46</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time. `Waterfall` and `Kettle` nodes are going to be enrolled to the Ubuntu Pro program, Docker and other packages will be updated, as well as new Linux kernel, resulting in temporary outages. The update process will be done one-by-one to not cause any unresolvable DNS requests..</p> ]]> Fri, 10 Feb 2023 06:30:00 +0000 https://cumlord.instatus.com/maintenance/cldy5m3dn152695v1mxywvvvssu https://cumlord.instatus.com/maintenance/cldy5m3dn152695v1mxywvvvssu Partial Outage of the “Waterfall” Node Type: Incident Duration: 33 minutes Jan 31, 06:47:00 GMT+0 - Investigating - We are currently investigating this incident. As of now, seems like the `Waterfall` node is unresponsive to some requests. Jan 31, 07:02:00 GMT+0 - Identified - The abnormal CPU and disk usage has been identified at the `Waterfall` node. ![](https://ln.vprw.ru/f/6euzuxqi.png) The root cause is declared to be a MariaDB process that leaked memory bit-by-bit over the past days leading to no RAM being left for other processes. ![](https://ln.vprw.ru/f/iuv5llfs.png) Jan 31, 07:05:00 GMT+0 - Monitoring - The fix has been implemented (reloading the node stack). Jan 31, 07:20:00 GMT+0 - Resolved - The issue has been resolved — the node is replying to requests normally. To remove any potential future occurrence of that issue, a notification system for anomalies has been added and tested to the infrastructure stack. Type: Incident

Duration: 33 minutes

<p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>06:47:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. As of now, seems like the `Waterfall` node is unresponsive to some requests..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>07:02:00</var> GMT+0</small><br><strong>Identified</strong> - The abnormal CPU and disk usage has been identified at the `Waterfall` node. ![](https://ln.vprw.ru/f/6euzuxqi.png) The root cause is declared to be a MariaDB process that leaked memory bit-by-bit over the past days leading to no RAM being left for other processes. ![](https://ln.vprw.ru/f/iuv5llfs.png) .</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>07:05:00</var> GMT+0</small><br><strong>Monitoring</strong> - The fix has been implemented (reloading the node stack)..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>07:20:00</var> GMT+0</small><br><strong>Resolved</strong> - The issue has been resolved — the node is replying to requests normally. To remove any potential future occurrence of that issue, a notification system for anomalies has been added and tested to the infrastructure stack..</p> ]]> Tue, 31 Jan 2023 06:47:00 +0000 https://cumlord.instatus.com/incident/cldjw0cu246383rbohjvvk94ih https://cumlord.instatus.com/incident/cldjw0cu246383rbohjvvk94ih