Outage on www.commcarehq.org

Incident Report for CommCare HQ

Postmortem

Overview

On Wednesday, October 06, 2021, at 01:05 UTC, CommCare HQ suffered an outage where it was inaccessible to our users for approximately 1 Hour and 15 minutes.

Our hosting service provider suffered a loss of power to some of their machines and networking devices at their data center, which led to an outage on our primary operational database services.

Response summary

On October 06, 2021, at 01:05 UTC, CommCare HQ experienced a sitewide outage and our team was alerted of the issue.

By 01:10 UTC, our engineers had started investigating the issue to identify a root cause, however, there were no indications or warning signs of any issues with our infrastructure, except for the primary database services not responding correctly to requests. Our engineers checked the status of services with our hosting provider, and there were no indications of an issue on their end either. Therefore, our engineers continued their investigation of our infrastructure.

At 1:30 UTC, our engineers continued to investigate the issue.

At 01:46 UTC, our hosting service provider posted an update on the network connectivity issues they were facing, and our engineers were immediately able to tie the CommCare HQ outage to the outage with the hosting provider services.

At 02:34 UTC, the hosting service provider provided an update that services were restored on their end and CommCare HQ was back online for all users.

Drilling down to the cause:

#1 Why did CommCare HQ face an outage?
The outage on CommCare HQ was a direct result of our hosting provider facing a power outage on their machines on which our primary database is hosted.

Next Steps

Our engineering team has filed a ticket with our hosting service provider to obtain more details on the incident. Our team remains dedicated to de-risking any possible future issues by working with the hosting provider to determine future courses of action, should such a failure occur again with the provider.

We apologize for the inconvenience this caused our users. Thank you for your patience and support! Please reach out to support@dimagi.com if you have further questions about the incident.

– CommCare HQ Support Team

Posted Oct 06, 2021 - 18:07 UTC

Resolved

Dear users, all services on www.commcarehq.org have fully been restored. In case you experience any issues or have any questions, please reach out to the Dimagi Support Team. We thank you for your patience while our service provider worked to restore services, and appreciate your understanding regarding this incident.

Posted Oct 06, 2021 - 02:53 UTC

Identified

Dear users, we have identified that the source of the issue is with our hosting service provider. We will provide an update on the situation within the hour or as we receive more information from our service provider. We sincerely appreciate your patience.

Posted Oct 06, 2021 - 02:23 UTC

Update

Dear users, we are continuing to investigate this issue as our top priority. We will provide an update in the next 15 mins regarding the progress of the issue. We sincerely appreciate your patience.

Posted Oct 06, 2021 - 01:59 UTC

Update

Dear users, our engineering team is continuing to address the outage with the highest priority. We sincerely apologize for the inconvenience that is being caused due to this outage. We will provide an update in the next 15 mins regarding the progress of the issue. We thank you for your continued patience during this time.

Posted Oct 06, 2021 - 01:41 UTC

Update

Dear users, we are continuing to investigate this issue and will have an update in the next 15 mins on the status of the investigation. We appreciate your patience while we work on this problem.

Posted Oct 06, 2021 - 01:21 UTC

Investigating

Dear users, we are currently experiencing a sitewide outage. Our engineering response team is working on a resolution as their highest priority. We appreciate your continued patience.

Posted Oct 06, 2021 - 01:18 UTC

This incident affected: www.commcarehq.org (Form Builder, App Release Management, User Management, Syncs with CommCare HQ, Form Submissions, Data Exports, Project Reporting, Report Builder, SMS / Messaging, Bulk Case Import, CommCare Data Export Tool, CommCare APIs, Data Forwarding, Web Apps, App Preview, Data Cleaning, Report an Issue).