On Wednesday, October 06, 2021, at 01:05 UTC, CommCare HQ suffered an outage where it was inaccessible to our users for approximately 1 Hour and 15 minutes.
Our hosting service provider suffered a loss of power to some of their machines and networking devices at their data center, which led to an outage on our primary operational database services.
On October 06, 2021, at 01:05 UTC, CommCare HQ experienced a sitewide outage and our team was alerted of the issue.
By 01:10 UTC, our engineers had started investigating the issue to identify a root cause, however, there were no indications or warning signs of any issues with our infrastructure, except for the primary database services not responding correctly to requests. Our engineers checked the status of services with our hosting provider, and there were no indications of an issue on their end either. Therefore, our engineers continued their investigation of our infrastructure.
At 1:30 UTC, our engineers continued to investigate the issue.
At 01:46 UTC, our hosting service provider posted an update on the network connectivity issues they were facing, and our engineers were immediately able to tie the CommCare HQ outage to the outage with the hosting provider services.
At 02:34 UTC, the hosting service provider provided an update that services were restored on their end and CommCare HQ was back online for all users.
#1 Why did CommCare HQ face an outage?
The outage on CommCare HQ was a direct result of our hosting provider facing a power outage on their machines on which our primary database is hosted.
Our engineering team has filed a ticket with our hosting service provider to obtain more details on the incident. Our team remains dedicated to de-risking any possible future issues by working with the hosting provider to determine future courses of action, should such a failure occur again with the provider.
We apologize for the inconvenience this caused our users. Thank you for your patience and support! Please reach out to email@example.com if you have further questions about the incident.
– CommCare HQ Support Team