Overview
Between 06:00 UTC and 07:00 UTC on April 12, 2021, CommCare HQ experienced and outage primarily affecting form submissions and syncs on CommCare HQ. Additionally, users may have also experienced errors when navigating to reports, settings or formbuilder. The root cause was determined to be an incorrect configuration in one of our databases which was applied after a routine maintenance activity that would have ideally resulted in no downtime.

Summary of the incident

At approximately 06:24 UTC on April, 2021, our developers were alerted of an outage on CommCare HQ. Users were experiencing error messages when trying to sync or submit forms to the server. Additionally, some users also reported experiencing errors when navigating to reports, settings or formbuilder.

After initial investigation, our developers identified the root cause to be an incorrect configuration on one of our databases, that had been applied as part of routine maintenance that had been completed at 06:00 UTC. Our engineers immediately reverted the change once the root cause was identified, and by 07:00 UTC, the issue was fully resolved.

Our Next Steps

Our developers have implemented more stringent checks to ensure the correct configuration is applied before the maintenance going forward.

We apologize for the inconvenience this caused a number of users. Thank you for your patience and support! Please reach out to support@dimagi.com if you have further questions about the incident.

– CommCare HQ Support Team