CommCare HQ Degraded Performance
Incident Report for CommCare HQ
Postmortem

Overview

CommCare HQ experienced a period of degraded performance which affected Web Apps users when submitting forms or attempting to sync. Some users experienced increased slowness and intermittent errors as a result.

Summary of the incident

On November 23, 20:05 UTC, CommCare HQ experienced a period of degraded performance, and our engineering team was alerted and started an investigation to identify the cause of the degraded performance. During the investigation, our engineers observed that one of our databases had an issue with memory and immediate actions were taken to resolve the issue. The issue was resolved at 20:45 UTC.

Our Next Steps

To prevent a re-occurrence of a similar incidence, our engineering team has tuned the database to better handle an increase in requests.

We sincerely apologize for the service disruption you may have experienced. Thank you for your patience and support. Please reach out to support@dimagi.com if you have further questions about the incident.

Posted Nov 26, 2020 - 17:54 UTC

Resolved
Dear user, we have resolved the degraded performance issue on CommCare HQ. Once more, we apologize for the inconvenience this has caused.
Posted Nov 23, 2020 - 20:45 UTC
Update
We are continuing to investigate this issue.
Posted Nov 23, 2020 - 20:35 UTC
Update
We are continuing to investigate this issue.
Posted Nov 23, 2020 - 20:25 UTC
Investigating
Dear user, we are currently investigating a performance issue on CommCare HQ. Web Apps users may experience intermittent errors. Please refresh, if you are getting an error. We apologize for the inconvenience that this issue may be causing and we are actively working towards its resolution.
Posted Nov 23, 2020 - 20:05 UTC
This incident affected: www.commcarehq.org (Syncs with CommCare HQ, Web Apps).