Overview

Our SMS carrier, Twilio, experienced an issue with their SMS delivery in which two-factor authentication tokens were delayed for users who have T-Mobile and Metro carriers. Users either received SMS tokens after several minutes and could not log in as the token had expired, or they did not receive tokens at all.

Summary of the incident

On December 3rd, 14:15 UTC, Twilio announced that SMS messages sent to T-Mobile and Metro carriers via the Twilio gateways were delayed. CommCare users who have these carriers and use “text message” as a form of two-factor authentication were prevented from logging into CommCare HQ because the two-factor authentication SMS tokens they received were expired by the time the SMS was sent. More details about the carrier incident can be found here. https://status.twilio.com/incidents/yhcqts4cbcdl

By 18:00 UTC, our engineers had devised a workaround to this carrier issue and implemented it. One of the limitations of this temporary solution was a reduction in SMS “throughput”, or the number of two-factor authentication messages we were able to send at any given time. While the acute issue was resolved for several hours for all users, by 21:15 UTC, the reduced throughput meant we were forced to revert back to the workflow that impacted T-Mobile and Metro carriers. Shortly after 22:00 UTC, our engineers had implemented a second, more resilient solution that removed the throughput limitation and resolved the issue.

Our Next Steps

We will continue to encourage our users to create backup tokens and save them, as they can be used when there is a delay with SMS token delivery. We will also emphasize the use of applications like Google Authenticator as they provide a more stable method of two-factor authentication.

Please reach out to support@dimagi.com if you have further questions about the incident.