On Sunday (5/27) as the 76th and final match of the IPL was being played, the txtWeb platform experienced an outage. 

The duration of the match was about 4 hrs (from 8 pm IST to 12 am IST). At around 8:27 pm we started experiencing issues where many of the user requests started experiencing delays in response or receiving no response, this lasted until the end of the match. We got alerted about this immediately as expected from our end-to-end monitoring devices. 

Impact:

- Approximately half of our user base accessing txtWeb during the match using our primary aggregator (including our primary number 9243342000 as well as several local numbers). Based on expected traffic this would be close to 125k users. (exact figures are awaited from our SMS aggregator)

There were two things that went wrong: 

Platform slowness due to database bottlenecks:

• During the processing of a user requests, we store the users' request details and related information in a database server. Under high loads, the database writes started getting backlogged leading to building up a backlog of requests that were dropped or eventually serviced with delay. High backlog time also resulted in the db connections timing out and becoming unusable. This in turn resulted in more backlog leading to delayed user requests. 

Latency on outgoing messaging on our aggregator: 

• Even when the platform recovered, our SMS aggregator was unable to send out requests on a timely fashion leading to many of the requests getting delayed. We notified the aggregator which investigated and addressed the issue. However, we lost critical time during this period and the impact was felt by the users.  

Learnings: 

• We had done extensive load testing and performance tuning in preparation for the event. However due to the complexity of the ecosystem of sending and receiving the SMSes, several components were simulated. Some of the underlying assumptions broke down during actual load. For the scale we are observing we have implemented a faster write thru server architecture. In the short-term we have replaced direct database writes to a scaleable/faster db (Dynamo DB). This allows us to be able to write to the DB at high scale without facing backlogs. We are running several load tests in order identify various other scenarios.

• We have built the aggregator redundancy in place to allow the requests to be piped to different channels. This will go live soon and will give us redundancy with alternate aggregators.

We are overwhelmed with the number of requests we got from our users on that day, seeing peaks of 18K per minute from across India. Our sincere apologies to our users who tried accessing txtWeb during the final IPL match. This was an important learning experience for us and one which will help us improve and continue to strive to delight our users.