Timeline

At 12:15pm UTC, we were notified of increased response times on a number of our Edge API endpoints. Investigation showed nothing immediately obvious but we suspected that it could be caused by Sentry, our APM tool. We set about removing the Sentry initialisation from our code and deployed it as soon as we could.

At 12:48pm UTC, this change was deployed and we observed the response times decrease immediately.

At 12:52pm UTC our monitoring confirmed that the average response time had returned to normal.

Next Steps

• Look into improvements to reduce / remove the impact of Sentry issues on our Edge API.

  • Decrease the shutdown timeout of the Sentry SDK.
  • Look at using Sentry relay to remove the impact on core Edge API services.

‌

• Add integration tests to simulate performance degradation / outages from all downstream services.