Summary

Between March 11, 02:49 PM UTC and March 12, 10:00 AM UTC, processing time of webhooks events increased significantly - resulting in delayed event deliveries towards our customers.

What happened?

The underlying database behind the webhooks service was getting slower, even though enough resources were available to the database. The database process that takes care of updating statistics to build optimized queries was not executed for a longer period of time due to an unoptimized configuration, which caused the database to execute inefficient queries.

Mitigation: What did we do about it?

The process of updating the database query statistics was triggered manually.

Follow-ups: How will we improve?

After the incident, the alerting on the webhooks event processing was improved. To resolve the problem, we improved the database configuration.