Summary

On June 4th 2024 at 13:33 UTC, reports were not loaded for various customer workspaces.

What happened?

In line with our continuous efforts to enhance the performance of our reporting system, we identified a key area for improvement within the report loading functionality. We have optimized this service to load specific data segments associated with the workspace in parallel.

The side-effect of this change caused several requests to be in waiting state during a steady load on the system.

Mitigation: What did we do about it?

We’ve identified the issue after monitoring the release, and reverted the change into production. Leading to a healthy state 30 minutes after the incident started, at 14:00 UTC.

Follow-ups: How will we improve?

After mitigation, we identified the issue introduced with the new change and we are putting our efforts to fix it. We’ll also extend the coverage of our QA process to also include performing load tests ensuring better reliability.