On June 4th 2024 at 13:33 UTC, reports were not loaded for various customer workspaces.
In line with our continuous efforts to enhance the performance of our reporting system, we identified a key area for improvement within the report loading functionality. We have optimized this service to load specific data segments associated with the workspace in parallel.
The side-effect of this change caused several requests to be in waiting state during a steady load on the system.
We’ve identified the issue after monitoring the release, and reverted the change into production. Leading to a healthy state 30 minutes after the incident started, at 14:00 UTC.
After mitigation, we identified the issue introduced with the new change and we are putting our efforts to fix it. We’ll also extend the coverage of our QA process to also include performing load tests ensuring better reliability.