Incident Description

On 2024-12-11 at 08:00 UTC, multiple customers in the EU region experienced incorrect KPI values. No customer data was affected. The service disruption lasted approximately 10 hours and 30 minutes until 2024-12-11 at 18:30 UTC.

Incident Resolution

Our team worked diligently to address the issue. We identified the affected customers and implemented a solution that corrected the KPI values. By 18:30, we had verified that all KPI values were accurate and the issue was entirely resolved.

Root Cause Analysis

The root cause of the problem was an overload of KPI calculation jobs that were not being processed due to limited resources and processing time. As the jobs grew, processing resources and processing time remained constant. We reached a point where calculations of KPI jobs for multiple customers were being dropped hence giving outdated KPI values.

Preventative Measures

We have identified the combination of factors that led to this oversight and will enforce the following practices to prevent similar issues in the future:

• Increasing our system's capacity to handle a more significant number of KPI calculation jobs.
• We have created an automated fallback script that can run the leftover jobs manually, decreasing downtime if the incident happens again.
• We are already working on a much more efficient solution. When it is completed, we will replace the current KPI calculation implementation.