Degraded performance in various product capabilities

Incident Report for SAP LeanIX

Postmortem

Incident Description

On December 20th, Survey and Navigation service faced a partial disruption in some regions for approximately 84 minutes [09:33 UTC] - [10:57 UTC].

Some customers experienced degraded performance, where requests to the database were either failing or taking longer to execute, resulting in certain actions, such as navigation and surveys, not being completed as expected.

Incident Resolution

A fix was implemented on December 20, 2024, at 10:27 UTC, restoring the services to full operational status. No data was lost during the disruption.

Root Cause Analysis

We implemented an enhancement in Surveys aimed at improving service monitoring and observability. However, this change significantly utilized infrastructure resources, leading to performance degradation in some other services.

Preventative Measures

Enhanced our observability best practices to manage such cases better.

Posted Dec 23, 2024 - 14:11 UTC

Resolved

The incident has been resolved.

Posted Dec 20, 2024 - 10:58 UTC

Update

The root cause has been identified, a fix has been deployed, and we are monitoring the result.

Posted Dec 20, 2024 - 10:53 UTC

Monitoring

The root cause is found and we are monitoring the result.

Posted Dec 20, 2024 - 10:50 UTC

Identified

Users may experience degraded performance in various product capabilities. Our team is working to identify the root cause and implement a solution.

We will send an additional update in 60 minutes.

Posted Dec 20, 2024 - 10:42 UTC

This incident affected: CA Instances (EAM), AU Instances (EAM), CH Instances (EAM), AE Instances (EAM), UK Instances (EAM), BR Instances (EAM), and SG Instances (EAM).