Service Disruption in EAM
Incident Report for SAP LeanIX


On June 11th 2024 between 4:35 UTC and 6:20 UTC, LeanIX Pathfinder API was inaccessible for customer workspaces.

What happened?

During the given timeframe, customers were not able to access workspaces. Due to a fault in our release pipeline, broken code was deployed to production, which resulted in an overall downtime of two hours for all customers and workspaces. A rollback was not possible, and a full release of the affected service was required.

Mitigation: What did we do about it?

Our team identified the issue and needed to prepare a full release to fix the broken code. Service availability was established by 6:20 UTC.

Follow-ups: How will we improve?

After the incident, we took steps to understand why the release was not stopped by our CI/CD pipeline checks and how we could prevent similar incidents in the future. We have identified several areas for improvement and are currently working on:

  • optimizing our release tooling to allow a rollback to previously working versions
  • improving alerting and uptime measuring, so we can detect outages quicker
  • fixing the bug in our pipeline to properly prevent the release of branches with failing tests
  • improving our pipeline to be more resilient against configuration mistakes and to detect them earlier
Posted Jun 14, 2024 - 11:20 UTC

This incident has been resolved. We appreciate your patience and understanding.
Posted Jun 11, 2024 - 07:07 UTC
A fix has been implemented and we are monitoring the results.
Posted Jun 11, 2024 - 06:33 UTC
We are currently experiencing a service disruption in EAM. Our team is working to identify the root cause and implement a solution.

We will send an additional update in 60 minutes.
Posted Jun 11, 2024 - 05:28 UTC
This incident affected: EU Instances (EAM), US Instances (EAM), CA Instances (EAM), AU Instances (EAM), DE Instances (EAM), CH Instances (EAM), AE Instances (EAM), UK Instances (EAM), and BR Instances (EAM).