Summary

On June 11th 2024 between 4:35 UTC and 6:20 UTC, LeanIX Pathfinder API was inaccessible for customer workspaces.

What happened?

During the given timeframe, customers were not able to access workspaces. Due to a fault in our release pipeline, broken code was deployed to production, which resulted in an overall downtime of two hours for all customers and workspaces. A rollback was not possible, and a full release of the affected service was required.

Mitigation: What did we do about it?

Our team identified the issue and needed to prepare a full release to fix the broken code. Service availability was established by 6:20 UTC.

Follow-ups: How will we improve?

After the incident, we took steps to understand why the release was not stopped by our CI/CD pipeline checks and how we could prevent similar incidents in the future. We have identified several areas for improvement and are currently working on:

• optimizing our release tooling to allow a rollback to previously working versions
• improving alerting and uptime measuring, so we can detect outages quicker
• fixing the bug in our pipeline to properly prevent the release of branches with failing tests
• improving our pipeline to be more resilient against configuration mistakes and to detect them earlier