[EAM] Service disruption loading translations

Incident Report for SAP LeanIX

Postmortem

Summary

Between September 20, 09:37 AM UTC and 10:54 AM UTC, workspaces with custom translations experienced errors when opening the application. Language files weren't loaded, resulting in labels not being translated.

What happened?

As part of our ongoing efforts to improve the usability of our application, we keep our language files up to date. One of these updates broke the logic used to load our translations into a workspace.

Mitigation: What did we do about it?

Once the team was notified of the incident and identified it. They rolled back the broken code, which resolved the situation.

Follow-ups: How will we improve?

After mitigation, we did a thorough analysis of why our CI/CD pipeline did not detect the bug. To detect such scenarios quicker, we are going to review our monitoring and alerting systems as well and implement further enhancements.

Posted Sep 27, 2023 - 08:36 UTC

Resolved

This incident has been resolved.
Posted Sep 20, 2023 - 12:26 UTC

Identified

The issue has been identified, we are working on a fix.
Posted Sep 20, 2023 - 10:40 UTC

Investigating

We are currently experiencing a service disruption in EAM. Customers experience problems loading translations. We are investigating the issue
Posted Sep 20, 2023 - 10:06 UTC
This incident affected: EU Instances (EAM), US Instances (EAM), CA Instances (EAM), AU Instances (EAM), DE Instances (EAM), CH Instances (EAM), AE Instances (EAM), and UK Instances (EAM).