[EAM] Service Disruption for Fact Sheet updates

Summary

Between September 19, 01:46 PM UTC and 7:20 PM UTC, customers experienced issues with saving Fact Sheets in the Inventory

What happened?

To continuously improve our security and standards in our services, we replaced a vulnerable library from our GraphQL API service. A bug in the code change resulted in undetected side effects when parsing incoming JSON data sent to the GraphQL API.

As a consequence, customers were unable to save changes to their Fact Sheets, if specific field types (Integer, Single Select) have been part of the updated section.

Mitigation: What did we do about it?

As soon as we identified the impact of Fact Sheet updates, we implemented a fix.

Follow-ups: How will we improve?

Following the mitigation, we did an in-depth analysis why our CI/CD pipeline did not identify the bug after removal of the library. We already implemented additional specific test cases to cover these unforeseen scenarios.

To detect such scenarios quicker, we are going to review our monitoring and alerting systems as well and implement further improvements.