Between September 19, 01:46 PM UTC and September 20, 03:25 PM UTC, customers experienced issues with missing Subscriptions in the Inventory
To continuously improve our security and standards in our services, we replaced a vulnerable library from our GraphQL API service. A bug in the code change resulted in undetected side effects when parsing incoming JSON data sent to the GraphQL API.
As a consequence, Fact Sheet that have been changed in that time frame, did not contain the latest subscription state in the respected events created from the update. This affected the two downstream services that (1) provide the Inventory Search and (2) the FACT_SHEET_UPDATED events via Webhooks. No subscription data has been lost during that time, as the latest state has always been correct in the main database. However, it was not propagated correctly to the aforementioned components. Only Fact Sheets are affected, which have been updated during the incident time window.
The missing subscriptions were added again through our Customer Support team by refreshing the search index so that the subscriptions reappeared in the Inventory. Any upcoming Webhooks event will contain the proper state, including the subscriptions with upcoming updates.
Following the mitigation, we did an in-depth analysis why our CI/CD pipeline did not identify the bug after removal of the library. We already implemented additional specific test cases to cover these unforeseen scenarios.
To detect such scenarios quicker, we are going to review our monitoring and alerting systems as well and implement further enhancements.
Update: As of 26.09.2023, 7 AM UTC, we can confirm that all customer workspaces see the correct subscription information in the Inventory.