In the time from 13 Nov 2024, 09:04 PM UTC to 14 Nov 2024, 00:01 AM UTC, all customers experienced a disruption of data access through the OData bookmark endpoints.
During that time, the OData bookmark endpoints would return an excessive amount of 400 errors.
The change causing the service disruption was rolled back at 14 Nov 2024, 00:01 AM UTC.
The code change that introduced the service disruption extended logs in case of OData endpoint failures. It was small in size, looked harmless as it did not affect business logic and caused no errors during local testing.
We did not further test the code on a production environment and due to an unrelated issue, automated service monitoring was disabled.
We plan to reintroduce the service monitoring for 400 and 500 errors as soon as possible. We will also be extending the existing threshold-based alerts to include 400 errors.