Service Disruption in OData Integration for all regions

Incident Description

On June 18, 2025, a partial outage affected the OData feature. The issue impacted only a subset of workspaces and caused validation errors when building the OData metadata. The disruption lasted from 07:30 UTC to 14:20 UTC.

Incident Resolution

After receiving customer reports and confirming the root cause, the team rolled back the recent release. This fully restored the service without any data loss.

Root Cause Analysis

A change in the data processing pipeline introduced a sanitation step that unintentionally modified the structure of the GraphQL response used by the OData feature. Although the change was intended to normalize data inconsistencies between different backend implementations that served the same data, it was not correctly gated behind a feature flag. The sanitation caused unexpected values that violated OData’s metadata schema, leading to validation errors for some Inventory Saved Searches. Existing tests did not capture this edge case.

Preventative Measures

• Improve monitoring of the OData integration and Excel export.
• Extend integration test coverage to include more complex configurations.
• Ensure stricter enforcement of feature flag boundaries during implementation and reviews.
• Add runtime validation and clearer error reporting in the metadata generation process.
• Improve coordination and rollout planning for future data source transitions.