Delay in delivering webhooks

Incident Report for SAP LeanIX

Postmortem

Incident Description

On Wednesday, April 9. In the period of 12:56 - 19:15 UTC Webhooks could not deliver events. After 19:15 UTC all events were processed and delivered without data loss.

Incident Resolution

We identified the broken release and rolled it back. Webhooks continued to deliver events again starting from 19:15 UTC.

Root Cause Analysis

The broken release changed the order of initialisation for components in Webhooks. This caused events to be stuck without being processed.

Preventative Measures

To prevent similar incidents in the future, we aim to improve in the the following areas:

We will improve our observability to react earlier when the event delivery is not working properly.
We will enhance our existing tests to detect broken event delivery before it reaches production.

Posted Apr 14, 2025 - 15:11 UTC

Resolved

The incident has been resolved. All webhooks events were processed and are being delivered.

Posted Apr 09, 2025 - 20:13 UTC

Identified

We have identified delays in webhooks deliveries and are now processing through the backlog of events.

We will send an additional update in 30 minutes.

Posted Apr 09, 2025 - 19:33 UTC

This incident affected: EU Instances (EAM), US Instances (EAM), CA Instances (EAM), AU Instances (EAM), DE Instances (EAM), CH Instances (EAM), AE Instances (EAM), UK Instances (EAM), BR Instances (EAM), SG Instances (EAM), and JP Instances (EAM).