Delay in custom code executions

Incident Description

On September 10, 2025, custom code executions were failing in the westeurope region, intermittently between 14:40 - 15:50 UTC.

Users might have noticed the impact in the following ways:

1. Calculation executions failing / delayed updates on fact sheets
2. Push Webhooks with Callbacks had delayed deliveries
3. “Add Filter with AI” in the inventory was not working

The incident was caused by a long-running migration, which blocked other operations on the database.

No data was lost.

Incident Resolution

We mitigated the failing long-running schema update by increasing the timeout for the update.

Root Cause Analysis

Code executions were failing because a long-running schema update blocked the database. This migration acquired an “exclusive lock” on the affected table, preventing other operations from succeeding. Code executions are logged and therefore require access to that table, causing execution requests to fail.

The affected database contains a large amount of data, which is why the migration took longer than expected.

Preventative Measures

We will add evaluating schema updates on the database more carefully to our review process, pre-determining the impact of the update, and running them without blocking other database operations.

In addition, we will keep the increased timeout, reducing the likelihood of a failed migration.