Incident Description

Between (UTC) Aug 7, 2024, 2:00:00 PM and 7:00 PM, there was an outage in the ServiceNow integration access to the configurations for mappings, making the integration unavailable for customers. After fixing the root cause, the integration configurations were again available, making the whole integration available without any data loss.

Incident Resolution

• An immediate attempt to revert to the previous configuration failed. Then, a more extended analysis of the root cause led to a definitive fix released, making the ServiceNow integration configurations accessible again.
• No data loss occurred since the problem was a temporary change in the service’s internal data format that was reverted, restoring the original data again (only ServiceNow integration configurations were affected and then fixed; no workspace data was involved).

Root Cause Analysis

• ServiceNow integration configuration was not readable during the incident because a planned change in the structure of those configurations had technical issues during its execution.
• Reverting that change as mitigation did not immediately solve the issue because the configurations had already been changed to the new format.

Preventative Measures

• Now the system handles both formats correctly, and configuration data is preserved using the expected format (the internal format will no longer be used in API communication but still supported for configs that may still present this format).

• Improve internal handling of the tools involved in this issue and move away from the one that caused unexpected technical issues when this problem was introduced.

• Improve internal alerts for this service availability. The service was running in this case, but the responses were not useful for the User Interface to work properly. Those cases will be considered as services unavailable for alerting purposes.