Some of our customers have been utilizing a specific IP for their ServiceNow and Signavio integrations. In June 2023, we upgraded our infrastructure and informed our customers about the need to transition to new IP ranges. While the majority of customers have successfully migrated, a small number have yet to complete this transition.
On Tuesday, April 2nd at 12:46 UTC, during a routine maintenance procedure, the legacy IP unexpectedly became inaccessible.
The issue was first identified on Friday, April 5th at 8:32 UTC. Upon investigation, we discovered that the IP had been deallocated from Microsoft Azure and could not be reinstated. As a result, all affected customers will need to transition to the new IP ranges.
The incident occurred during a routine manual clean-up of legacy resources. Due to human error, the dedicated IP for the legacy proxy was inadvertently included in the batch of resources slated for deletion. This oversight was possible because the dedicated IP was not created through Infrastructure-as-Code, which would have necessitated a peer review for any changes affecting it.
Unfortunately, as we cannot request specific IPs from Microsoft (and there's no guarantee they would be available), it is impossible to recreate the previous configuration. The only viable solution to reestablish connectivity is to transition to our standard infrastructure, utilizing the new IP ranges.
To prevent such incidents in the future, our current infrastructure and any new infrastructure is set up through Infrastructure-as-Code. This requires peer-review and approvals for any changes, significantly reducing the likelihood of similar errors occurring.
Furthermore, we use an extra layer of protection to sensitive resources, such as IPs. These resources are delete-protected by default and require a two-step process for removal. This additional safeguard ensures that critical resources can only be removed after careful consideration and approval, further minimizing the risk of unintentional deletions.