On November 28th 2022 between 14:50 and 16:10 UTC, users that were trying to access our products in their browser saw nothing but a white screen for 30 seconds or even 2.8 minutes in the worst case. After this unusually long delay, the LeanIX web services would load as usual.
By analyzing the network traffic of web applications, we quickly found that an HTTP request to hello.myfonts.net, which is the provider of the font you see in our products, was taking a long time to respond. This request is not used for loading the actual font assets, which are stored directly on our servers, but as a mechanism for myfonts to know how many page loads LeanIX is generating with their font on it.
The request to hello.myfonts.net was sent from within our entry point CSS file, which contains the styles to be applied to our web applications. The CSS @import rule used there blocks the rendering of a website until the request is complete. This resulted in the long loading times of the LeanIX web applications.
We have removed the HTTP request to hello.myfonts.net from all our products and verified that this was the only @import CSS rule to third parties in our codebase.
In addition to this, we now have an automated alert in place, which will notify our engineers in case of such a spike in loading times in the future, so that we can react as quickly as possible.