On August 4, 2025, users experienced a disruption to our Store Catalog system. 22 security reports disappeared from the catalog, making them unavailable to users. At the same time, multiple surveys also became inaccessible through our platform.
The incident was first noticed during a routine review when we observed that several assets were missing from the live catalog. This triggered our emergency response procedures to investigate and resolve the issue.
After evaluating multiple recovery options, we chose to restore our database from a backup taken before the incident occurred. This approach ensured:
The restoration process required was successfully completed on the morning of August 5th.
Our automated archival system contained a flawed database query that updated more records than intended. Instead of archiving only the specific reports that met certain criteria, the faulty code archived additional items, including the surveys.
To prevent similar incidents in the future, we are implementing the following improvements: