Resolved -
The incident has been resolved, all system fully operational and no degraded performance.
Nov 22, 21:27 UTC
Update -
We are continuing to monitor for any further issues.
Nov 22, 21:26 UTC
Update -
We are continuing to monitor for any further issues.
Nov 22, 20:24 UTC
Update -
The Element451 platform is now fully back online and operating normally. We will continue to closely monitor performance.
A few search-related features are currently experiencing degraded performance as our indexes continue to recover. We expect these to improve over time and are keeping a close eye on them.
We were able to restore from an early-morning snapshot and are currently reviewing for any potential data loss. Our team is using point-in-time continuous snapshots to identify and recover any missing data.
Preliminary investigation shows the root cause was a misconfiguration that allowed an automated test process to write to our production environment. We are updating our safeguards and deployment processes to prevent this from happening again.
Nov 22, 19:13 UTC
Monitoring -
Services are starting to come back online across Element451. Some sites may load more slowly than usual while our database continues to fully recover. We’re monitoring performance closely and will share another update once everything is stable.
Nov 22, 18:18 UTC
Update -
We’ve restored our primary database services and are now bringing all affected Element451 services back online. Some features may still be intermittently unavailable while we complete these updates. We’ll provide another update once all services are fully restored.
Nov 22, 17:17 UTC
Update -
We are continuing to work on a fix for this issue.
Nov 22, 17:01 UTC
Identified -
We are currently experiencing database issues that are impacting parts of the Element451 platform. Our team is actively working to restore normal performance and recover any affected data as quickly as possible. We’ll provide another update as soon as we have more information.
Nov 22, 16:15 UTC