Root Cause Analysis: Partial API Service Disruption – November 25, 2025
On November 25, 2025, at 07:48 UTC, some users began experiencing issues in parts of the Tonkean platform. These included problems with the main menu, unexpected logouts with a “Board not found” error, and failures when connecting to integrations. 09:08 UTC, the incident was resolved, and all services were restored.
The root cause was misconfigured API servers. Some of our backend systems had incorrect internal permissions. This caused specific requests routed to that server to be rejected, resulting in the observed errors. Restarting the service removed the faulty component and replaced it with a properly configured one.
This issue only affected users whose requests were routed to the problematic server, resulting in inconsistent but frustrating impacts. No data was lost, and all services were fully restored within 80 minutes.
To prevent similar problems in the future, we are adding automated checks that verify internal permissions are correctly set whenever services start. These checks will also include self-healing logic to fix misconfigurations when possible automatically.
Our systems are currently stable and are being closely monitored.