When: 9:05 - 10:04a KST July 26 (6:05 - 7:04p PDT July 25; 1:05 - 2:04a GMT July 26) Impact: Degraded performance of chat messaging features such as push notifications and webhooks. No impact to API and websocket connections. Cause: RabbitMQ could not create enough connections to support the number of connection requests that were created as a result of instance scale up changes. Remediation: Reduced number of instances to alleviate the number of connection requests until overall system load returned to standard operational levels. Further investigation is on-going to improve scale up methodology and transparency.
Posted Jul 26, 2022 - 10:05 KST
Investigating
We're experiencing an elevated level of API errors and are currently looking into the issue.