Infrastructure: Partial Outages and Degraded Performance

Incident Report for Red Canary

Resolved

This incident has been resolved.
Posted Sep 26, 2020 - 13:30 UTC

Update

We are continuing to monitor for any further issues.
Posted Sep 25, 2020 - 19:43 UTC

Monitoring

A fix has been implemented and we are monitoring system health and processing a backlog of data. Detections may be slightly delayed.
Posted Sep 25, 2020 - 18:16 UTC

Update

We are continuing to work on a fix for this issue.
Posted Sep 25, 2020 - 17:21 UTC

Update

A very small subset of customers may see error messages for their web portals. Other customers may experience degraded performance.
Posted Sep 25, 2020 - 17:21 UTC

Identified

We have identified the root cause of the issue and are evaluating fixes. API requests are now operational.
Posted Sep 25, 2020 - 17:12 UTC

Investigating

We are currently investigating an issue with Red Canary cloud infrastructure that is causing partial outages on customer web portals and API requests with degraded performance on telemetry ingestion.
Posted Sep 25, 2020 - 16:30 UTC
This incident affected: Web Portal, API Requests, Detections and Endpoint Telemetry Ingestion (VMware Carbon Black Response hosted by Red Canary, VMware Carbon Black Hosted EDR, VMware Carbon Black Cloud, CrowdStrike Falcon, Microsoft, Elastic Endpoint Security).