Alert Ingestion Delays Causing Failing Status Checks

Incident Report for Red Canary

Resolved

This incident has been resolved and all backlogged alerts processed.
Posted Feb 10, 2025 - 23:02 UTC

Update

We have successfully deployed a fix for the alert ingestion issue and alerts are now being ingested. Detections may still be delayed while we process the backlog of failed alerts. In addition, status checks that appeared to fail during this incident will automatically resolve; no customer action is required. We will continue to monitor the situation and will close this incident once all backlogged alerts have been ingested.
Posted Feb 10, 2025 - 20:42 UTC

Monitoring

We have successfully deployed a fix for the alert ingestion issue and alerts are now being ingested. Detections may still be delayed while we process the backlog of failed alerts. In addition, status checks that appeared to fail during this incident will automatically resolve; no customer action is required. We will continue to monitor the situation and will close this incident once all backlogged alerts have been ingested.
Posted Feb 10, 2025 - 20:41 UTC

Identified

We have identified an issue causing some alerts to fail ingestion, which may result in failing status checks for certain customers. Our team is actively testing a fix and assessing its impact. No alert data has been lost and all failed alerts will be reprocessed once the fix is implemented. Please note that detections based on these alerts may be delayed until ingestion is fully restored. We appreciate your patience and will provide further updates as they become available.
Posted Feb 10, 2025 - 20:00 UTC
This incident affected: Alert Ingestion and Correlation (SentinelOne, Microsoft, Proofpoint TAP, Crowdstrike Falcon, Lacework, Carbon Black Cloud) and Detections.