CrowdStrike issues Preliminary Review on Falcon Sensor and Windows OS Outage

CrowdStrike has issued a preliminary Post Incident Review (PIR) regarding the recent content configuration update that caused a Blue Screen of Death (BSOD) error on Windows systems.

This overview will be followed by a detailed Root Cause Analysis, to be made publicly available soon. The PIR uses general terms to describe the Falcon platform for clarity.

On Friday, 19 July 2024, at 04:09 UTC, CrowdStrike released a routine content configuration update for its Windows sensor to gather telemetry on emerging threat techniques. This update, however, caused a system crash on Windows hosts running sensor version 7.11 and above. The issue was quickly identified and reverted at 05:27 UTC the same day. Systems that connected after this timeframe or were offline during the update were not affected. Mac and Linux hosts were also unaffected.

CrowdStrike distributes security content configuration updates in two ways: Sensor Content, included in the sensor release, and Rapid Response Content, designed to quickly address emerging threats.

Sensor Content, part of the sensor release, includes AI and ML models and goes through extensive quality assurance (QA) processes, including automated and manual testing. These updates are thoroughly validated before being deployed in stages, starting with internal testing at CrowdStrike, followed by early adopters, and then general availability. Customers control the deployment of these updates via Sensor Update Policies.

Rapid Response Content performs behavioural pattern-matching operations on the sensor, represented as fields and values in a proprietary binary file. These updates, known as Template Instances, allow dynamic configuration without requiring sensor code changes. Despite rigorous testing, a bug in the Content Validator allowed a problematic update to pass, leading to the BSOD error.

Detailed Timeline

1. Sensor Content Release: On 28 February 2024, sensor version 7.11 introduced a new IPC Template Type to detect attacks using Named Pipes.
2. Template Type Stress Testing: The IPC Template Type passed stress testing in March 2024.
3. Template Instance Deployment: Between March and April 2024, several IPC Template Instances were successfully deployed.
4. Incident on 19 July 2024: Two new IPC Template Instances were deployed. Due to a Content Validator bug, one instance containing problematic data caused an out-of-bounds memory read, resulting in the BSOD.

Preventive Measures

To prevent future incidents, CrowdStrike is enhancing its software resiliency and testing processes:

1. Enhanced Testing:
   • Local developer testing
   • Content update and rollback testing
   • Stress testing, fuzzing, and fault injection
   • Stability and content interface testing
2. Improved Validation:
   • Adding new checks to the Content Validator to prevent deployment of problematic content.
   • Enhancing error handling in the Content Interpreter.
3. Staggered Deployment Strategy:
   • Gradual deployment of Rapid Response Content, starting with smaller, canary deployments.
   • Enhanced monitoring of sensor and system performance during deployments.
   • Providing customers with greater control over update delivery and detailed release notes.

CrowdStrike is committed to transparency and will publish a comprehensive Root Cause Analysis once the investigation is complete.