Postmortem study guide - CrowdStrike faulty content update, July 2024¶

A real public postmortem repackaged as exam prep. Everything below comes from CrowdStrike's own published root cause analysis. Where that report does not say something (for example the exact clock times of the release), this page leaves it out rather than guess.

What happened¶

On 19 July 2024, CrowdStrike delivered a Rapid Response Content update (a new version of Channel File 291) to its Falcon sensor on Windows. On sensors that received it, the update caused a system crash. CrowdStrike's report says that as of 29 July at 5 p.m. PT, about 99% of Windows sensors were online compared with before the update, against a normal week-over-week variance of about 1%.

Two terms matter:

• Sensor Content ships inside a sensor release (code and compiled "Template Types").
• Rapid Response Content is configuration data delivered from the cloud as channel files. CrowdStrike states it is "configuration data; it is not code or a kernel driver."

Timeline (as the RCA states it)¶

The RCA gives no hour-by-hour timeline of the release. Those details are in CrowdStrike's separate preliminary post incident review, which this page does not use.

Root cause¶

flowchart TB
    A[IPC Template Type defines 21 input fields] --> D[Content Validator checks content assuming 21 inputs]
    B[Sensor integration code supplies only 20 inputs] --> E[Content Interpreter has no bounds check]
    C[Tests used a wildcard for the 21st field] --> F[Mismatch not caught before release]
    D --> G[19 July content reads the 21st value]
    E --> G
    F --> G
    G --> H[Out-of-bounds read, system crash]

The diagram shows three independent weaknesses combining into one crash.

The RCA names the combination: "the mismatch between the 21 inputs validated by the Content Validator versus the 20 provided to the Content Interpreter, the latent out-of-bounds read issue in the Content Interpreter, and the lack of a specific test for non-wildcard matching criteria in the 21^st field."

The report states the fault was an out-of-bounds read, "not an arbitrary memory write issue," and that this was independently reviewed.

Findings and what CrowdStrike changed¶

CrowdStrike also engaged two independent third-party vendors to review the sensor code, and is reviewing its quality process end to end.

📖 CrowdStrike - External Technical Root Cause Analysis, Channel File 291 - the primary source for this page (PDF, 6 August 2024).

What this teaches¶

Treat configuration as code¶

The update was "just content," yet it crashed machines. Anything that changes runtime behavior needs the same tests, review and staged rollout as a code release.

Staged rollout is a blast-radius control¶

The RCA's own fix is rings, canaries and bake-in time. Compare cell-based architecture and the canary lesson in the Cloudflare regex postmortem.

Validators must match the runtime¶

The validator assumed 21 inputs while the runtime supplied 20. Two components that each look correct can disagree. Contract tests across the boundary catch this.

Wildcard-only tests hide bugs¶

Every test used a wildcard in the one field that was broken. Test each input with a realistic, non-trivial value.

Fail safe in privileged code¶

The faulting module ran in the kernel at early boot, so a bad read became a crash. Bounds checks and fuzzing matter most where a fault cannot be recovered. This guide does not cover recovery steps for affected machines; the RCA does not either.

Cert mapping¶

The mapping below is this site's own editorial judgment, not from the RCA; the CKA / CKS row in particular is a conceptual stretch.

Discussion questions¶

1. Which of your own "just config" changes bypass your normal test and rollout path?
2. How would you detect a validator and a runtime that disagree about a schema?
3. What would a ring-based rollout look like for an agent running on every endpoint you own?
4. Which vendors can push changes into your production without your approval? Can you control the timing?
5. What tests would you write for a field that is currently always wildcarded?

Related¶

• Cloudflare regex postmortem
• GCP networking postmortem
• Cell-based architecture
• Chaos engineering patterns
• SRE roadmap