The CrowdStrike and Windows Outage (BSOD) – A Comprehensive Breakdown

A major software outage involving CrowdStrike and Windows in the past week captured global attention. Critical services worldwide—including airlines, banks, supermarkets, police departments, hospitals, and TV channels—were affected by the incident, resulting in widespread disruption.

This article delves into the incident, its root causes, the arduous recovery process, the parties responsible, and the lessons software engineers can learn from it.

The Outage: A Recap

On July 19, 2024, a colossal software failure caused millions of Windows 10 and 11 machines to crash with the infamous “Blue Screen of Death.” The impact was unprecedented, affecting essential services globally, from the United States to Europe, Asia, and Australia.

Airports were thrown into chaos, with Alaska’s emergency services number going offline, and UK’s Sky News unable to broadcast.

In Japan, McDonald’s had to close several outlets due to inoperative cash registers. Even Formula One’s Mercedes team faced issues at the Hungarian Grand Prix, highlighting the widespread ramifications of this outage.

The businesses impacted by this crisis were all clients of CrowdStrike, a leading cybersecurity firm specializing in endpoint security. The issue stemmed from a problematic update to CrowdStrike’s Falcon product, which led to the crash of approximately 8.5 million Windows machines.

Delta Airlines was particularly hard hit, with around a third of its flights canceled over three days, leading to substantial financial losses and reputational damage.

Understanding the Root Cause

CrowdStrike issued an update attempting to improve the detection of malicious processes, specifically targeting named pipes in Windows.

Named pipes facilitate inter-process communication, and the new rules aimed to enhance security by identifying suspicious activity.

However, the update introduced a critical error in the CSAgent.sys process, which attempted to write to an invalid memory address, resulting in system crashes.

The problematic file, identified as “C-00000291-*.sys,” contained configuration rules that triggered a logic error, causing the operating systems to crash. Despite CrowdStrike’s swift identification and isolation of the issue, the damage had already been done.

The Recovery Process: Slow and Manual

Recovering from this massive outage was an incredibly slow process, as each affected machine required manual intervention. IT staff had to physically access and fix each device, navigating to the CrowdStrike directory and deleting the offending file.

This process was time-consuming and labor-intensive, highlighting the challenges of mitigating such a widespread system failure.

While some developers created tools to expedite the recovery process, the scale of the outage meant that millions of machines remained non-functional for days.

By the fourth day, a significant number of devices were still awaiting repair, underscoring the complexity of recovering from such a large-scale incident.

Assigning Responsibility

The immediate responsibility for the outage lies with CrowdStrike. The company’s update process, which lacked adequate testing and canarying, directly led to the crashes. However, the incident also raises questions about Microsoft’s role and broader regulatory implications.

CrowdStrike’s Role

CrowdStrike’s update process skipped crucial testing phases. Questions about whether the changes were adequately tested, whether there was a staged rollout, and whether the company assumed that “content” files couldn’t cause system crashes remain unanswered.

Additionally, previous similar incidents should have served as a warning to improve their processes.

Microsoft’s Role

Microsoft’s Windows operating system allows third-party software, like CrowdStrike, to run at the kernel level, which poses inherent risks. Unlike Apple, which restricts third-party software to user space, Microsoft’s approach is influenced by regulatory requirements.

In 2009, Microsoft agreed to provide security software vendors the same level of access to Windows as its security tools, preventing it from restricting third-party access to the kernel space.