CrowdStrike CEO apologizes for crashing IT systems around the world, details fix

News

Jul 20, 20244 mins

Security

Attempts to mitigate a novel Windows threat caused systems running CrowdStrike’s Falcon sensor to crash.

CrowdStrike CEO has apologized to the company’s customers and partners for crashing their Windows systems, and the company has described the error that caused the disaster.

“I want to sincerely apologize directly to all of you for today’s outage. All of CrowdStrike understands the gravity and impact of the situation,” CrowdStrike founder and CEO George Kurtz wrote in a blog post on the company’s website titled “Our Statement on Today’s Outage.”

He reiterated the company’s earlier message that the incident, which brought down computers around the world on Friday, July 19, was not the result of a cyberattack.

But he played with words to suggest that there was no fault in the company’s Falcon security platform and to suggest that the incident was an accident.

What caused the CrowdStrike crash?

“The outage was caused by a defect found in a Falcon content update for Windows hosts,” Kurtz said, as if the defect was a naturally occurring phenomenon discovered by his staff.

The defective content update in question was pushed out to Windows machines running the company’s Falcon sensor at 04:09 UTC (0:09 Eastern Time) on Friday, with a fix pushed out just 79 minutes later, the company said on Saturday in a separate blog post providing technical details of the incident.

By then, of course, it was too late: Many of the systems that received that update were already offline.

“Systems running Falcon sensor for Windows 7.11 and above that downloaded the updated configuration from 04:09 UTC to 05:27 UTC – were susceptible to a system crash”, the blog post said.

In some cases, those crashes of systems running the Falcon sensor resulted in missed flights, closed call centers, and cancelled surgeries as many affected Windows systems displayed the infamous Blue Screen of Death.

Nevertheless, Kurtz insisted in his letter to customers, “There is no impact to any protection if the Falcon sensor is installed.”

That may be true for systems that didn’t receive the flawed content update, and strictly speaking a system that is no longer running doesn’t need protection, but affected customers will be questioning whether CrowdStrike truly protected their systems during those critical 79 minutes.

What was in CrowdStrike’s defective content update?

CrowdStrike updates configuration files for the endpoint sensors that are part of its Falcon platform several times a day. It calls those updates “Channel Files.”

The defect was in one it calls Channel 291, the company said in Saturday’s technical blog post. The file is stored in a directory named “C:WindowsSystem32driversCrowdStrike” and with a filename beginning “C-00000291-” and ending “.sys”. Despite the file’s location and name, the file is not a Windows kernel driver, CrowdStrike insisted.

Channel File 291 is used to pass the Falcon sensor information about how to evaluate “named pipe” execution. Windows systems use these pipes for intersystem or interprocess communication, and are not in themselves a threat — although they can be misused.

“The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 [command and control] frameworks in cyberattacks,” the technical blog post explained.

However, it said, “The configuration update triggered a logic error that resulted in an operating system crash.”

A quick fix, but a slow recovery

All it took to stop the problem reoccurring was to remove the defective content from the file: “CrowdStrike has corrected the logic error by updating the content in Channel File 291.”

That didn’t solve the problem for the many, many, Windows machines that had already downloaded the defective content then crashed, though.

For those, CrowdStrike published another blog post containing a far longer set of actions for affected customers to perform, with suggestions for remotely detecting and automatically recovering affected systems, with detailed sets of instructions for temporary workarounds for affected physical machines or virtual servers.

“Systems that are not currently impacted will continue to operate as expected, continue to provide protection, and have no risk of experiencing this event in the future,” the technical blog post concluded.

SUBSCRIBE TO OUR NEWSLETTER