A CrowdStrike update is breaking computers running Windows, causing them to crash and display the Blue Screen of Death. Companies around the world have been unable to reboot, according to reports. Firms affected by the outage include Sky News, which has been unable to broadcast.
Concerned users have taken to forums such as Reddit to report the issue, with one user saying: “Wow, stuck in a boot loop, and entire org taken out.”
So if you got into work this morning and were met by frankly, carnage, know that you are not alone. Here’s what happened and what to do next.
What happened
As you might have gathered, an issue with CrowdStrike is causing the widespread global issue. CrowdStrike engineers say they are working on the issue, which affects its Falcon Sensor product. CrowdStrike calls Falcon “the CrowdStrike platform purpose-built to stop breaches via a unified set of cloud-delivered technologies that prevent all types of attacks—including malware and much more.”
The IT outage has affected airports, businesses and broadcasters, according to the Sky News website. Planes have been grounded in the U.S., trains in the U.K. are impacted, as well as boarding scanners at Edinburgh airport in Scotland.
Microsoft says it is taking “mitigation actions” after service issues it said started at about 6pm Eastern Time. The company says it is investigating issues with cloud services in the U.S. and “an issue impacting several of its apps and services,” Sky News says.
I have contacted CrowdStrike and Microsoft for a comment and will update this article when the firms reply.
While initial reports focused on a dodgy update, a user named Brody, who is director of CrowdStrike Overwatch posted on X, formerly Twitter that it is “a faulty channel file, so not quite an update.”
There is a workaround, he added.
1. Boot Windows into Safe Mode or WRE.
2. Go to C:WindowsSystem32driversCrowdStrike
3. Locate and delete file matching “C-00000291*.sys”
4. Boot normally.
What To Do
It’s not easy to say what to do next, as while there is a workaround, it’s not scalable as it would need to be applied manually, system by system. In a large company, this could mean it takes hours or more to get back up and running.
By its nature the issue is going to be very hard to resolve once systems are in a reboot loop, says Adam Harrison, managing director at FTI Cybersecurity. “Manual fixes are going to take time for system admins to apply: CrowdStrike can’t push a new update remotely to fix. It’s going to need manual intervention on each system.”
You might be lucky and be able to roll back to known good states but the majority won’t have anything that supports doing that, says Harrison. “The fix itself is quick to perform, but when you scale that up to thousands of servers and/or thousands of workstations, it’s going to be a bad day in the office for lots of folks.”
It’s also going to be a bad day for CrowdStrike. What can the firm do to help people?
“They can only communicate that fix as quickly and widely as they can,” says Harrison. “My assumption would be that the update is already down, so any systems which hadn’t updated for any reasons shouldn’t still get pushed a bad update.”
Ian Thornton-Trump, CISO at Cyjax says CrowdStrike “will certainly do their very best to pull the update and instruct the old agents not to update till they can get it sorted.”
However, he says, “what has been done can not be undone for those blue screen machines. If the machines can be booted in safe mode they may be able to issue an out of band update or patch. That’s time consuming—if the machines are critical, they might actually consider restoring from backup or a shadow copy (a built in MSFT recovery feature). Whatever path they have, they will try and fix as quickly as possible.”
CrowdStrike might be able to put a tool together that would apply the fix at the disk level, such as bootable media, says Harrison. “This would maybe help some people out who have thousands of systems to fix. It’s still not a solution that solves the problem fully remotely or at huge scale, but it could bring recovery times down.”
This is a breaking story. Keep your eyes peeled and check back to my Forbes page for updates.
