\n<\/p>\n
CrowdStrike has published a post incident review (PIR) of the buggy update it published that took down 8.5 million Windows machines last week. The detailed post blames a bug<\/a> in test software for not properly validating the content update that was pushed out to millions of machines on Friday. CrowdStrike is promising to more thoroughly test its content updates, improve its error handling, and implement a staggered deployment to avoid a repeat of this disaster.<\/p>\n<\/div>\n CrowdStrike\u2019s Falcon software is used by businesses around the world to help manage against malware and security breaches on millions of Windows machines. On Friday, CrowdStrike issued a content configuration update for its software that was supposed to \u201cgather telemetry on possible novel threat techniques.\u201d These updates are delivered regularly, but this particular configuration update caused Windows to crash. <\/p>\n<\/div>\n CrowdStrike typically issues configuration updates in two different ways. There\u2019s what\u2019s called Sensor Content that directly updates CrowdStrike\u2019s own Falcon sensor that runs at the kernel level in Windows, and separately there is Rapid Response Content that updates how that sensor behaves to detect malware. A tiny 40KB Rapid Response Content file caused Friday\u2019s issue.<\/p>\n<\/div>\n Updates to the actual sensor don\u2019t come from the cloud, and typically include AI and machine learning models that will allow CrowdStrike to improve its detection capabilities over the long term. Some of these capabilities include something called Template Types, which is code that enables new detection and is configured by the type of separate Rapid Response Content that was delivered on Friday.<\/p>\n<\/div>\n On the cloud side CrowdStrike manages its own system that performs validation checks on content before it\u2019s released to prevent an incident like Friday from happening. CrowdStrike released two Rapid Response Content updates last week, or what it also calls Template Instances. \u201cDue to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data,\u201d says CrowdStrike.<\/p>\n<\/div>\n While CrowdStrike preforms both automated and manual testing on Sensor Content and Template Types, it doesn\u2019t appear to do as much thorough testing on the Rapid Response Content that was delivered on Friday. A March deployment of new Template Types provided \u201ctrust in the checks performed in the Content Validator,\u201d so CrowdStrike appears to have assumed the Rapid Response Content rollout wouldn\u2019t cause issues.<\/p>\n<\/div>\n This assumption led to the sensor loading the problematic Rapid Response Content into its Content Interpreter and triggering an out-of-bounds memory exception. \u201cThis unexpected exception could not be gracefully handled, resulting in a Windows operating system crash (BSOD),\u201d explains CrowdStrike.<\/p>\n<\/div>\n To prevent this from happening again, CrowdStrike is promising to improve its Rapid Response Content testing by using local developer testing, content update and rollback testing, alongside stress testing, fuzzing, and fault injection. CrowdStrike will also perform stability testing and content interface testing on Rapid Response Content.<\/p>\n<\/div>\n CrowdStrike is also updating its cloud-based Content Validator to better check over Rapid Response Content releases. \u201cA new check is in process to guard against this type of problematic content from being deployed in the future,\u201d says CrowdStrike.<\/p>\n<\/div>\n On the driver side, CrowdStrike will \u201cenhance existing error handling in the Content Interpreter,\u201d which is part of the Falcon sensor. CrowdStrike will also implement a staggered deployment of Rapid Response Content, ensuring that updates are gradually deployed to larger portions of its install base instead of an immediate push to all systems. Both the driver improvements and staggered deployments have been recommended by security experts<\/a> in recent days.<\/p>\n<\/div>\n