New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints (2024)

The Microsoft Recovery Tool was updated 7/22/2024 as version 3.1. While fundamentally there are no functional changes to the tool, for the Recover from WinPE option, we have expanded the logging, reattempt logic, and error handling. For the Recover from safe mode for USB delivery, we’ve added in better user awareness cuesforwhen to run the repair command.

7/23/2024: Microsoft notes that CrowdStrike has updated its Remediation and Guidance Hub: Falcon Content Updates for Windows Hosts.

As a follow-up to the CrowdStrike Falcon agent issue impacting Windows clients and servers, Microsoft has released an updated recovery tool with two repair options to help IT admins expedite the repair process. The signed Microsoft Recovery Tool can be found in the Microsoft Download Center: https://go.microsoft.com/fwlink/?linkid=2280386. In this post, we include detailed recovery steps for Windows client, servers, and OS's hosted on Hyper-V. The two repair options are as follows:

• Recover from WinPE – this option produces boot media that will help facilitate the device repair.
• Recover from safe mode – this option produces boot media so impacted devices can boot into safe mode. The user can then login using an account with local admin privileges and run the remediation steps.

Determining which option to use

Recover from WinPE (recommended option)
This option quickly and directly recovers systems and does not require local admin privileges. However, you may need to manually enter the BitLocker recovery key (if BitLocker is used on the device) and then repair impacted systems. If you use a third-party disk encryption solution, please refer to vendor guidance to determine options to recover the drive so that the remediation script can be run from WinPE.

Recover from safe mode
This option may enable recovery on BitLocker-enabled devices without requiring the entry of BitLocker recovery keys. For this option, you must have access to an account with local administrator rights on the device. Use this approach for devices using TPM-only protectors, devices that are not encrypted, or situations where the BitLocker recovery key is unknown. However, if utilizing TPM+PIN BitLocker protectors, the user will either need to enter the PIN if known, or the BitLocker recovery key must be used. If BitLocker is not enabled, then the user will only need to sign in with an account with local administrator rights. If third-party disk encryption solutions are utilized, please work with those vendors to determine options to recover the drive so the remediation script can be run.

Additional considerations
Although the USB option is preferred, some devices may not support USB connections. In such cases, we provide detailed steps below for using the Preboot Execution Environment (PXE) option. If the device cannot connect to a PXE network and USB is not an option, reimaging the device might be a solution.

As with any recovery option, test on multiple devices prior to using it broadly in your environment.

Prerequisites to create the boot media

1. A Windows 64-bit client with at least 8GB of free space from which the tool can be run to create the bootable USB drive.
2. Administrative privileges on the Windows client from prerequisite #1.
3. A USB drive with min 1GB and max of 32GB. All existing data on this USB will be wiped and will be formatted automatically to FAT32.

Instructions to generate the WinPE recovery media
Tocreate recovery media, follow these steps on the 64-bit Windows client mentioned in prerequisite #1:

1. Download the signed Microsoft Recovery Tool from the Microsoft Download Center.
2. Extract the PowerShell script from the downloaded solution.
3. Run MsftRecoveryToolForCSv31.ps1 from an elevated PowerShell prompt.
4. The ADK will download and media creation will start. It may take several minutes to complete.
5. Choose one of the two options mentioned above for recovering affected devices (see additional details below).
6. Optionally select a directory that contains driver files to import into the recovery image. Keyboard and mass storage drivers may be needed. Network or other drivers are not required. We recommend you select “N” to skip this step. The tool will import any SYS and INI recursively under the specified directory.
7. Select the option to either generate an ISO or USB drive and specify drive letter.

Prerequisites for using the boot media
The BitLocker recovery key for each BitLocker-enabled impacted device on which the recover media is used may be required. If you are using TPM-only protectors and using the safe boot option, then the recovery key will not be required. If you are using TPM+PIN protectors, then you may need the recovery key if you do not know the PIN for the device.

Using Recovery from WinPE media

1. Insert the USB key into an impacted device.
2. Reboot the device.
3. During restart, press F12 (or follow manufacturer-specific instructions for booting to BIOS).
4. From the BIOS boot menu, choose Boot from USB and continue.
5. The tool will run.
6. If BitLocker is enabled, the user will be prompted for the BitLocker recovery key including the dashes. The recovery key options are provided here. For third-party device encryption solutions, follow any steps provided by the vendor to gain access to the drive.
7. The tool will run the issue-remediation scripts as recommended by CrowdStrike.
8. Once complete, remove the USB drive and reboot the device normally.

Using Safe Boot media
To repair an impacted device without using the BitLocker recovery key and if you have access to the local administrator account:

1. Insert the USB key into an impacted device.
2. Reboot the device.
3. During restart, press F12 (or follow manufacturer-specific instructions for booting to BIOS).
4. From the BIOS boot menu, choose Boot from USB and continue.
5. The tool runs.
6. The following message appears: "This tool will configure this machine to boot in safe mode. WARNING: In some cases you may need to enter a BitLocker recovery key after running."
7. Press any key to continue.
8. The following message appears: "Your PC is configured to boot to Safe Mode now."
9. Press any key to continue.
10. The machine reboots into safe mode.
11. The user runs repair.cmd from the root of the media/USB drive. The script will run the remediation steps as recommended by CrowdStrike.
12. The following message appears: "This tool will remove impacted files and restore normal boot configuration. WARNING: You may need BitLocker recovery key in some cases. WARNING: This script must be run in an elevated command prompt."
13. Press any key to continue.
14. The user repair will run and the normal boot flow will be restored.
15. Once successful, the user will see the following message: “Success. System will now reboot.”
16. Press any key to continue. The device will reboot normally.

Using recovery media on Hyper-V virtual machines
The recovery media can be used to remediate impacted Hyper-V virtual machines. To do so, select the option to generate an ISO when creating the recovery media using the steps above. For non-Hyper-V virtual machines, follow instructions provided by your hypervisor vendor to utilize the recovery media.

Steps to Recover Hyper-V virtual machines

1. On an impacted virtual machine, add a DVD Drive under Hyper-V settings > SCSI Controller.

Screenshot for where to add the DVD Drive.

2. Browse to the recovery ISO and add it as an Image file under Hyper-V Settings > SCSI Controller > DVD Drive.

Screenshot of where to add the image file.

3. Note the current Boot order so that it can be restored back manually later.

Screen shot of the original boot order.

4. Change the Boot order to move the added DVD Drive the first boot entry.

Screenshot of the change to the boot order.

5. Start the virtual machine and select any key on keyboard to continue booting to the ISO image.
6. Depending on whether the option to use WinPE or safe mode was used when creating the recovery media, follow the steps above to repair the system.
7. Set the boot order back to the original boot settings from the virtual machine’s Hyper-V settings.
8. Reboot normally.

Using PXE for Recovery
For most customers, the options listed above or following the steps in the KBs linked towards the end of this post will help restore your devices. However, if devices are unable to use the option to recover from USB, for example, because of security policies or port availability, IT admins can use PXE to remediate.

To use this solution, you can use the Windows Imaging Format (WIM) that the Microsoft Recovery Tool creates in an existing PXE environment as long as the impacted devices are on the same subnet as the PXE server. Alternatively, you can either use the PXE server approach outline below. This option works best when the PXE server can be moved subnet to subnet easily for remediation purposes.

Prerequisites for PXE Recovery

1. An x64 machine (referred to as the “PXE server”) which will host the boot image.
   1. The PXE server can run on any supported Windows client x64 operating system.
   2. The PXE server should have network access to download the tools either from https://go.microsoft.com/fwlink/?linkid=2281008 or from an internal link on your network.
   3. The PXE server should have inbound firewall rules created for UDP ports 67, 68, 69, 547, and 4011. The PXE tool downloaded (MSFTPXEToolForCS.exe) will update the Windows Firewall settings on the PXE Server. If using a third-party firewall, create rules following their recommendations.
      NOTE: This script does not clean up the firewall rules. You should remove these firewall rules after remediation is complete. You can run MSFTPXEToolForCSv31.ps1 Clean from an elevated PowerShell prompt to remove these rules from the Windows firewall.
   4. You'll need admin privileges to run the PXE tool.
   5. PXE server requires the VC Redistributable. The latest version can be downloaded and installed from: https://aka.ms/vs/17/release/vc_redist.x64.exe
2. The affected Windows devices should be on the same subnet as the PXE Server and should be hard-wired instead of using a Wi-Fi network.

Configuring the PXE server

1. Download the package from https://go.microsoft.com/fwlink/?linkid=2281008.
   1. The zip file contains all the files needed. Extract the contents of the zip to any directory.
2. From an elevated PowerShell prompt, change to the directory where you extracted the files and execute the following from an elevated PowerShell prompt: MSFTPXEToolForCSv31.ps1
   1. The script will initiate a scan for ADK and ADK WinPE Add-On installation on the machine and install them if missing. Accept the on-screen license request to proceed with installation.
   2. The script will generate the remediation scripts and create a valid boot image.
   3. If required, accept the prompt and provide a path containing the driver files. Driver files may be required for keyboard and/or mass storage. Generally adding drivers will not be required. If no additional driver files are needed, select ‘n’.
   4. You will be given the option to set up the PXE Server to deliver a default remediation image or a safe mode image with the following prompts:
      1. “1. Boot to WinPE to remediate the issue. It requires entering BitLocker recovery key if system disk is BitLocker encrypted.”
      2. “2. Boot to WinPE configure safe mode and run repair command after entering safe mode. This option is less likely to require BitLocker recovery key if system disk is BitLocker encrypted.”
   5. The script will generate the required distribution files and provide the path where the PXE server tool is copied.
3. Ensure that the prerequisite, https://aka.ms/vs/17/release/vc_redist.x64.exe, is installed and all the prerequisites are met.
4. From an elevated command prompt, change to the directory where the PXE server tool is copied. Run .\MSFTPXEToolForCS.exe to launch the listener process.
   1. You will not receive additional responses as this is the PXE server handling connections. Do not close this window as it will stop the PXE server.
   2. You should monitor the PXE server progress in the MSFTPXEToolForCS.log file within the directory.
      NOTE: If you would like to run multiple PXE servers for different subnets you will need to copy the directory where the PXE server tool is copied and execute steps 3 & 4 above.
   3. Additional references:
      • PXE boot in Configuration Manager - Configuration Manager
      • Advanced troubleshooting for PXE boot issues - Configuration Manager
      • You want to PXE Boot? Don't use DHCP Options

Recover an impacted device

1. The impacted device must be on the same subnet as PXE Server.
   1. If the devices are in a different subnet, configure IP Helpers in your network environment to enable the discovery of the PXE server.
2. If the impacted device is not configured for PXE boot, follow these steps:
   1. On the impacted device, enter the BIOS\UEFI
   2. This operation is different across different models and manufacturers. Refer to documentation provided by the manufacturer for your machine (make and model).
   3. Common options for accessing the BIOS\UEFI involve inputting a key like F2, F12, DEL, or ESC during the startup.
   4. Ensure Network boot is enabled on the device.
   5. Refer to documentation from your manufacturer for additional guidance
      1. Reference documentation: How to Enable Pre-Boot Execution Environment (PXE Boot) in BIOS?
   6. Configure the network boot option as the first boot priority.
   7. Save the new settings and reboot the client device for the settings to apply.
   8. Device will now boot from PXE.
3. PXE boot the affected machine
   1. Depending on whether you chose to create WinPE or safe mode media, the user will either be prompted to boot to Windows PE and the remediation script will execute automatically, or they will boot to safe mode where the user needs to login with local admin credentials and execute the script manually. See the detailed steps above for the experience for both WinPE and safe mode.
      1. If you created the safe mode option, you will need to run the following commands as admin, from an elevated Command prompt
         a. del %SystemRoot%\System32\drivers\CrowdStrike\C-00000291*.sys
         b. bcdedit /deletevalue {current} safeboot
         c. shutdown -r -t 00
4. Once complete, reboot the device normally by responding to the prompt on the screen. Enter the BIOS\UEFI and update the boot order manually to remove PXE boot if needed.

For more information on the issue impacting Windows clients and servers running the CrowdStrike Falcon agent, please see:

• A wide variety of Windows information is available at aka.ms/WRH
• Additional recovery options are described in the following articles:
  • KB5042421: CrowdStrike issue impacting Windows endpoints causing an 0x50 or 0x7E error message on a ...
  • KB5042426: CrowdStrike issue impacting Windows servers causing an 0x50 or 0x7E error message on a bl...
  • KB5042429: New recovery tool to help with CrowdStrike issue impacting Windows devices - Microsoft Su...
  • Windows 365 Cloud PC, customers may attempt to restore their Cloud PC to a known good state prior to the release of the update (July 19, 2024) as documented here: Enterprise or Business
• For Windows Virtual Machines running on Azure follow the mitigation steps in Azure status

• Refer here for recovery options for Azure Virtual Machines (VMs): Recovery options for Azure Virtual Machines (VM) affected by CrowdStrike Falcon agent - Microsoft Co...

• Additional details from CrowdStrike are available here: Statement on Falcon Content Update for Windows Hosts - CrowdStrike Blog

[7/21/2024] - PXE recovery option added.

[7/22/2024] -Signed Microsoft Recovery Tool updated (multiple changes summarized below). Also updated a Windows KB article link.

[7/23/2024] - Minor text updates.

Changes to Recover from WinPE for USB, PXE and ISO delivery:

• Additional logging for the commands that get run to help with troubleshooting.
• Added the capability to try again or change the drive if printing BitLocker information or unlocking the drive does not process successfully as expected.

Screenshot of remediation script to try another drive

• In the event that you wish to recover a drive that is not encrypted with BitLocker,select C to continue when prompted.
• To select a different drive to re-attempt BitLocker recovery, select T to Try again when prompted.
• Depending on configuration, a different message may appear. When prompted with “Error: No key protectors found”, hit enter to return no recovery key and to continue with remediation:

Configuration screen share.

• Error handling improvements based on customer feedback.

Change to Recover from safe mode for USB and ISO delivery

• Better user awareness of cues of when to run Repair.cmd

The zip and PS1 names have been updated to:

• MsftRecoveryToolForCSv2.ps1 -> MsftRecoveryToolForCSv31.ps1
• MSFTPXEInitToolForCS.ps1 -> MSFTPXEToolForCSv31.ps1 - Init removed from the file name