RSMUS.comYou’ve been rebooked!
That was the subject line of the email I received from American Airlines at 12:32 AM CST on July 19, 2024. My original 7:00 AM flight from Cedar Rapids, IA to Charlotte, NC had been canceled, and I was rebooked for a later flight that afternoon.
For months, my dad and I had been looking forward to spending the weekend together in Charlotte. When I saw the cancellation, I sprang out of bed to contact the airline. Our itinerary was time sensitive. I managed to rebook a 7:00 AM flight out of Madison, WI, but that one was quickly canceled too. That’s when I knew something bigger was going on.
Source: AP Images
It didn’t take long to learn the cause: a routine update from cybersecurity giant CrowdStrike had triggered a critical failure in Windows systems worldwide. Needless to say, we never made it to Charlotte.
As we approach the one-year anniversary of that unprecedented outage, it’s worth reflecting on the lessons learned—particularly how it exposed the risks of relying too heavily on a single cloud platform. In response, many organizations have begun exploring multicloud strategies to build greater resilience, flexibility, and control into their IT infrastructure.
Trusted and Recognized
Headquartered in Austin, Texas, CrowdStrike is a publicly traded company (CRWD) that specializes in managed detection and response (MDR) services. The company positions itself as a leader in cyber threat intelligence, endpoint protection, and is known for offering the world’s first AI-native Security Operations Center (SOC) platform. CrowdStrike serves a variety of industries including:
- Small Business
- Law Firms & Insurance
- Federal, State, and Local Government
- Financial Services
- Healthcare
- Education
- Retail
In 2023, Gartner recognized CrowdStrike as a leader for Endpoint Protection Platforms (EPPs) for the third consecutive time and a consumer choice for MDR services. They scored highly by consumers across the various industries they serve in the following categories, with a willingness to recommend score of 97%:
- Service Capabilities (5 out of 5)
- Execution Experience (4.9 out of 5)
- Transition Experience (4.8 out of 5)
- Sales Experience (4.7 out of 5)
By summer 2024, they had around 21,000 customers and were continuing to look for new ways to innovate and invest more into their Falcon Extended Detection and Response (XDR) platform. Following their IPO in 2019, CrowdStrike’s stock price was trading at an all-time high of $390 per share as of July 8, 2024.
What happened on July 19th, 2024?
At the heart of CrowdStrike’s reputable XDR solution is the Falcon platform. As part of the company’s proactive approach to cybersecurity, the platform deploys lightweight sensors to protected endpoints including computers, laptops, and servers. These sensors collect telemetry data and enforce security policies. According to CrowdStrike, updates are delivered as sensor content and rapid response content. Sensor content is packaged with the sensor itself. Rapid response content is delivered from the cloud and can be updated several times a day. This allows CrowdStrike to respond quickly to emerging threats. The data collected by the sensors is sent back to the vendor for diagnostics, threat detection, and analysis. Updates to these sensors are tested and validated internally for any bugs or defects before they are released to the protected devices.
Unbeknownst to CrowdStrike, a bug in their content validator allowed for a faulty InterProcessCommunication (IPC) template—introduced to detect new attack techniques by bad actors—was released into production on July 19th, 2024. Essentially, an update with problematic coding was missed by internal controls. To compound the issue, the updates were not rolled out on a staggered basis but rather pushed to all of CrowdStrike’s protected devices at once. This triggered memory errors that Windows operating systems couldn’t handle, and thus one by one these devices started to crash worldwide. By 11:30am EDT, the US Cybersecurity & Infrastructure Security Agency (CISA) issued an initial alert confirming the outage.
To make matters worse, Azure customers in the Central US region had experienced an outage of their own. Due to storage availability failures, servers across the region started to go down. Microsoft cloud services across Microsoft 365, Dynamics 365, and Microsoft Entra were affected as they run off the Azure platform causing login and connectivity failures for end users. Although the Azure outage was an isolated event to the CrowdStrike outage, this only compounded consumer confusion and frustration.
By the morning of July 21st, Microsoft had released a recovery tool to repair affected any Windows systems impacted by the CrowdStrike outage. They had also restored functionality to all affected resources in the Central US region. But by that time, the damage was done. Across the globe, planes were grounded, doctor’s appointments were cancelled, and news agencies were taken off the air. The CISA also reported that cyber threat actors were using the outage as an opportunity to conduct malicious activity, including phishing attempts on affected CrowdStrike customers.
Other notable outcomes from the fallout include:
- An estimated 5 million Windows devices of all kinds were affected by the CrowdStrike outage
- The outage cost Fortune 500 companies around $5.4 billion in damages
- Delta Airlines is pursuing legal action against CrowdStrike and Microsoft after they lost $500 million due to the outage
- August 2nd: CrowdStrike’s stock price dropped 44% in about two weeks before reaching its lowest point in early August
- August 5th: Between July 5th and August 5th, Microsoft’s stock price fell by 15%
The Value of Multicloud
For many organizations, July 19th brought a perfect storm of chaos, disrupted operations, and lost revenue. Businesses of all sizes and across various industries came to a standstill due to what many initially believed was just a simple computer glitch. The frustration among stakeholders and executives was understandable. They had invested significant time and money into integrating cloud solutions into their technology infrastructure. Their workloads were hosted in Azure—wasn’t that supposed to be enough? Unfortunately, it wasn’t, and they learned that the hard way.
Still, the disruption offered a valuable opportunity. Many companies took a step back to reassess their technology strategies and look for ways to build greater resilience. This is where a multicloud approach proves to be not just a good idea, but something all businesses need to consider moving forward.
Buzzwords like private cloud, public cloud, and hybrid cloud and have been a part of the IT lexicon for a while now. Hybrid cloud has become a very popular option for organizations that are hesitant to fully commit to the cloud. This approach combines on-prem infrastructure with a public cloud platform, offering IT managers a balance of control and scalability.
So what’s the deal with multicloud? A multicloud approach expands an organization’s public cloud presence, as their workloads are spread out across multiple public cloud platforms. For example, instead of running everything that’s not on-prem or in their data center in Azure, organizations strategically identify specific workloads to run on other public clouds like Amazon Web Services (AWS), Google Public Cloud (GCP), or a mixture of all three.
This approach allows organizations to not only leverage the best features on these respective platforms, but also increases their resiliency in the face of outages or service disruptions. Furthermore, companies that embrace a multicloud strategy avoid being locked into a single vendor’s pricing structure. If costs begin to strain the budget or the quality of service from one provider declines, companies that adopt multicloud have the flexibility to shift workloads to another platform. This reduces the risk of over-reliance on any one cloud provider and gives organizations greater control over both performance and cost.
Of course, multicloud isn’t for everyone. There are notable drawbacks to this approach that stakeholders need to take into consideration. Spreading workloads across multiple public clouds introduces more complexity into an organization’s technology solution—and where there’s an increase in complexity, there frequently comes an increase in cost.
Integrations become extremely important in a multicloud approach, and often these workloads spread across different cloud platforms need the ability to talk to each other. As a result, organizations find themselves investing more in virtual networking appliances (NVAs) and associated licensing fees. For businesses that are sensitive to downtime and service interruptions, their costs can increase significantly when examining the expenses related to replication and recovery for these workloads.
Are You Ready?
When considering a multicloud approach, here is the question that IT managers, executives, and stakeholders need to ask themselves: Is it worth it?
These leaders can start to answer this big question by reflecting on the following:
- What is your recovery time objective (RPO)? What is your recovery point objective (RTO)?
- When was the last time you tested your business continuity and disaster recovery (BCDR) plan? Were you happy with the results? What changes do you need to make?
- How did the CrowdStrike outage affect your business from a financial, operational, and a reputational perspective?
- Is your company prepared for the next major outage?
Consider a multicloud strategy for increasing your organization’s resiliency, control, and flexibility. In today’s global and interconnected economy, it is no longer enough to rely solely on the most trusted providers or the most reliable cloud platforms. The question is not whether another outage will happen, but when it will happen. Are you ready?
Learn more about RSM’s customizable cloud solutions here, or reach out to Ken Osterhaus (Ken.Osterhaus@rsmus.com) to understand whether a multicloud strategy is the right approach for your organization.
Ken Osterhaus
As a first-choice advisor, Ken helps organizations make informed decisions about their cloud environments and plan for future growth. While specializing in roadmapping well-architected solutions, optimizing Azure costs, and enhancing performance, his ultimate goal is to deliver comprehensive and resilient technology strategies tailored to each client’s unique needs. Beyond his day-to-day advisory work, Ken shares insights, emerging technology trends, and updates on new RSM services through articles on the RSM Technology Blog, his personal Medium page, and RSM Cloud Clips. Ken is committed to helping clients maximize the value and effectiveness of their cloud investments while staying ahead of evolving industry trends. Feel free to contact Ken at Ken.Osterhaus@rsmus.com