The Perils of Patching

It’s every IT pro’s nightmare.

A massive outage at the cloud computing service provider Fastly
essentially broke the Internet, taking down Amazon, Reddit, and countless other websites worldwide for more than an hour. Ironically, the outage wasn’t the work of hackers. It was triggered by a bug in a software deployment that was activated when a single customer changed their configuration settings. Fastly quickly restored service, but the massive disruption quickly rippled through the industry.

This disastrous incident shines the spotlight on an issue that is often overlooked: the need for greater rigor in rolling out software updates and ensuring patch management doesn’t accidently introduce a risk.

Software vendors regularly patch their products to address issues and make their programs more usable for customers. The problem is that they don’t always follow strong quality procedures to ensure that updates won’t introduce new, potentially catastrophic, problems. What does this mean for you if you’re one of their customers? It means you must proactively do all you can to protect yourself.

How to Limit Third-Party Risk

Increasingly, enterprises are at the mercy of the software vendor, as evidenced by supply-side attacks involving SolarWinds
and Kaseya. From an IT management perspective, that’s both a blessing and a curse. If something goes wrong, you’re powerless (but it’s a relief to know it’s not your fault).

Still, it’s a net loss when it impacts business. The good news is that while much of the update process is beyond the control of the customer, there are some fairly simple things you can do to minimize vendor risk:

  • Authorize your operations team to patch known issues as soon as possible but consider asking them to wait for the next regular update if they are confident the known issue doesn’t impact your IT environment.
  • Have your legal team evaluate compliance and possible gotchas in the third-party software vendor documentation. Sometimes clicking through a vendor’s terms and conditions reveals unexpected exceptions — such as product security features that are only available in a higher product tier.
  • Use an integrated vulnerability management (IVM) tool to audit your infrastructure on a continual basis.
  • Put a change management policy in place that requires you to always roll out patches and updates to a test group before deploying them to a larger audience.

Make Sandboxes Part of Your Change Management Strategy

It’s inevitable that mistakes will happen at some point, and that’s why putting a formal change management process in place is invaluable. Recognize that today, updates can contain malicious code. The question becomes, how do you kick the tires to make sure a software update does what it is supposed to?

First of all, make sure you regularly back up your critical IT infrastructure. This way, if a third-party vendor’s bug impacts you, it will be possible to restore your entire IT environment quickly. The ITIL framework offers a good change control mechanism and is a good starting point for most companies looking to implement change management.

Every time you make a change, you should be able to rest easy knowing you can roll it back if something goes wrong. Here are three strategies to consider:

  1. Document the steps you take when rolling out an update or patch, much like a pre-flight checklist. Make sure you identify the change, it’s purpose and each step in its deployment. Your goal is to be able to reverse engineer the change so it can be quickly rolled back in case of disaster.
  2. Make it a policy to always test updates in a sandboxed environment to learn how the update or patch will impact the rest of your environment. Consider using a digital twin to make sure your test environment is as close to your production environment as possible.
  3. Once changes are vetted in the sandbox, begin the process of deploying them to the production environment.

When it comes to software updates and patching, security must be front of mind. It’s key. On the vendor level, you have to feel confident that the software companies you work with will quickly identify issues and can restore operations to pre-patch levels for their customers. On the customer level, it’s incumbent upon all of us to limit supply chain risk and protect our businesses as much as possible. Remember, once an environment is compromised, it’s like dominoes. There can be no end to the issues, and the fallout can be catastrophic. A cautious approach to rolling out updates and software patches may slow things down — but sometimes, that can be a good thing.