Patching should reduce the risk to your business, but it also brings its own risks. There is a risk that installing updates can cause unintended side effects. These can include performance reduction, breaking the updated software, removing custom and needed changes or inadvertently changing configuration settings to weaken security, or disrupting other software running in parallel. Another risk is that the update itself might be compromised: it might not have come from an authorised source, or it might have been tampered with. It must be checked before installation.
Standard patching process
To minimise the risk of patching, you should define a standard patching process, such as:
- Understand what the patch is for
- Which software it applies to.
- What issues it should be resolving.
- What the dependencies are for implementation.
- If there are any known issues with the implementation of this patch.
- Confirm that the patch files come from the correct source, and document it:
- They should come from the vendor site
- They should be downloaded over a secure channel
- If you are downloading from a mirror site, make sure the patch files are signed
- Download the patch files. Ideally download them to a central location, to make things easier.
- Check that the patch files are complete, and not corrupted, by running a checksum (the vendor should provide you with a hash).
- Agree an appropriate patch window with the business to minimise disruption to service and to your staff:
- This should be long enough to allow the patching to complete.
- It should allow time to roll-back, in case of a failed patch.
- You may need to consider multiple patch windows if you operate in multiple time zones.
- You may need to deploy patches in batches.
- Test the patch in a mirror test environment to avoid unexpected consequences. What you test will depend on the patch, but you should make sure that the core functions still work the same way, and nothing else is impacted by the change.
- Backup your production environment–just in case you need to roll-back (undo the patch, and revert to an earlier version of the environment). Determine under what conditions you would decide to roll back.
- Patch systems in order of criticality
- Patch redundant systems (such as a cluster of servers) separately
- Re-run if necessary to patch any of your systems that were missed out for any reason
- Verify that the patching is complete:
- If appropriate, check the version number post-patch, to see if it has updated
- Re-run a vulnerability scan to see if the vulnerabilities being patched have been resolved
- Monitor to be sure no problem has arisen
- Measure your patching performance – what percentage of systems are up to date? What percentage of patches failed? How long did it take to patch the most recent release?
- Assess how the process went, and apply any lessons learned.
- Documentation at every step is essential: you need to know exactly what you’ve done at every step. This documentation will support your knowledge of your asset estate, ready for the next release, but would also support roll-back if necessary, or the pinpointing of any problems that might arise.
Emergency patch release
Are you sure that an announced vulnerability affects you? Don’t be swayed by the level of publicity surrounding a particular vulnerability online—it may not be relevant to your environment, or you may already have controls in place that would cover it.
Discuss the emergency patch release with the business risk owners, so that they understand and agree that it must be done.
If you’re sure that you need to install the patch, install it in a mirror environment and then test to make sure it works and has no adverse effects. Do the standard regression testing you’d do for any patch release. While we know that emergency patches need to go out quickly, don’t rush this stage: it would not help to create extra damage while trying to roll out a patch.
Notify your end users if you need to apply a patch out of the usual timeframe, or if they’ll need to reboot.