This article is part of our ongoing podcast blog series exploring practical challenges and key considerations in securing Operational Technology (OT) environments. In our first installment, we covered asset identification under NERC CIP standards—a foundational step in any strong OT cyber security program. In this post, we turn our attention to another critical element: patching. While software patching in IT environments is already complex, the stakes and constraints are even greater in OT. Below, we walk through some of the most common patch management challenges unique to OT environments and why a well-informed, strategic approach is essential.
Software patching is difficult, whether it’s for IT or OT assets. While there are many challenges that are encountered in patching both IT and OT assets, there are also a host of challenges that only apply to OT. Any OT patching program must take account of those challenges and include a strategy for addressing every one of them. Here are some of the most important challenges.
I. Patch prioritization in OT environments
Unlike in IT, applying almost any OT patch carries a large amount of risk, since the cost of disrupting a process like the smooth operation of a manufacturing plant or the unimpeded flow of electric power through a transmission network is usually significantly higher than the cost of, for example, bringing down the mail server in an IT network.
This means the first question to be answered in OT patch prioritization is whether each patch needs to be applied at all. Considerations for answering that question include:
A. Is the asset being patched a critical one?
For non-critical assets, the risk of downtime is much lower than for critical assets, meaning it’s easier to approve application of a patch. However, for critical assets, the question really is, “Do we need to apply this patch at all?” In many cases, the correct answer is “no”. – but arriving at that answer requires careful consideration of items like:
- Is there already a control in place that will mitigate this vulnerability? Or is there a mitigation we can easily put in place – such as closing a particular firewall port – that will remove the need to apply this patch? In OT, it’s almost always better to apply an easy mitigation than to apply a patch.
- Will the risk that the asset could be disrupted outweigh the benefit of applying the patch? If that is the case, it might be worthwhile to implement a mitigation that isn’t easy – for example, installing the critical device on its own network segment, protected by a data diode.
- Assets deployed in a DMZ, while not necessarily critical in themselves, deserve to be treated as critical because of their much higher level of exposure to internet-based attacks than assets deployed on the OT network or NERC CIP Electronic Security Perimeter (ESP).
B. How risky is the CVE being patched – relative to other CVEs being patched?
Today, most organizations have more patches to apply than they have time or resources to apply them. They need to prioritize, so that a) patches that mitigate the most software vulnerability risk get applied first, and b) patches that mitigate little or no risk don’t get applied at all. Patch prioritization is especially important in OT, because of the much more severe consequences that can ensue if a patch application goes awry.
There is a lot that can be said about software patch prioritization, but the most important consideration is how the organization will measure the degree of vulnerability risk that is mitigated by each patch. The three most important measures of vulnerability risk are CVSS Base Score, presence of the CVE in the CISA KEV Catalog, and the EPSS score of the CVE.
These measures can be combined to create a risk score for each patch that needs to be applied. The score will determine both whether the patch should be applied at all and the order in which it should be applied – relative to other patches that are also in the queue.
C. Is an outage required?
Prioritizing patching in OT often runs into the problem that many OT assets can’t be patched until they can be rebooted. Sometimes that requires an outage of the entire facility, which might not happen for six months or even longer. Therefore, it doesn’t do a lot of good to place an asset at the top of the priority list for applying a patch, when the patch can’t be applied for six months. Assets that need to wait for an outage to be patched should be prioritized separately from assets that don’t require an outage.
II. What are the dependencies?
Both IT and OT assets have dependencies – i.e., other assets on which a particular asset depends for critical information, controls, etc. The big difference in this regard is that IT assets typically depend on a central asset, not each other. For example, there might be 1,000 IT workstations that all utilize a single database.
However, OT assets often have complex dependencies on each other, since the different parts of the process being controlled by the OT devices depend on information from both upstream and downstream systems. This means that applying a patch to an OT system requires understanding all the system’s dependencies.
It would be easy to say that the person applying a patch to an OT system simply needs to first read the organization’s documentation of all the system’s dependencies. However, a lot of organizations with OT systems don’t have good documentation of those dependencies. This means that anyone who wants to patch a critical OT system needs first to consult with on-site experts – preferably the change management team, if there is one – beforehand.
III. Where’s your inventory?
It’s close to impossible to apply patches without having a good inventory available, which has been recently updated. The inventory should include the name and version string of every software and hardware product installed in the OT network. Of course, it is harder to have a good inventory for an OT than for an IT network, since OT controls like network segmentation and air gaps often make it impossible to develop a complete inventory without manually updating the information.
IV. Managing obsolescence
OT networks often contain many legacy assets, some of which are in end of life status – meaning the manufacturer is no longer producing patches for them. To prevent unpleasant surprises, it is important to manage obsolescence. Even if you can’t immediately replace an EOL (end of life) product, you should always have a plan in place for the day when a serious new vulnerability in that product requires you to immediately replace it with a supported product. You can’t afford to take a few days to figure out how to replace an obsolete product if the entire plant is down, awaiting your decision.
V. Where’s the vulnerability?
IT professionals are used to being able to scan an asset before applying a patch, to make sure the vulnerability that is mitigated by the patch is in fact present in the software product. However, trying to do the same before patching an OT asset is usually much harder. The reasons for this include:
- Because OT assets are much more sensitive to scanning than IT assets, and because even the smallest amount of downtime can be very expensive, it is often impossible to scan an asset.
- Even when a vulnerability (i.e., a certain piece of vulnerable code) is present in a software or hardware product, that doesn’t mean an attacker will be able to “reach” the vulnerable code as easily as they might in an IT environment. So, for all practical purposes, this instance of the product isn’t affected by the vulnerability (in other words, the vulnerability is not “exploitable” in that instance of the product). However, a scanning tool will often report a positive identification of the vulnerability based on general information it finds in a vulnerability database like the National Vulnerability Database (NVD).
- When prioritizing vulnerabilities for patching, it is a good idea to look at each of the different types of products that a CVE might be found in. This is because the CVE will often be exploitable in one type of product, but not another. Understanding this will help prevent wasting time patching products that aren’t vulnerable to the CVE that is fixed by the patch.
VI. Maintaining focus
- OT staff members are often their own worst enemies when it comes to searching for vulnerabilities on their networks. It isn’t a good idea to read about a big, scary vulnerability in a newsletter in the morning, then go looking for it on your network and immediately patch it if you find it. Instead, you need to take a disciplined approach and allow a defined process to prioritize vulnerabilities for remediation.
- It is also a waste of time to continually focus on recently announced vulnerabilities, especially if they have a high severity rating (CVSS score). Many people think that recently announced vulnerabilities are more likely to be exploited, but that is simply not true. In fact, hackers much prefer to stick with the vulnerabilities they’ve been exploiting for years, rather than continually take their chances on exploiting new vulnerabilities.
- Most OT networks have a huge number (often in the hundreds of thousands) of unpatched legacy vulnerabilities. It is much better to patiently whittle those down, patching a certain percentage of them every month, rather than continually chase the latest shiny vulnerability, which might not be present at all.
- The great majority of vulnerabilities in OT environments today are found in Windows software. The most prevalent cyberattack type today – ransomware – almost always attacks Windows software. The best way to score a “quick win” in OT vulnerability management is to remove or upgrade software products that are known to be full of unpatched vulnerabilities. Windows 7 Professional, Adobe Flash Player
1and Google Chrome are three particularly bad actors; the first two have been out of support for years. Upgrading or removing any of these three products will usually result in a big reduction in your network’s attack surface.
VII. Control centrally, deploy locally
OT assets are very often highly distributed, with tens or hundreds of power transmission or distribution substations, pumping stations, plants, etc. spread all over. In distributed IT environments, it is a common practice to deploy a single patch from a central location to all remote locations at once.
However, this practice is dangerous in OT environments, since there are many reasons why an unexpected patch deployment might bring down a device because of something else that was going on at the same time. OT patches almost always need to be deployed by someone onsite, working on the asset itself. Only they can make the decision on whether it’s better to install a patch now, an hour from now or tomorrow.
On the other hand, the decision to deploy a particular patch – often as part of a patch prioritization process – should usually be made centrally. It is inefficient to require each remote location to decide for itself which patches should be applied and when, especially if the systems installed in each location are similar, if not identical (which is often the case).
This means that patching in distributed OT environments should usually
- Be under central control, so that decisions regarding applicability, prioritization, etc. are made uniformly, but also
- Make use of local staff members to apply the patches.
VIII. Testing, testing…
It is well known in both IT and OT circles that it is important to test any patch in a controlled environment that simulates the production environment as much as possible. However, the importance of doing this is much higher in OT environments, given the possible physical consequences that could occur if a patch includes an unintended incompatibility with another element in the environment.
Fortunately, the cost of setting up a test network has declined significantly in the past 5-10 years, due to the increased use of software virtualization technologies, as well as the increased availability of automated deployment and testing tools.
IX. Compensating controls
In both OT and IT environments, it is important to deploy compensating controls whenever a patch can’t be deployed for any reason. However, it is common in OT environments for a patch not to be applied immediately due to the need to wait for an outage, even though it will be applied later. Of course, compensating controls need to be in place during the entire period before the outage makes it possible to apply the patch.
If a critical OT asset is being patched, it is also a good idea to have compensating controls in place before the patch is applied. This way, if the patch application has to be aborted for some reason, the asset will still be protected until the patch is successfully applied.
Moving Forward Together
At Foxguard, we understand that patching in OT environments is never one-size-fits-all. From prioritizing what truly needs to be patched to deploying updates safely across distributed systems, our goal is to reduce complexity and help you take decisive, well-informed action.
Our Cyberwatch platform is built to support OT teams by providing actionable vulnerability insights, patch prioritization tools, and robust tracking all in one place. It’s designed specifically with industrial systems in mind, so you get a clear view of where the real risks lie without the noise of traditional IT-centric tools.
In addition, our team publishes monthly ICS Patch Updates, offering a curated, up-to-date summary of the most critical vulnerabilities and vendor patches relevant to the OT space. These updates are crafted to help you stay ahead of evolving threats and make patch planning more efficient.
Whether you’re just getting started or looking to enhance an existing program, Foxguard is here to support you with solutions that meet the unique demands of operational environments.
Need help patching your OT environment? Contact us today to learn how Foxguard can simplify your patching needs.
- You may be surprised to learn that, more than four years after Adobe Flash Player went into EOL status amid a deluge of negative news reports, it is still found in many OT networks. However, that is the case. ↩︎