Episode Two: Patch Management Uncovered: Lessons, Mistakes, and the Path Forward
In the second episode of Decoding NERC CIP & OT Security host Chris Humphreys sits down with Greg Valentine, Senior Vice President of Solutions Engineering at Industrial Defender, to explore the realities of OT patch management. From common mistakes and lessons learned to aligning compliance with operational efficiency, this conversation dives into how utilities and critical infrastructure organizations can strengthen their NERC CIP compliance programs while building sustainable patching strategies.
Listen now or read the full NERC CIP OT patch management podcast transcript below.
Looking for more context on the topics covered in this episode? Explore related posts from our Decoding NERC CIP & OT Security series:
A Guide to Patching by Asset or Risk | Decoding NERC CIP and OT Cyber Security Podcast | Episode One
Decoding NERC CIP & OT Cyber Security Podcast: Episode 2 Transcript
Chris Humphreys: Welcome to Foxguard’s Decoding NERC CIP and OT Security podcast, episode two. (Not Episode II: Attack of the Clones—though Star Wars is always what springs to mind.)
This is episode two of our Decoding NERC CIP and OT Security podcast: Patch Management Uncovered: Lessons, Mistakes, and the Path Forward. I’m very grateful today to have Greg Valentine with us, from Industrial Defender.
Greg is the Senior Vice President, Solutions Engineering at Industrial Defender. We’ll get more into his bio shortly before we get stuck into talking about some cool things today.
My name is Chris Humphreys, and my background is definitely NERC CIP inside and out. I was the first Manager of Ordnance and Investigations for versions one and five of the NERC CIP standards at ERCOT and Texas RE. Greg and I also share that same Texas sort of background ourselves.
We’re going to get into and what to expect from this podcast, and we’re going to talk a lot about OT, and OT, cyber security. We’re going to focus a lot on OT patch management, vulnerability assessments and management, and how to implement best practices in OT cyber security.
So, the mantra we always come back to is this: compliance should align with being secure and being operationally efficient, and it should be a byproduct of being secure and operationally efficient first. It should not be driving things, it should be an output, and so, I think our focus is always around how to implement people, processes and technologies, the solutions associated with that. And
I think that’s the overall reoccurring theme that we want you guys, our listeners, to take away from this.
Some of the future topics we’re going to discuss are approaching vulnerability strategies when it comes to CIP, by leveraging things like CVE and system ratings, and then best practices around asset inventory is also going to be a good one, because, I mean, you can’t do any of this stuff until you know what you have in your environment.
And as always, I’d love for you to send your questions and feedback via LinkedIn and let us know what challenges you’re facing. If there are topics you’d like us to cover in future episodes, we really encourage that feedback.
On the agenda for today, we’re going to draw on Greg’s 15+ years of experience in patch management to share lessons learned. We’ll cover the evolution of OT patching, what works, what hasn’t, field-tested takeaways, and some of the common mistakes we still see today around patch classification, manual processes, and weak validation. That’s a huge thing we see as well.
We’ll also look at compliance blind spots.If we’re making all this effort to do these things, but you’re expected to do them manually, you’re completely undermining the effort and the investment of technology and controls. So modern improvements in patch programs, modernizing patch workflows, secure supply chain.
We’re going to talk about how Foxguard and Industrial Defender work together to deliver visibility and automation when it comes to patching and patch management. We’ll talk about compliance for security, which will be the overall theme and everything we talk about, but avoiding the checkbox trap, and aligning security goals with CIP mandates.
And then the last thing we’re going to cover (and we’ve got a lot to cover), is closing insights, what’s next, final thoughts from Greg, and a preview of episode three. We’ll also talk about the blog tie ins we’ve released in between each podcast episode.
So, a little bit about Foxguard real quick. We specialize in providing comprehensive Patch Management and Vulnerability Solutions for OT environments, helping organizations improve their risk posture and achieve NERC CIP compliance. Our core offerings are around Foxguard Discover, which is our Passive Asset Discovery and Threat Detection platform. We have our Cyberwatch platform, which is our new Compliance, Vulnerability and Asset Management solution thata lot of people have been seeing throughout the industry lately.
Foxguard Patchintel is our streamlined OT patch management, intelligence and acquisition solution. This is what we’re known for, for those of you who are familiar with Foxguard. First and foremost is our OT patch intelligence capability.
Deploy is our other model for actually offering for deploying of those patches after we provide all that intelligence and acquisition. And then we have our full realm of services: cyber security services, risk assessments, program development for audits, training, system hardening, Patch Management as a Service (PMaaS), Vulnerability Management as a Service (VMaaS), and our specialized OEM services. That’s where we cut our teeth from, we were born out of our relationship with OEMs. We also provide full end-to-end compliance and regulations support, not just NERC CIP, but CSF, NIS2, TSA guidelines, as well as water regulations from the EPA and CISUP.
On our computing side, we have custom computing services and platforms and racks for mission critical industries like energy, aerospace and manufacturing. And then the graphic that we show all the time that you guys should be familiar with, is our Virtuous Loop, where we talk about continuous improvement, patch management, vulnerability remediation, and minimizing risk and downtime. It’s that figure of eight loop that has the nice sort of components baked into it, that gives you that ultimate maturity visual.
With that overview of Foxguard, I’ll now introduce Greg. I want to get into Greg’s background a lot here. Again, he’s the Senior VP for Solutions Engineering at Industrial Defendor. You know that role leading the design and delivery of OT cyber security solutions for critical infrastructure. He’s been in the industry for over 30 years. The last 15+ has been focused on cyber security, especially in OT and ICS environments. He holds CISSP and GIAC grid certifications and serves on the GIAC Advisory Board. He and I share the same passion for certification, GIAC as well, and CISSP. I did a lot of work with the SANS Institute previously, and I think Greg and I probably had some overlap in our career roles at Lockheed Martin, Leidos, Capgemini, Signacert, CoreTrace, and Winternals Software, which was later acquired by Microsoft. He and I speak the same languagewhen it comes to translating compliance frameworks like NIST, NERC CIP, and IEC 62443 into operational solutions that emphasize automation, data integrity, and audit readiness.
Greg went to Saint Edward’s University in Austin Texas. I’ve been back in Florida for the last year, but I spent the previous 15 years in Austin as well. Greg and I have both been local advocates for cyber security awareness and technology in the area, which has really become a new Silicon Valley. Greg, did I cover that? Is that an accurate intro for you?
Greg Valentine: It’s very strange to hear it all mapped out like that, but yes, and thank you, by the way. Thank you for the introduction and I’m happy to be here.
Just to give everyone here a bit more detail on my background, I have been with Industrial Defender now for over ten years, I’d say. And yes, my title is SVP of Solutions Engineering. But in reality, that means I manage the global team of solution engineers—what other companies often call ‘sales engineers’. Basically, it’s the team that handles the technical side of the selling process, including demonstrations, proof of concepts, Q&A, RFPs, etc.
My cyber security journey started with a company called Winternals Software, which most people probably know as Windows Sysinternals. Sysinternals provided —and still provides—free utilities for Windows to manage operating systems and networks. And from there, that’s really where I caught the bug and then moved on to an application whitelisting company called CoreTrace.
This is where I met the Industrial Defender team because they were looking at NERC CIP which had just come out and everyone was a little freaked out, for lack of a better word. They now had to put antivirus inside of the OT network and everybody was very nervous about that. They didn’t want the overhead or the CPU usage dragging down these critical devices. So Industrial Defender reached out to CoreTrace, the company I was with at the time, and basically said, “hey, this could be a compensating control to deal with antivirus issues. Instead of blacklisting like antivirus, we’re going to focus on whitelisting.” So, it was much lighter. I became the technical liaison between the two companies and would bounce back and forth between Austin and Boston and work with them to learn about and promote application whitelisting.
Industrial Defender was then acquired by Lockheed Martin, which is when I joined them. From there, I went on to Leidos and Capgemini as different divisions were acquired and reorganized—that was pretty common back in the day. And now we’re back to being a standalone, PE backed company selling directly, which is nice. There are far fewer lawyers now in our conversations, which I really appreciate, that’s always great.
Chris Humphreys: Yeah. I always tell people I’ve been asked more by lawyers than I have cyber security professionals over my career when it comes to the compliance side of things. But you know that it’s important. I don’t mean to knock it.
Greg Valentine: It’s important when you can relate. You can relate your message to lawyers and cyber security, and they both get it. That’s the balance. That’s the dance done right.
Greg Valentine: It’s a skill. Yeah.
Chris Humphreys: And to your credit, again, me being a nerd from day one, rolling that out in the ERCOT and TRE regions at the time, Industrial Defender was always a standard we saw in many environments, especially when it came to the configuration baseline management component. You guys were a hit and are still the prominent player we see, when we go to utilities. And that makes sense too, because I think, you know, back in 2012, I got to help Foxguard win their grant for their Patchintel solution because we came out and said, “hey, we know patching for OT is going to be a huge challenge for the bulk electric system and we need a solution.” And that’s where Foxguard kind of cut their teeth and then plugged into discovery and asset management, visibility and that kind of thing, and configuration management. It only made sense that Industrial Defender became one of our early partners in that journey. Right?
Greg Valentine: Yeah, that’s interesting. And just so everyone is aware, Industrial Defender has been around for nearly 20 years, as Chris says, we’re at 19 I think, something like that. We bring to the table asset discovery and inventory, and most importantly for this conversation I think, the automated up to date current list of software and firmware versions of the different devices that you want to monitor. And so that’s kind of a perfect feed for what Foxguard brings to the table.
Chris Humphreys: Yeah. And I guess just to kind of turn it up on Industrial Defendor, how has that model evolved from back then to today? What are you seeing? Are you seeing the same drivers? What are you guys doing in Industrial Defender: are you remaining kind of static or how are you evolving your solution from an OT and a compliance perspective? I guess we could introduce that a little bit.
Greg Valentine: Yeah, that’s a good question. So, everyone has heard about the IT OT convergence. I think that that was a big driver for us. So opening up our infrastructure for other systems to work with, i.e. Foxguard being one of them. But, you know, we added a RESTful API for anyone who wants to read or pull data out of our solution. That’s definitely a possibility, sending syslog data off to SIEMs, etc., that’s something that we can do. I’d say the earliest adjustment we made to our roadmap was opening up and integrating with more systems. Since then, I’d say we have two main purchasers or people interested in our product. One group includes people who are really concerned and frustrated, they have issues with compliance in general, whether it’s NERC CIP, SAP, IEC 62443, or whatever framework applies. They’re looking for guidance, and they want to minimize the time and effort it takes to produce the evidence required to get through these audits.
Chris Humphreys: Right.
Greg Valentine: So that’s one of our main feature sets and it’s all around NERC CIP, for example, and providing evidence for that through our reporting. The other would be more cybersecurity. There are systems out there requiring the implementation of true, proper cyber security programs. And you’ve probably heard this before. You can’t really implement a cyber security program without knowing what assets you have, how they’re configured, and what’s changing in your environment. Some of that foundational stuff you really have to have nailed down before you can actually generate or produce some interesting cyber security programs on top of that.
Chris Humphreys: Right. And correct me if I’m wrong, but I feel like Industrial Defender, y’all’s use case around inventory, you know, baseline configuration management, are the things that your suite does very, very well. From the word go, it was a very easy use case for end users to see that they couldn’t do it manually. I think where Foxguard is now, we’re a little bit you know, people have tried and are still trying to sustain patch management manually and they’re realizing they can’t do that. And I think you guys were ahead of that in the initial rollout of things like NERC CIP and things like that. You realize inventory and discovery cannot be done by spreadsheets.
Greg Valentine: Exactly. I think that’s fair because, to be completely transparent, I think we still have one customer who walks the plant, going device to device, and logs it in a spreadsheet so that we can import that data. But the reason you’d want to bring that into our system is because now you can basically compare all 500 spreadsheets over the past year or two (or ten), and see what’s changed over time, generating reports off of that data. It’s far, far, far easier and less burdensome to automate that whole process.
And that way, you know, you’re always looking at the current stuff, right? You don’t want to be looking at it if its outdated. Possibly human error is part of the problem. When you’re inputting that data, automation is definitely the way to go, for both configuration management and also for patching, etc. It just makes the most sense.
Chris Humphreys: So the great thing we see, and I’m sure you guys see this a lot too, but what we see when we visit customers and prospects is, you know, that C level or director level guy will say, “oh, our process is really mature, we’re good, we know what we’re doing.” But they have no purview into the 20 people that are running around like chickens with their heads cut off because of the manual process that they’re doing to try to make that work.
Greg Valentine: And that’s not fun work either.
Chris Humphreys: It’s not something that people enjoy doing and the amount of FTE hours spent on that is massive, and all that administrative work of writing all that stuff down. So, we make that a very tangible use case. And I think the complement of Industrial Defender in that same sort of automation space just bolsters that argument as it’s even easier, I think, to achieve the win that we were talking about earlier, getting lawyers and security people to see the light at the same time, singing from the same sheet of music, that’s truly the dance. And when you can get in there and show this manual workload being automated and optimizing your resource efficiency, you know that use case is going to resonate across a wide variety of stakeholders at an OT organization.
Wouldn’t you agree?
Greg Valentine: Yep. Absolutely. Yeah. 100%.
Chris Humphreys: So, keeping on the ‘what worked and what didn’t’ kind of component, automation and audit ready reporting always reduces errors and delays. And I’m the auditor, I’m the audit guy that’s always in there and saying, “you know, if it’s not documented it didn’t happen.” But producing that documentation in a digestible format and how that relates to patching for OT environments like, you know, what are you guys seeing in your side of the house, or is it not in that realm?
Greg Valentine: So, OT is a different beast, as you know, right? I mean, you have to—in a very non-disruptive way—play by the rules of the OT network,right? So, you have to be wary when you connect to a device, you have to use the OEM approved way of connecting to that device to then collect that configuration data. So, it could be using OT protocols like Modbus, DNP3, IEC, and all the others out there (and there’s a lot of acronyms out there.), or using a very lightweight agent for your Windows or Linux machines that doesn’t require a reboot to install, or uninstall things like that. So, you have to start with the velvet glove, right? You have to be very passive, just watching the network traffic go by and learning what you can. That’s dicey though, because you just don’t get enough detail and you don’t know what applications or software are on that device just by watching the network traffic. You don’t know what users are on that device and you don’t know what the rules are of your firewall, or lots of other things. So that’s the challenge that you have to play with. Once you’re able to deal with that synced hypersensitive environment, then the benefits are huge because not everybody can do that. And you can’t just run an Nmap scan in an OT network, things will fall down and break, and when that happens, really, really terrible things can occur.
Chris Humphreys: I remember the early days, the horror stories. , We’d go in just to do a netstat–a, and suddenly everything’s running and you’re like, “how do you reconcile this?” Right? But that’s essentially what you guys, you know, again from an inventory perspective, knowing what you have at that time, where it is, what’s happening on the network, is paramount. You can’t patch until you know what’s there, right?
Greg Valentine: Exactly. And like I said, passive is safe, right? You’re just watching the network traffic go by, but you don’t get that detailed information. And so, if you’re looking for vulnerabilities, for example, it’s really difficult, as you’re passively watching your network traffic. It’s really, really difficult to tell the difference between Windows 7 and Windows 10. So now I have to worry about the vulnerabilities for both operating systems even though I might not have any Windows 7 systems. Well, that’s not good. That’s not right. You want to be able to actually get a true inventory. What operating system is it? Is it Windows 10? Okay. If it’s Windows 10, is it Windows 10 Pro, which version of Windows 10 is it? And then what are the patches that are on that system? What software is on that system? What version is all of that stuff? So that’s what I mean by playing by the rules to get the really meaty, critical information that you have to come up with to appropriate, accurate patches.
Chris Humphreys: And I think both of us, both our solutions share the importance around validation. Right? You know, validating that sourcing, validating that inventory. We pride ourselves in our ability to validate patch sourcing and where we find patches for things. But I think the validation of that data inherent in your solution, in your tool, is also the checkbox for compliance. Right? But that’s an operational and security thing that needs to be done regardless. But it produces that. That validation serves as a nice byproduct example to easily show in demonstrating a compliance audit, right?
Greg Valentine: That’s right. And what you bring to our solution for our customers and your customers as well, is the fact that these patches have been approved by that particular OEM. That’s the secret sauce that Industrial Defender doesn’t have. We can tell them all day long about all the patches that will fix them, or mitigate the different vulnerabilities or challenges that they have. But Industrial Defender just isn’t aware of which of those have been approved by ABB or Honeywell or Schneider or whoever it happens to be. And that’s where Foxguard comes in.
Chris Humphreys: And I think that Segways nicely into the general patching challenges we see in NERC CIP environments. You know, asset diversity, that’s where again, I think we see these equally, as far as Industrial Defender and Foxguard goes. But asset diversity, management and tracking of multiple patch sources, third party vendor approvals, tracking applicability, evaluation and prioritization. You know, acquisition and transfer in and out of the ESP. All these things—closed loop patch status reporting, time, and the availability of skilled personnel. That last one is the huge one, where it’s these people’s job to constantly run this cycle, to run all this stuff down.
I think again, that human dependency on making sure you got a patch right is why, you know, CIP 7 R2 continues to be—and has been since day one (despite the great solution we created from the DoE grant)—the most violated requirement in NERC CIP. So, you know, I think what we’re talking about is how we can collectively address that with our solutions. I think, again, the asset diversity component is a huge one for Industrial Defender, right?
Greg Valentine: Exactly. So, Industrial Defender collects all this data from wildly different types of things out there, like firewalls, switches, engineering workstations, servers, PLCs, RTUs—all these completely different types of things, and their configurations are obviously completely different. We normalize all of that information in the Industrial Defender way of looking at configurations, and then we provide Foxguard with that normalized data. So, now you can take that data and compare it against your information to provide the list of patches that are best for the consumer.
Chris Humphrey: Yeah. Let’s jump over to the common mistakes that fall into the same challenges around NERC CIP. I think it’s a challenge in any ICS / OT environment you know, missteps in patch classification. NERC CIP is very specific about security patching, and obviously I understand why, but I think operational patches and things like that could have just as much of an impact if you don’t validate that and test that. We also have issues where a vendor might release a patch and not classify it at all. How do we prioritize and evaluate those things as it applies to your inventory; that’s a challenging thing to do.
Greg Valentine: 100%. Right. It’s more than just the CVSS ranking score for a vulnerability. Is it actively being exploited in the wild? What’s the criticality of the devices that have that particular vulnerability? So, it might be a device that’s at Level 1 of the Purdue model, and there’s no way to take advantage of the vulnerability because the ports are closed in the firewall, or whatever it happens to be. So, it’s a lower priority even though it’s a critical device, versus a ‘hi, my hair is on fire’ situation,
because this other critical device is actually outside of the DMZ and close to the enterprise network, which is much more easily accessible by a bad person. So yeah, risk is really a keyway of approaching it.
Chris Humphreys: The mitigation, right. And I think, you know, in our Cyberwatch platform, we show every CVSS score for every vulnerability out there in the wild. But you can also throttle that risk based on varying different factors, like focussing on CVSS. scoring of 7.0 or higher or defining that. The point is, defining that risk register across the board for your environment is essential, because you’re going to get an overwhelming amount of data. And how to throttle that data down to actionable risk is what I think industrial Defender and Foxguard enable our customers to do very, very well together, right?
Greg Valentine: Right. And now again, it’s the combination, Right? So Industrial Defender gives you the inventory including all the patches that have been deployed and Foxguard tells you these are the approved patches and can do the deployment of the patch. Or the user can do that themselves and then leverage Industrial Defender again to validate that the patch was successfully deployed and compare across systems. You know, maybe all my HMIs should be at the same patch level. Are they? Well, you have a system in place that can tell you the answer to these questions.
Chris Humphreys: And then, the last point of the missteps/compulsion component that I’d like to talk about real quick is, you know, distinguishing IT versus OT patch priorities. In our environments, we see a lot of the IT corporate environment, you know, and then we have the OT environment. But still, the myopic sort of perspective on patching still seems to be very heavily favored toward an IT discussion. Do you guys encounter that a lot, or find that as far as the maturity around IT versus OT patch prioritization?
Greg Valentine: Yeah, I would even say yes is the short answer, but I’d also say there’s justifiable resistance to change in general in the OT world, right? That includes patching which is an authorized change, but a change nonetheless. Which is why knowing whether or not it’s been approved by the OEM is so critical, right? That means somebody has tested this damn thing and it’s not breaking, it’s not kicking devices over, which is great.
Now, the customer should obviously do their own testing before they roll it out during a maintenance window, but that’s usually the process. They have to wait, they plan this out. They have a maintenance window where it’s safer—I guess is a better way of putting it—to do the approved changes, including patches and other upgrades. And then it’s validated after the fact, to confirm that everything was done appropriately.
Chris Humphreys: Absolutely. And having that single pane of glass with the data there to see, versus going into these manual processes, week validation, compliance, blind spots, components of things, you know? Those are the very low hanging fruit. Tangible use cases from an administrative perspective, I think Industrial Defender and Foxguard together holistically address these.
I always used to say, ‘Bill goes and gets hit by a bus—what happens to the patch process? But now I say, ‘Bill wins the lottery and doesn’t show up to work every day.’ I try to not be as morbid, but you know that knowledge transfer, when you’re heavily dependent on people manually executing this process, when that knowledge walks out the door, that’s a massive risk to your organization, and a major attribute.
I think being able to have that sustainable sort of solution from a technology standpoint, with controls and risk defined, means ‘Bill’ can take a vacation for a week and somebody else can pick up the slack.
Greg Valentine: Well, a real problem in the OT world right now is the graying out, so to speak, which I have plenty of experience with, Right? People are retiring, and if you’re not going through a formal process of handing over the reins to the next generation, then what are you going to do when people actually do walk out the door and retire? You know, you see it coming. It’s happening.
Chris Humphreys: We were ready for it. I was literally at a prospect yesterday, giving sort of a workshop/pitch, and the leading champion for this basically said, ‘Yeah, I’m looking to retire in the next two years. This needs to be my succession plan.’ It’s like, well, better late than never. At least you’re laying it in the right hands.
Greg Valentine: Right, and it’s good that she has one!
Chris Humphreys: But you know, seeing that she spent her entire career doing this manually and now at the end is like, ‘hey, I could have been doing it this way all this time, but I want to make sure we’re in a better place when I leave.’ It’s a very compelling argument we encounter.
Greg Valentine: You know, these systems, they’re not getting simpler. They’re getting more complicated. They’re growing. Systems are growing. Companies are making acquisitions. They’re adding sites to their environment. You want to automate it as much as possible, but you also want it to be flexible enough to be able to add additional systems, additional sites, additional devices to the overall solution.
Chris Humphreys: Yeah. And we’ll talk about this a little bit more in a minute. But the compliance blind spot component is, unfortunately, something I do see—where compliance for patching ends up being the driver to look at this holistically. At least it’s something to get you in the door, but I would argue this is a security risk based, operational efficiency best practice that you have to do regardless.
But compliance, it’s beating you with a stick, always, you know? It gets the ball rolling, but we need it to be seen as a security best practice, an overall philosophy in security 101 for OT, right?
Greg Valentine: Yeah. So especially with NERC CIP, where the penalties are huge, just to put it bluntly. Everybody is worried about the penalties and about making compliance, meeting compliance, and passing the audit basically. But in reality, that is the minimum bar from a cyber security perspective, right? Yes, great, you are compliant. Now take it further. It’s okay, there are some additional things that can be done to get you past just being compliant. Compliance is absolutely good, but it’s not great and it’s not perfect.
Chris Humphreys: I always argue that compliance is the floor, not the ceiling. I always like it when—from a maturity model perspective for utilities—NERC CIP is the floor, and then you mature towards NIST, naturally. You should have a transition plan for that. That’s where you get that security and compliance chasm being bridged over time, right? But it needs to get there, and I think it needs to get there faster rather than slower. Because as we’re seeing, the scope isn’t going to decrease as we see these regulations come into play.
Greg Valentine: Right back to CIP. Sep 15 is coming, which is the newest one, and it’s significant, it’s not small. So yeah, once compliance starts, it generally as a rule doesn’t get smaller, it tends to expand.
Chris Humphreys: And the argument I always had at NERC was, you look at supply chain and the response to Stuxnet. And then the physical security attacks, the reactive posture to put these standards in place. I would argue, from a security best practices perspective, I don’t need a regulatory requirement to tell me I need to be patching my systems, right? But that’s the world we live in, unfortunately. And then every time there’s a compromise—whether it be ransomware or anything like that—the utilities PR teams are very well trained to tell them ‘we’re heavily regulated and we’ve been compliant with all of our rules’.
Yeah, so it’s kind of that catch-22 we see: why should I have to do more than be compliant with what I’ve got right now? And I think—through our enablement with our solutions—we’re seeing how feasible it is for companies to be secure and compliant. I think it’s about showing people the light of, ‘hey, you can do this the right way,’ it actually makes things easier for you, and you’re ahead of the curve when you’re not always waiting for compliance and NERC to dictate what you’re supposed to do from an electric utility perspective.
Greg Valentine: And honestly, in the past ten years that I’ve been involved in this particular space, I’ve seen things move from just basic compliance and meeting those base requirements, to actually engaging and giving rules, becoming more cyber aware, and adding additional programs on top of that minimum baseline.
Chris Humphreys: Yeah. And I think, you know, we both do a great job with messaging around holistic models, you know, for saying, ‘let’s take the most mature control in IEC 60443 and apply it to NERC CIP.’ Let’s recommend that to our customers as a maturity model. I always talk to them and say, ‘Hey, you’re probably doing 75% of this stuff. It’s not documented on your controls document, but guess what? You’re already doing this, so let’s pick that approach for everything, right? So, I think we’re going to continue to see that evolution go, you know?
We have sort of a mapping from our solution stack to how we address it from our solutions—not just our services, but our solutions: For CIP 2, from a cyber system categorization perspective, our Cyberwatch platform does that, but I think Industrial Defender’s platform has a play in CIP 2 as well. From an inventory and identification perspective, we have ours in CIP 7—patching is obviously our bread and butter. But CIP 10, configuration change management and vulnerability assessments—I would say configuration change managementis your wheelhouse, specifically around CIP, wouldn’t you agree?
Greg Valentine: Yeah, absolutely. So, as I mentioned before, we automate the collection of the configurations. But one thing that I haven’t mentioned is that once we collect that data—and we by default do it once a day—we compare it with the baseline for that device. Thats baseline is really just a previous data collection that someone designated as the baseline. And if there are any changes, we highlight them to the customer. You can receive a notification alert, use the UI to discover it, or have a report generated, lots of options.
That process is also automated to see what’s changed. When you acknowledge a change, that’s important from a NERC CIP perspective: ‘Yes, I saw the changes and we approve them.’ Then you have a new baseline that incorporates all the changes performed during the maintenance window.
Chris Humphreys: And the sub requirements of CIP 10 for establishing a baseline, monitoring for changes against that baseline, documenting those changes—you guys are the Poster Child for September CIP 10, checking the box there. I think that works out really well, you know? For CIP 13 supply chain, we have our binaries acquisition as part of our Patchintel solution. And we always use secure FTP and half-hash values to verify that from a supply chain perspective. But I think you brought that up earlier.
You know, I think internal network security monitoring for us is going to be huge from a vulnerability correlation and identification perspective. How are you guys addressing CIP 15 for your customers coming forward? Is that an initiative on Industrial Defender’s radar?
Greg Valentine: It is, yeah. And in fact, we have a webinar coming up to discuss just this. Basically, the short answer is, if you’re leveraging the passive monitoring capability that is built into our solution—which most of our customers are—then you are doing internal network security monitoring already. We have an IDs built into our solution. We actually track all connections on the network that are happening, as long as we’re monitoring that network. So, we can tell you which devices are talking to which devices on what port, and how much traffic is going back and forth between the different systems. We can tell you if there’s any anomalous network traffic from the outside world coming in, or from the internal network going out. So, lots of ways of slicing and dicing that data.
So now what we’re working on is trying to discern how to produce the evidence for the NERC CIP auditor in order to demonstrate that, yes, we have the network security monitoring challenge under control, that we are doing that.
Chris Humphreys: And I think that aligns perfectly to complement our two solutions, just the way we always have, I think. For this use case, with our Cyberwatch platform, our ability to intake your data and make it digestible and palatable for an audit, those outputs might be the same use case. I see that being the same supportive complement that we’ve maintained for everything from NERC CIP to this point.
Greg Valentine: Yep. Absolutely.
Chris Humphreys: I think, coming to our insights, what’s next, and final thoughts, lets emphasize the importance of tailored OT patch management. You know, OT is not IT? I think we can all agree to that. I think that is a, you know, a new frontier from a compliance perspective. But it’s not new from an ICS, OT, you know, veteran type kind of perspective, right? Being able to align your solutions, your controls, your process, and your risk model to evaluate OT, and in a more stringent way than you normally would, but factoring that in and not looking at them as one in the same.
Greg Valentine: Right, right. I think there’s just inherently more planning that needs to take place on the OT space than the IT space. If you knock over a Windows desktop, yeah, it’s painful, it’s frustrating, but it’s not the end of the world. But it can literally be the end of the world if something bad happens on the OT side. I would say for patching, what’s critical is working with an accurate up to date set of information. What are the today’s: what systems, applications, and versions do I have today? As of now, I don’t want to be working with two months ago data because that might be different. You don’t want to patch something that doesn’t exist anymore because it’s been uninstalled, or it’s been upgraded or something along those lines.
So, you want to make sure it’s current and you want to make sure that it’s accurate, because you don’t want to get that extraneous data that you just don’t need. That’s just a waste of time, for lack of a better word.
Chris Humphreys: Yeah. I think when it comes to the current state of your network and your OT environments, with Foxguard and Industrial Defender, this is what we do! I mean, that’s what we were built for. That’s where we go as far as combining those solutions—visibility and automation around that is our bread and butter, together. It really is. The other thing with Foxguard, what we do on our Intel side too, is our ability to capture end of life stuff, which is also important for forecasting, as you’re looking ahead. But as far as current state—what’s going on at that very second in your environment—from an OT perspective, having you guys as our front end and us pulling that data into our solution, putting that visibility automation together for the customers, it sets the stage for everything.
Greg Valentine: It works well. I think we give you the data that you need and then you take that and correlate it to meaningful data that all of our customers need.
Chris Humphreys: Cool. Well, Greg, I really appreciate you taking the time to talk about how we do these things together. I feel like every time we’re in a prospect meeting or with a customer, we always bring up Industrial Defender in one way or another, and vice versa. You know, it’s almost like I wish you guys were in the room sometimes, you know? But, yeah, hopefully this gives people that ammo to see the combo that we have in our great team. We are together when we come up with our solutions. I think as we go forward here, in our next episode, we’re going to be talking about bridging patch management and vulnerability strategies for NERC CIP. Some of the key topics around that are going to be CIP 715 vulnerability management. And Greg, I really think it’d be good for us to revisit CIP 15 together, maybe in a later episode, as that gets closer, so we can translate what we just talked about into that use case.
Greg Valentine: Yeah, I’d be happy to.
Chris Humphreys: I think the other thing we’re going to talk about in episode three, is we’ll focus on vulnerability management for low impact. Low impact is another area where—let me ask you that while you’re here—are you guys seeing low impact NERC CIP, but still ICS OT customers? Are you seeing that prevalence in your customer base?
Greg Valentine: Yes and no. It’s client specific, Right? Some clients are anticipating NERC growing right now, more than high impact/medium impact. Well now they’re kind of hinting there may be some low impact areas that’ll need to comply with different areas. So, some clients are well ahead of the curve and are leveraging our technologies for that. But others are, you know, kicking and screaming.
Chris Humphreys: What we see—and maybe this is the use case—is a lot of the renewables that were traditionally low-impact by themselves, are now getting so big by merging their footprint that they’re growing into higher categories, and they need to do that. But then we see the municipal and cooperative utilities that might have a water side, for example, or, you know, the TSA side with gas, or maybe a little bit of generation on the electric utility side but they might have a huge water presence. With those folks, the regulation conversation is a completely different conversation, right. But it’s still a valid use case, right?
Greg Valentine: Yep
Chris Humphreys: So another thing I want to remind all our listenersof, is that we have these blog tie ins between each episode. So, of the four blogs we’re going to have, one will also be co-written with Industrial Defender. But we’re going to cover patch classification and prioritization to help focus on critical updates, patch source management, and securing the supply chain to ensure trust in patches. And then the third blog—again, co-written with Industrial Defender—is about vendor approved patchingand talks about how Industrial Defender and Foxguard’s integration maps patches to specific assets, enabling confident and traceable deployments. So, we’re looking forward to that and like we said, we do a podcast episode with four blogs in between, to keep things going. But, Greg, is there anything you want to add to close?
Greg Valentine: I think we covered it all, Chris. Honestly, thank you for the invitation. I had a lot of fun, and it’s always good to bounce ideas off of each other. I’d be happy to do it again if you want!
Chris Humphreys: Absolutely. I really appreciate it! Your partnership has always been valued with us, and I think together we form a pretty formidable duo in the OT space. And I think our customers see that and we’re not going anywhere!
So, again, thank you guys for listening, and if you have any questions or feedback, you can reach us through LinkedIn. Tell us if there’s anything you’d like us to bring up in the next episode, if you have any feedback, or if there’s anything you’d like us to bring to the conversation. We really appreciate connecting through LinkedIn.
Thanks a lot for listening and we’ll see you next time.