Patching Lessons Learned – Asset & Patch Analysis

As we rounded out 2017, we were sharing a series of blog posts pertaining to some of the best “lessons learned” with regard to Patch Management. FoxGuard has been serving the energy industry for over 25 years. We have also been providing patch management solutions for industrial control systems via original equipment vendors (OEMs), as well as directly to energy utilities for many years. We have a long history of doing this work which provides us with a unique perspective, as well as gives us extensive knowledge of the patching burden. As such, we want to share our insight and some “lessons learned” along the way.

If you missed the previous “Patching Lessons Learned” blog series posts, you can download the comprehensive Whitepaper here.

Today, we want to touch on the importance of Asset Analysis. In order to determine patch status, the utility must know which equipment they have, along with its current patch status. It is easy to assume this is as simple as gathering a list of assets. However, the list alone may not be enough. In order to determine the correct patch for safe installation, it may be important to understand the operating system, serial number or other unique characteristic of the device. Additionally, many devices have sub-components, which have unique dependencies on each other, making it more challenging to properly understand the assets and related baselines at each location. Another critical point to understand is that an aggregated list of assets may not be sufficient. Understanding the nature of the equipment and/or varying patch levels of each instance of the same piece of equipment means that each item must be inventoried and managed independently. For example, some products have patches provided based on the serial number of the device, so even though the make and model number appear to be the same, the serial number may indicate that a different patch applies from one device to the next.

Secondly, let us stress the impact and worth of Patch Analysis and mining accuracy. As stated previously, it is critical to know the baseline of each item. Equally as important is mining the patch data. The person completing this work must know where to go to obtain all of the critical information provided from the vendor. For some vendors, different parts of the information are located in different areas through the website. Without understanding this and ensuring complete accuracy in data entry, it could put certain devices at risk of having incorrect or incomplete patches applied.

WANT TO LEARN MORE ABOUT PATCH MANAGEMENT?
FoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. We host a webinar series to discuss ways to develop and implement a robust patch management program. Reserve your spot in our next session.

RESERVE YOUR SPOT

If you want to discuss something specific, we will do that too! Just reach out, tell us what your challenges are, and we will have one of our security experts contact you.

TALK TO AN EXPERT

Wand Waving

Source: Chris Thomas, Software Engineering Manager

Arthur C Clarke once wrote that “any sufficiently advanced technology is indistinguishable from magic.”  I suppose FoxGuard’s software engineering team should take it as a compliment that the Patch Gap Solution is being spoken of in such terms but it nonetheless pains me to see the technical details glossed over like that.  If we’re to be doing any “wand waving,” someone should at least put on a robe and wizard hat and explain what’s going on.

FoxGuard’s Gap Solution is broken down into two parts.  The first, which we’ll call “Asset Identification” is provided by our partner, TDi Technologies.  TDi’s ConsoleWorks application gathers information on tracked assets using safe, non-destructive scripts.  This isn’t a broad-based scan of a network – carelessness like that can knock older, more sensitive systems off line – but a polite and intelligent identification of system state.

The results of that are encrypted and shipped off to FoxGuard. 

At FoxGuard the results of that asset identification are paired up with the vast catalog of patches and assets which we track as part of our Patch Availability Report program.  This listing of “Available” patches forms the basis of the analysis yet to come.

The real magic of Patch Gap is in the relationships between patches.  You can think of the patches like the limbs, branches, twigs, and trunks (yes, trunks – plural) of a mangrove tree.  There might be more than one path from the leaf on the top of the tree to the ground and, when the tree grows a little, the path from the new-tallest-leaf to the ground might be very similar or very different.  

Storing that kind of data in traditional database or – perish the thought – a spreadsheet, is essentially impossible so FoxGuard uses a graph database to model it.  You’re probably more familiar with that technology than you think; it’s the same kind of database that underpins social networking sites like Facebook and LinkedIn. 

And just as LinkedIn can help you find the shortest path of contacts between yourself and Edward Snowden, FoxGuard’s Patch Gap solution can find the shortest path of patches between the current state of a system and its secure state.  The path, if you will, to the top of the mangrove tree.

There is, of course, a bit more to it than that.  There’s encryption, data transmission, anonymization, asset analysis, patch mining, patch identification, the problem of bi-temporal data, and a host of others besides.

But a magician never tells the audience exactly how the trick is done.

WANT TO LEARN MORE ABOUT PATCH MANAGEMENT?
FoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. We host a webinar series to discuss ways to develop and implement a robust patch management program. Reserve your spot in our next session.

RESERVE YOUR SPOT

If you want to discuss something specific, we will do that too! Just reach out, tell us what your challenges are, and we will have one of our security experts contact you.

TALK TO AN EXPERT

How the Graphics Card Shortage is Affecting the Simulation Market (and What to Do About It)

Gamers and system builders alike are experiencing a widespread shortage of high-end graphics cards, including the popular NVIDIA® GeForce® 10 series gaming GPUs. In response, FoxGuard Solutions advises customers to switch to a comparable NVIDIA® Quadro® graphics card.

What is driving the shortage?

The shortage of the GeForce 10 series is a result of two disruptions in the marketplace.

Cryptocurrency mining

The cryptocurrency craze surrounding Ethereum and other non-Bitcoin mining is contributing to the shortage. (Bitcoin mining is no longer cost-effective on consumer GPUs.) Cryptocurrency mining has risen in popularity and profitability since June 2017. It requires a vast amount of GPU resources, resulting in a drastic increase in the demand for GeForce (and other high-end) cards. 

What we expect from manufacturers

It is hard to know what will happen after the next generation is announced, but it is certainly plausible manufacturers will end (or at least keep reducing) production of the 10 series. Customers in a long queue for those cards may never receive them. FoxGuard knows of at least one OEM that has struggled to deliver its simulation solution due to the lack of GeForce. 

Replacing the GeForce 10 Series

In light of the current shortage of GeForce 10 series and the issues that often surround end-of-life (EOL) notifications, FoxGuard is advising customers to switch to a comparable NVIDIA Quadro card. NVIDIA supports these professional-grade cards and offers a minimum six-month EOL notification to reduce lifecycle surprises.

The table below suggests the comparable Quadro card that offers comparable performance.



Special pricing to support the switch

FoxGuard is offering special pricing for customers who purchase NVIDIA’s Quadro cards.  Contact FoxGuard for special programs or discounts that may be available.

Simulation solutions using Quadro

FoxGuard’s Simitar suite of computers is designed for simulation and training. Simitar FLEX is a 2U rackmount computer available with one or two NVIDIA Quadro P5000 cards. Simitar FORCE is a high-end desktop featuring the ASUS® Prime Z270 motherboard and one NVIDIA Quadro P5000.

To discuss these products or a custom solution, contact FoxGuard today.

Patch Tuesday

It’s that time of the month again – Patch Tuesday! It’s the date we circle on our calendars, plan for, and anticipate the whole month long.  You do that too, right?  Just us?  Well if you hadn’t tuned in and updated yourself about the latest in patch management for the Microsoft product suite I will summarize for you here.   Last week Microsoft released a multitude of security updates to address more than 50 different security bugs in Windows, Internet Explorer/Edge, Microsoft Office and Adobe Flash Player, among other products.  Many of the patches issued are rated as “critical” which means the flaws that the patches fix could be exploited to seize complete control over vulnerable systems.  The latest Windows monthly patches are essential and if you use Outlook, there are two especially critical updates to install as soon as possible.  The Microsoft Outlook vulnerabilities could let the “bad guys” into your Windows system just by getting you to click on a link, document or visit a compromised/hacked Web page.

Adobe patches for the month included security updates to address critical vulnerabilities.  Even though Adobe Flash is being phased out (completely by 2020), there are serious vulnerabilities addressed by this month’s patches.  In fact there are active attacks on these Adobe Flash vulnerabilities which means that it is suggested to patch now. 

Lastly, Microsoft offered an updated advisory (ADV180002). This advisory was originally released in January but underwent several updates since then. The latest version released on “Patch Tuesday” includes references to new updates released for Windows 10 (32-bit) to mitigate speculative execution of side-channel vulnerabilities associated with notorious security bug, “Meltdown”.  This advisory also states that there is no release schedule for older versions of Windows, but that they are working on releasing updates for pre-Windows 10 operating systems.  As for Windows Server 2008 and Windows Server 2012 platforms, customers who are awaiting a fix were told in this advisory that, “Addressing a hardware vulnerability with a software update presents significant challenges with some operating systems requiring extensive architectural changes. Microsoft continues to work with affected chip manufacturers and investigate the best way to provide mitigations.”

As evidenced each month with Microsoft “Patch Tuesday” and especially now with more frequent out of band security updates, we recognize that the burden and importance of a comprehensive patching program is higher than ever before.  The number of “critical” security updates each month is increasing and the ability to stall or delay installation of a patch just isn’t in the cards.  We firmly believe that our Patch Management program offers the best protection for these and other type of updates.

For more information on these vulnerabilities, please see: https://isc.sans.edu/forums/diary/February+2018+Microsoft+and+Adobe+Patch+Tuesday/23341/

https://krebsonsecurity.com/2018/02/microsoft-patch-tuesday-february-2018-edition/

 

WANT TO LEARN MORE ABOUT PATCH MANAGEMENT?
FoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. We host a webinar series to discuss ways to develop and implement a robust patch management program. Reserve your spot in our next session.

RESERVE YOUR SPOT

If you want to discuss something specific, we will do that too! Just reach out, tell us what your challenges are, and we will have one of our security experts contact you.

TALK TO AN EXPERT

Energy Management Control Systems

ENERGY MANAGEMENT CONTROL SYSTEMS KEY INSIGHTS TO IMPROVE CYBER SECURITY Ensuring effective cyber security protections are in place for critical infrastructure has been a priority of US Federal agencies, including the Department of Defense (DoD), for some time.  Cyber security guidance an regulations for critical infrastructure have been published by nearly every Federal agency.  

Take the First Step

Source:  Roger Rademacher, Solution Architect

The First Steps

“Every journey starts with the first step”

Unfortunately, the cyclical nature of patch management ensures that the journey never ends and we take that first step over and over again.  It is kind of like training on a track team.  We sprint around the track a couple times, take a short break (if you are lucky), and do it again.  With patch management, we up the stakes a bit and hope that we don’t find ourselves in sand or missing a shoe at the beginning of each lap.  

A mature patch management program starts with a master baseline of what we had (always past-tense) and establishes the scope of the program.  What assets do I care about?  What software is running/installed on them?

And then we ask the hard question… Are there any updates?

 

What’s in a Name

I hate to be captain obvious and I shouldn’t be the first to tell you… you can’t find what you want if you don’t know what you are looking for.  Naming standards build a solid foundation to all asset baselines; thus, all patch management programs.  No program is 100% automated, there are always outliers and manual data entry is inevitable. 

If you start with an automated solution then I hope it uses a well-formatted naming standard and is easy to understand.  If not… well, manual data entry will erode the effectiveness of the program.

If you are starting from the ground up then you will quickly find yourself in the position to make a critical decision.  Go for what’s behind door number 3… or adopt something like the common platform enumeration (CPE) standard.

part:vendor:product:version:update:edition:language:sw_edition:target_sw:target_hw:other

A well formatted name (WFN) will provide, at first glance, most of what you need to know in order to identify any asset and start the search for available updates.  Of course, after your first search, I hope you drafted up a treasure map to find your way back.  The “other” field can be very useful.

Buried Treasure Escape Room

That which we are looking for always seems to be buried under a mountain of web pages, emails, alerts and other clues.  To find our way out of the escape room we need to follow the clues, find the treasure and deliver it before time runs out… then do it again.

Think of all the effort it takes to establish good communications with vendors, to determine how they notify customers of security patches and details, and how to best record the results to allow validation processes to be useful.  Mining documentation can serve as our treasure map.  Figure it out once then skip over the hard part next time.  Your mining documentation and their results should utilize the same naming standard in order to maintain correlation with the assets that you are tracking.

Lost in Translation

When I crawled out from under the rock I first noticed that most vendors had multiple personality disorder.  They were switching between abbreviated, full, and legal names (and monstrous mutations) to reference themselves and their products.  On occasion I even found version labeling discrepancies where leading 0’s were removed to save space (ex. 5.001 became 5.1).

It is a safe bet that you will need to bridge the translational gap between what you have in your master baseline with what the vendor’s appear to have on their websites.  Mining documentation helps in some regard but whatever system you have needs to be able to associate your treasure with an owner.  Patches need to be associated to assets and devices in order to be actionable.

Six steps later

  1. Find out what you have
  2. Give it a name that makes sense to you
  3. Document your search
  4. Trace your results back
  5. Take action
  6. Do it again

At some point, the master baseline will change due to an update or addition.  When that happens you will be ready.

WANT TO LEARN MORE ABOUT PATCH MANAGEMENT?
FoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. We host a webinar series to discuss ways to develop and implement a robust patch management program. Reserve your spot in our next session.

RESERVE YOUR SPOT

If you want to discuss something specific, we will do that too! Just reach out, tell us what your challenges are, and we will have one of our security experts contact you.

TALK TO AN EXPERT

Custom Industrial Computing Leader Introduces Products To Support The Energy Industry

CHRISTIANSBURG, Va., Jan. 18, 2018 /PRNewswire/ — FoxGuard Solutions, Inc. announced the release of GRID-FS1, a new fanless rugged industrial computer for substation application. Designed to meet IEC 61850-3 and certified for IEEE 1613, the GRID-FS1 is an ideal communication gateway, SCADA system, or IOT platform. For Additional Product Information, Click Here: https://foxguardsolutions.com/product-categories/rackmount/ “We designed the GRID-FS1 to be the perfect platform for ISVs to deploy their applications,” says Anderson Peeples, Technical Program Manager for the GRID-FS1. “At FoxGuard, we ship over 40% of our products internationally and understand the challenges faced when deploying solutions across the globe. You shouldn’t have to redesign your system for each unique deployment and that’s the mindset we had when we designed the GRID-FS1. With versatility in mind, the GRID-FS1 is offered in wallmount and rackmount configurations with wide range AC or DC power supplies. It’s well suited for the high temperatures, shock, and vibration common to industrial environments and utilizes long life cycle embedded components so you won’t be surprised with any sudden EOL dates.” SecurityMatters provides a passive automated network monitoring solution for industrial environments empowering asset owners to identify, react, and respond to industrial threats and flaws. SecurityMatters selected the GRID-FS1 for their flagship software product, SilentDefense, to provide a turnkey product to the substation medium and low voltage energy market. SilentDefense provides instant OT network and process visibility, and reports internal and external cyber threats in a clear and actionable way. As a result, operators can easily identify the source of a threat and take quick responsive action. “FoxGuard demonstrated advanced capabilities to model products before building, which stood out from all others, and made us feel more like a partner than a customer,” said Cliff Gregory, Chief Executive Officer USA of Security Matters. “We are better together.” About FoxGuard Solutions FoxGuard Solutions, Inc. has been bridging the gap between IT and OT technology environments for over 35 years via integrated hardware, software and security solutions. Based in Southwest Virginia, FoxGuard serves customers in more than 60 countries from their secure, ISO-certified, ITAR-registered facility. Providing configuration, testing, imaging, certification, integration, regulatory/export compliance, and life cycle management programs, FoxGuard’s solutions are “Built for Security.” Learn more at www.foxguardsolutions.com Media Contact: Jonas Baranauskas p. 877 446 4732 e. jbaranauskas@foxguardsolutions.com

Patch this… Wait! Don’t patch that (yet)

Cybersecurity is a top concern for utilities and power companies. While cybersecurity as a whole has many facets, one of the top cybersecurity threats that a utility or power company is likely to face is unpatched software.  With the number of devices and connection points to the grid increasing every day, the chance for a breach is higher than ever before and the burden of patching ever increasing, and might we say even puzzling for the average operator. 

There have been many recent threats (most recently named incidents “Spectre” and “Meltdown”) that have been a wake-up call to take the matter seriously. The Meltdown and Spectre vulnerabilities, first revealed at the beginning of the year, affected most anything with a chip in it which made the process of releasing patches justifiably grueling. Every type of impacted hardware and software required its own specially tailored solution, and even a fix that worked as intended for one product may have had inadvertent results on other system processes requiring recalls on certain patches and in general propagating confusion.  Likewise, patches were certainly not all encompassing.  A fix might have been released for product X and Y, but not include products A, B, or C. 

Developing stable patches for every processor, every firmware stack, and every operating system adds up to a tall and arduous mission. Meltdown and Spectre were critical enough vulnerabilities that they certainly needed to be patched quickly, even if this meant moving forward with imperfect fixes or leaving some devices patched and others “un-patched” waiting for a necessary fix to be released.  Organizations continue to struggle with understanding whether they have the right updates installed to actually protect their systems without causing more problems. 

We know patch management can be time consuming and very labor intensive and as said above even puzzling at times like with Spectre and Meltdown. Utilities can spend significant time and resources manually searching websites, receiving vendor notifications, calling vendors and tracking patches. FoxGuard offers various levels of Patch Management from Patch Availability Reporting, Patch Applicability Reporting to Patch Deployment.  And, FoxGuard has just released a Patch GAP Analysis solution.  Have you heard of it?  Our solution determines the current patch level of your assets and analyzes the relationships between your packages, updates and dependencies of available patches. It even offers vulnerability intelligence by identifying risks, threats, and vulnerabilities based on missing patches.  Now wouldn’t that be imperative in situations like with named Meltdown and Spectre.  Don’t be distracted by the exploit of the week. Invest your time and money defending against the threats you’re apt to confront i.e. unpatched software. 

WANT TO LEARN MORE ABOUT PATCH MANAGEMENT?
FoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. We host a webinar series to discuss ways to develop and implement a robust patch management program. Reserve your spot in our next session.

RESERVE YOUR SPOT

If you want to discuss something specific, we will do that too! Just reach out, tell us what your challenges are, and we will have one of our security experts contact you.

TALK TO AN EXPERT

Finding The Missing Link

Source: Roger Rademacher, Solution Architect

“I don’t know where to start”

Said pretty much everyone who was hired into or bravely volunteered to captain a patch management, asset management, or configuration management program.

It’s easy to dismiss the complexities of managing any program that must deal with changes; particularly those where changes may be catastrophic.  Let’s face it, we don’t want the finger pointed at us when forensics discovers the root cause.

Before you sit down in the captain’s chair to manage any program, big or small, you need to understand what you are getting into.  You will find there are some very foundational practices you can implement to make your job much easier.

For the sake of examples, let’s use a patch management program as an example for the remainder of this post.  Of course, I am biased for somewhat obvious reasons.

Know what you are working with

My first advice is to understand the scope of the program. 

This means that you need to know what physical and virtual devices are out there, as well as, what software is running on them.  That includes applications, operating systems, and firmware and their associated versions.

You may aggregate this data into a device specific “software baseline”, a program wide “software library”, and anywhere between.

Important point #1 – You should be able to link unique instances of applications to unique devices. 

Changes happen

“What changed?”

Whether accidental or not, when something goes wrong this is usually our first question.

When we install a patch we are introducing a change into the environment.  That change may consist of updated files, configuration changes, new services, or any other modification that might impact operations.  How many vendor patches are accompanied with copious notes on their changes?

Patch management solutions vary in scope and often target the most widely used and insecure applications.  What happens when your application is out of scope?  Many of us are forced to sneaker updates to end devices.

Important point #2 – You should be able to detect version changes to applications and track those changes back to a change request.

Important point #3 – You should be able to detect changes to application configurations and track those changes back to a change request.

Trust but… Validate

“Where did that file come from? Is it a security patch?  Does it install? Did it break anything?”

Vendors should be verifying internally whether a patch is acceptable within their own testing requirements.  It is the responsibility of the asset owner to validate the acceptability of a vendor’s patch in an integrated environment.

Validation is the action of proving the validity or accuracy of something and declaring it acceptable.

We may validate many aspects of a patch including…

  • A file is, indeed, from the identified patch source
  • The applicability of a patch within the scope of our patch management program
  • The patch may be installed without error and we can discern that installation
  • And, above all else, the file does not adversely impact operations.

Validation may also be considered the QA of someone else’s work and the completion of that work when necessary. 

Important point #4 – Your validation procedures should support all patch management processes.

SANS Boston Security Training Event to Detail How to Protect Industrial Control Systems

SANS Boston Security Training Event to Detail How to Protect Industrial Control Systems

NEWS PROVIDED BY
SANS INSTITUTE

BETHESDA, Md., Jan. 24, 2018 /PRNewswire-USNewswire/ — SANS Institute, the global leader in cyber security training, today announced the agenda for SANS Boston Spring 2018 taking place in Massachusetts March 25 – 30. Included among the course line-up is ICS410: ICS/SCADA Security Essentials which will arm security professionals and control system engineers with the cyber security skills they need to defend national critical infrastructure.

According to Monta Elkins (@montaelkins), Hacker-in-Chief FoxGuard Solutions and a SANS ICS410 course instructor, “Want a real cybersecurity challenge? In Industrial Control System security you’ll face the newest, incredibly sophisticated, most well financed and executed nation state sponsored attacks on the planet. How’s that grab you? Want to learn how to defend these systems? Join me at the SANS Boston Spring ICS410 class where I will share the cyber security concepts needed to defend your critical infrastructure.”

For additional information on the ICS410: ICS/SCADA Security Essentials course or to register, please visit: www.sans.org/u/zNL

SANS Boston Spring 2018 features hands-on immersion style training courses covering a variety of topics including cyber security, security management, incident response and digital forensics and industrial control system. Some of the courses offered include SEC401: Security Essentials Bootcamp Style, SEC542: Web App Penetration Testing and Ethical Hacking, FOR508: Advanced Digital Forensics, Incident Response, and Threat Hunting, and MGT512: SANS Security Leadership Essentials for Managers with Knowledge Compression™. SANS Faculty Fellow, Rob Lee (@robtlee), will deliver the keynote address Welcome Threat Hunters, Phishermen, and Other Liars.

For a complete list of courses, or to register for SANS Boston Spring 2018, please visit: www.sans.org/u/zNQ

About SANS Institute
The SANS Institute was established in 1989 as a cooperative research and education organization. SANS is the most trusted and, by far, the largest provider of cybersecurity training and certification to professionals at governments and commercial institutions world-wide. Renowned SANS instructors teach over 50 different courses at more than 200 live cybersecurity training events as well as online. GIAC, an affiliate of the SANS Institute, validates employee qualifications via 30 hands-on, technical certifications in information security. The SANS Technology Institute, a regionally accredited independent subsidiary, offers master’s degrees in cyber security. SANS offers a myriad of free resources to the InfoSec community including consensus projects, research reports, and newsletters; it also operates the Internet’s early warning system–the Internet Storm Center. At the heart of SANS are the many security practitioners, representing varied global organizations from corporations to universities, working together to help the entire information security community. (www.SANS.org)

SOURCE:
SANS Institute

RELATED LINKS:
PR Newswire release: https://www.prnewswire.com/news-releases/sans-boston-security-training-event-to-detail-how-to-protect-industrial-control-systems-300587497.html

http://www.sans.org

 

FoxGuard Solutions Media Contact
Marcie Killen
Marketing Manager
p. 540.382.4234 x152

WANT TO LEARN MORE ABOUT PATCH MANAGEMENT?
FoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. We host a webinar series to discuss ways to develop and implement a robust patch management program. Reserve your spot in our next session.

RESERVE YOUR SPOT

If you want to discuss something specific, we will do that too! Just reach out, tell us what your challenges are, and we will have one of our security experts contact you.

TALK TO AN EXPERT