- CYBER SECURITY
- INDUSTRIAL COMPUTING
- ABOUT US
Source: Monta Elkins, Hacker-In-Chief
Was your security patch “available” in May when it appeared for the first time on your vendor’s website or was it “available” in January as it says on the vendor’s published documentation–that didn’t appear until May?
Tracking security patch availability dates is key to compliance and for proving compliance during an audit. Unfortunately, on-line vendor information documenting patch availability dates can vary over time. In our process of tracking patch availability for Industrial Control System entities, FoxGuard has discovered multiple incidents where vendor patches appear in vendor documentation in an apparently “backdated” fashion. That is to say, checking a patch source diligently (every 35 days according to the CIP standard for instance) might suddenly reveal an available patch that is dated from 3 months prior, without warning. I call these patches “retropatches”.
Retropatches are a problem in audit, because an entities’ documentation for installation of security patches will show that, at minimum, this retropatch was installed 3 months after it was available according to the vendor documentation. It can be hard to prove to an auditor that retropatches suddenly appeared and were not instead missed by the entity 3 months prior, as the auditor might reasonably suspect.
Before looking at some examples of retropatches from FoxGuard audit documentation, I’d like to clarify that I don’t have any evidence nor do I believe that retropatches represent any malevolence on the part of the software patch vendor. The original term we used was “backdated patches,” because, when the patch appeared on the vendor website, it appeared to be backdated. I believe this is just an artifact of perspective. From the vendor perspective it’s likely the patch process began at the earlier date, but for various circumstances (perhaps documentation, testing, clarification, etc.) the actual release date was pushed back and the related documentation was not edited/updated showing the new actual release-to-customers date. Because I think the term “backdated patches” could imply malicious intent on the part of the vendor I recommend “retropatches” as a more neutral term. I would also bet that most vendors have no idea that their customers care about the documented release date of a patch, or that it might theoretically cost a customer $1 million per-day-per-event in the case of a CIP audit. Vendors that I have spoken with who track such things show some very low update rates for security patches overall. That a small percentage of their customers care whether the documentation is dated Jan 2018 or June 2019 is likely unfathomable to them.
It could even be that some vendor(s) defines the “patch date” to be the date the patch was created, as opposed to defining it as the date the patch was released to their customers. It’s easy to understand that many processes can exist between writing and releasing a patch that would separate those dates, testing for instance.
Let’s look at 3 examples of retropatches from FoxGuard’s documentation:I picked these examples only because they are representative and not to call out these vendors in particular. There are, after all, many reasonable processes that can cause this to occur. In these screen shots note the dates displayed in the bottom right hand corner. I have cropped the screen captures for clarity and to de-emphasize vendor identities somewhat. I also use a term we’ve coined internally, “patch mining,” to refer to the process of collecting all of the information about a patch.
Figure 1 captured June 6, 2018 shows the latest available driver software is version 23.1 dated February 21, 2018. But in figure 2 captured June 22, 2018 it shows the latest driver software is version 23.2 dated May 30, 2018. We might expect that firmware to appear in our collection on June 6, but it does not. If you revisited the site on July 6, (a month after your first visit) and managed to evaluate the patch on that very same day, an audit seeing a May 30 firmware date, would still conclude it took you 36 days: outside of your 35-day window.
An especially forgiving auditor might overlook a one-day gap, but some retropatches are anachronistic by months. See example 2.
Retropatches and out of order firmware version example.
In Figure 3, note the patch mining date at the bottom right hand of the page is May 24, 2018. The vendor documentation states that the most current patch available is Version 6.24.005 dated December 22, 2017. So far so good.
Now look at Figure 4 where the patch mining date is June 22, 2018, a little less than a month from the previous information collection. In this figure we see the vendor documenting multiple patches that should have appeared in the mining information from Figure 3. In particular, this screen shot documents a Version 6.24.011 dated April 24, 2018, which we would have expected to appear in the May 24 information collection.
Now, assuming you updated your software the very day of collection (May 24, 2018), from an audit perspective you have potentially missed firmwares:Version 6.24.006 January 23, 2018Version 6.24.007 February 23, 2018Version 6.24.008 February 22, 2018 * additional retropatchVersion 6.24.009 March 15, 2018
Depending on your calendar the following two updates might be in processVersion 6.24.010 April 19, 2018Version 6.24.011 April 24, 2018
And the latest Firmware Version 6.24.012 should be entering your patching pipeline. This puts you months out of date and out of compliance.
Also notice that firmware version 6.24.008 is dated BEFORE version 6.24.007. In this instance the retropatch phenomena causes the firmware version numbers to be non-sequential. In our follow up emails with the vendor, to provide additional documentation for discrepancies, the vendor clearly says that this was a special case based on internal build dates and that this should be the only instance of this discrepancy.
It matters exactly where and how you get your patch information. Vendor information can be differ between multiple documents from the same vendor. In Figure 5 we see the vendor website indicating that the latest firmware version for a particular product is [Vendor X]-[Product 4]-5-R324-[blah] dated January 25, 2018.
Figure 6 shows that the documentation manual for the product (mined on the same date as above: see bottom right hand corner) lists a newer version of firmware as current: version [Vendor X]-[Product 4]-5-R325-[blah] dated March 29, 2018
In this example we see different latest firmware version numbers collected on the same date from different documents on the vendor website. Looking at the bottom-right of both pages we see this seemingly contradictory information was mined on the same date: July 19, 2018. It’s not hard to imagine where documentation from email, newsletters, etc. might differ from that available on a web page.
It’s important to carefully document your patch information gathering (patch mining) as well. In addition to the information we enter into our patch information database, FoxGuard also documents the actual act of patch mining with screen shots and timestamped evidence. That gives us and our customers a visual record in the form of snapshots to support audit concerns should a question arise. The only evidence of exactly what you saw on a vendor’s site, is the documentation you make yourself of that visit. Because different sources from the vendor may conflict, your process should be consistent and document exactly where you mine patches from (email, web site, and other documentation). Be aware that firmware versions may appear out of order and that the most recent patch version may not be the one with the highest version number. If you are responsible for patch mining in your organization, best practice would be to store this meta-documentation as well, for every product firmware, from every vendor, every month, as you mine it.
WANT TO LEARN MORE ABOUT PATCH MANAGEMENT?
FoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security of their systems and prepare for NERC CIP audits. We host a webinar series to discuss ways to develop and implement a robust patch management program. Reserve your spot in our next session.
If you want to discuss something specific, we will do that too! Just reach out, tell us what your challenges are, and we will have one of our security experts contact you.