Security Patch Management for Field Devices
Security patch management, long an arcane field, finally entered public consciousness this year. Several critical vulnerabilities were disclosed, such as the Windows SMB Remote Execution Vulnerability disclosed in March (CVE-2017-0143). Malware such as the WannaCry ransomware that exploited these vulnerabilities wormed through unpatched computers across the world and sowed chaos. Some of the exploits were based on an allegedly leaked NSA exploit kit. The WannaCry ransomware and the NSA exploit kit were just two among the numerous cybersecurity stories that received wall-to-wall media coverage this year. These and other incidents such as the various Android exploits really brought home the importance of timely security updates.
Due to this bumper-crop of vulnerabilities and exploits this year, system admins have had their hands full. The patch discovery and application task is especially complicated for utility field devices. These devices are laptops and tablets that are used for asset testing and management tasks such as protection relay testing and applying firmware updates to substation cyber assets. There are a number of challenges with security patch management for the field devices:
- Remote and Unreachable: The field devices are often at remote locations and don’t have network connectivity. The device users may turn the device off when they are not in use, further reducing update opportunities.
- Specialized Build Images: Due to the specialized needs of the field crew, the device may be different from the corporate images. Furthermore, multiple images may be in use for different field crews – e.g., substation testing and SCADA testing.
- Unique Software: Many of the software applications on these devices are not used elsewhere in the corporate environment, and may be old and not easily patchable. Not only are the IT personnel unfamiliar with the software applications, but the discovery and application of patches may not conform to the standard corporate means.
An effective security patch management program for the field devices would consist of the following elements:
- Monitoring to Ensure Patching of Unreachable Devices: Real-time monitoring capability is needed for awareness of the patching status of all the devices, so that appropriate intervention can be undertaken for devices that aren’t patched in a timely manner.
- Sophisticated Asset Management: Asset management system that has accurate and up-to-date information on the various field devices, images, software applications, and versions in use, so that they can be updated in a manner consistent with their use.
- Comprehensive Vulnerability Monitoring and Patching: A vulnerability monitoring program that continuously monitors, identifies and evaluates new vulnerabilities and corresponding patches or workarounds; and multiple patch deployment means in order to ensure that even the most difficult applications can be remotely patched – when all else fails, it may take the entire application to be replaced with the new version.
The purpose of field device security patch management, ultimately, is to ensure the safety of the grid. With this in mind, security patch management should be looked at as part of a larger vulnerability management program. The most common ways that unpatched vulnerabilities are exploited is by targeting unpatched services to gain access or escalate privileges after initially gaining access through social engineering. It follows that the most effective way to protect critical grid systems is to not use the field devices as general purpose corporate computers. By using dedicated field devices, the attack surface is drastically reduced by taking away exploitable services and social engineering vectors such as email and web browsing. This is an essential and complementary aspect of a holistic field force vulnerability management program.
Security patch management has NERC CIP implications as well, with security patching requirements in CIP-007-6 for Bulk Electric System Cyber Assets and CIP-010-2 for Transient Cyber Assets. For NERC CIP audit purposes, careful collection and retention of evidence is necessary. A well-designed security patch management program can greatly enhance system security and provide NERC CIP compliance without impinging on work performance.
Doble has put in place security patch management programs for multiple utility field fleets. In our experience, security patch management for the field devices is a unique blend of art and precisely measurable enterprise, one that takes in-depth understanding of the asset management work processes to get right.
Additional Resources:
- Analysis of WannaCry Ransomware
- Shadow Brokers and Leaked NSA Exploits
- NERC Glossary of Terms
- NERC Standards and Effective Dates
- NERC CIP-007-6 for System Security Management
- NERC CIP-010-2 Requirement R4 for Transient Cyber Assets and Removable Media
