Working with Relays in the Era of Record NERC Penalties
NERC’s first enforcement action of 2019 was a whopper: a $10M penalty levied on a large, multi-state utility for violations of Critical Infrastructure Protection (CIP) standards. This is a wakeup call to the electric power industry on two counts. First, the level of detail of the NERC disclosure, which consists of over 700 pages in four parts, signals that utilities cannot cut corners in CIP compliance. Second, the size of the penalty, which is the first eight-figure penalty for CIP violation, signals severe financial implications for egregious non-compliance. This latest NERC action follows one in 2018 that saw a then-record $2.7M.
Examination of the Notice of Penalty of this enforcement action reveals that NERC expects a well-thought through approach to CIP compliance. For instance, some of the commitments by the entity, as part of the settlement, are to add security and compliance resources, invest in tools, and conduct industry surveys and benchmarks of best compliance practices. In other words, it is not sufficient to have a program that does just enough. There is clearly an expectation that organizations would identify and implement industry best practices.
What are the implications of this new aggressive approach by NERC on substation engineers? What are the best practices relevant to CIP compliance in working with relays and other such critical substation cyber assets? We focus here on two key aspects: (1) Transient Cyber Assets (TCAs) that are used to work with the substation devices, and (2) Security Patch Management that ensures timely identification, evaluation and implementation of security patches. In our experience, these two are among the most challenging requirements because they entail frequent tasks that are performed by multiple team members, all of whom have to play their part correctly.
Transient Cyber Assets (TCAs)
Typical approaches to TCAs have been IT-centric, wherein vendors and utilities repurpose existing IT solution templates for substation work processes. In the case of TCAs, a utility might reimage them to approved baselines prior to use or permanently place a hardened laptop in the substation for any work with the substation assets. These approaches are cumbersome and ultimately ineffective when applied to substation environments.
For instance, unforeseen circumstances come up in the field, such as the need to connect to serial ports through USB, which cannot be easily solved in the remote substations. This is a common malaise of TCAs that are hardened in a one-size-fits-all fashion. In the cases of laptops that are permanently placed in the substation, the TCAs often fall out of maintenance since they are not centrally managed and updated on an ongoing basis. Furthermore, the field personnel do not have working familiarity with the TCA if it is permanently located at the substation, and not a device they carry and use everyday.
A more effective security and compliance approach for the TCAs is to start with the work processes that need to be protected and tailor IT security controls to them. In order to realize this vision, IT, Operations, and Compliance departments have to work closely together in order to develop a clear, common understanding of the work processes and compliance requirements. Off-the-shelf security products may not entirely address the unique considerations of field devices, and custom security controls may have to be developed.
In partnership with leading utilities, we have developed and widely deployed a TCA solution, the DUCe (Doble Universal Controller enterprise), that takes a holistic approach by considering the viewpoints of IT, Operations, and Compliance. The DUCe solution consists of several security controls that can be configured to best fit the utility infrastructure and work processes. Key aspects include:
- TCAs are not used for email, Internet access, or general non-field corporate work.
- The TCA support infrastructure is separate and isolated from the utility IT infrastructure, thereby preventing malware propagation from the corporate network to the substation network.
- All interfaces other than the ones required for connecting to the BCA are disabled prior to working with the BCA.
- Data management features automatically deliver work items such as test plans and settings files to the field device and retrieve the artifacts of completed work back to designated databases or fileshares.
Security Patch Management
In the case of Security Patch Management, the CIP-007 requirements on patch discovery and the strict timelines for handling discovered patches can be onerous. Satisfying the requirements can involve multiple teams performing recurring tasks, all within strict timelines. Patch discovery in particular can be resource intensive. All told, a large utility may have hundreds, and perhaps even thousands, of product models and applications. For this fleet, the utility has to identify update sources, keep the update sources current, and check the sources regularly. Once an update is discovered, there is typically additional communication with the vendor to obtain release notes and additional information about updates, since many vendors don’t provide these as part of the public release; and verify the authenticity and integrity of the updates.
Doble’s patch management solution, PatchAssure, provides patch discovery for substation devices and transient devices, and incorporates a workflow that allows for the verified patches and instructions to be automatically pushed to the field devices for application. The workflow is designed to encompass the entire patch management lifecycle, so that it bridges the typically found gap between the patch discovery and analysis by security analysts, and the deployment and verification to substation devices by field personnel. Often there is disconnect between the two business processes, creating room for misses and violations. When PatchAssure and DUCe are deployed together, they provide a secure and seamless experience for working with substation devices. Such synergistic and coordinated approaches ensure CIP compliance and also increase productivity.
ADDITIONAL INFORMATION
- Product Information: Doble PatchAssure Patch Management
- Further Reading: NERC Enforcement Actions 2019
- Learn More: Grid regulator hits utility with record $10M fine
- More on NERC: FERC approves $2.7M cyber fine without naming names
- Product Information: Doble DUCe TCA
