Electric utility operational technology (OT) networks have largely transitioned from legacy and time-division multiplexing (TDM) Synchronous Optical Network (SONET) networks into modern Internet Protocol (IP) and Ethernet packet networks. The SONET and TDM network technology served the electric utility OT wide area network (WAN) well over several decades, but few vendors still support this technology, driving network modernization efforts. At the same time, internet service providers (ISPs) are phasing out the same legacy systems and further decreasing the market for this technology and associated components.

Data needs continue to rise for grid operations, especially as more communications are needed to control and monitor distribution assets. Because of their increasing size and complexity, electric utility OT WAN networks are becoming nearly impossible to simulate in a physical lab environment. Without the capability to test at some sort of scale, change implementation is being forced into production OT networks.

Implementing changes on a production network carrying traffic needed to operate the bulk electric system introduces unnecessary risks to the OT network. These changes could include:

• Firmware upgrades
• Policy updates
• New protocol implementations
• Architecture changes
• Network optimizations

It has been common practice to deploy and test network changes in physical lab implementations to determine how they will impact the production network, typically at a reduced scale. There are now alternatives to physical labs for testing changes, including testing at scale, without the cost, power and space of a physical lab. Using a digital twin of the OT network can address many challenges to testing in a physical lab and provide a more accurate replica of the scale of a production network, leading to more reliable simulation results.

An OT digital twin network can be implemented virtually, increasing flexibility and agility when conducting network simulations and testing. It is possible to scale up or down in a virtual environment to suit the simulation’s or test’s needs. These OT digital twin networks can be replicas of the production network, so test results provide increased knowledge of overall production impacts, increasing confidence in network change rollouts and minimizing overall risk introduced to the OT production networks.

The use of virtualization technologies to implement electric utility OT digital twin networks will lead to increased accuracy of simulation results, reduce risk introduced to the production network, optimize network planning, and be crucial to efficiently and effectively operating these networks.

Virtualization software creates a compatibility layer between physical hardware and emulated software. A traditional virtual machine simulates the hardware on a physical device, such as the CPU, memory, hard disks and network cards. Software that creates and runs virtual devices is referred to as a hypervisor. A disk image that includes an operating system and kernel resides on the hard disk.

Virtual machines are functionally identical to physical machines but offer flexibility in resource assignments and other benefits, depending on the hypervisor used and constraints required by the operating system. Routing vendors provide images and documentation for implementing their equipment in virtualized environments. Some products are meant for testing and include rate limits that prevent their use in production, while others are fully featured and meant for production use.

There is an increase in physical hardware requirements when using virtualization, compared to running on bare-metal hardware, but the benefits of virtualization generally outweigh this disadvantage.

Hypervisors come in two types:

• Type 1 hypervisors, also known as bare-metal hypervisors, are purpose-built operating systems for virtualization. This allows equipment to use the underlying physical hardware more efficiently.
• Type 2 hypervisors are software installed on top of an existing operating system. These are easier to use but cannot allocate hardware resources that are otherwise needed by the operating system running on the physical machine.

Both hypervisor types operate using the same principles, but there is a trade-off between usability and efficiency.

Containers are another option. They are an evolution of machine virtualization meant to be more flexible and lightweight. Containers use the hypervisor’s kernel to communicate with the physical hardware, so only a stripped-down operating system image is needed to create a container. No hardware-level drivers are needed in a container; only the binaries and configuration files to run the containerized application are required. This results in the shared kernel architecture using fewer resources and disk space when compared to traditional virtual machines.

It is worth considering that the container environment having access to the hypervisor’s kernel creates additional security constraints that must be accounted for. Neither solution is strictly superior when it comes to virtual machines versus containers, but the efficient and flexible nature of containers has contributed to their popularity.

This paper discusses several virtualization technologies but is not intended to be a complete, in-depth review of all technologies available in the industry. Virtualization technology implementation should be decided on a case-by-case basis, according to the needs of the OT network operator. Since either kernel-based virtual machines or containers can be used to create virtual routers, the term virtualization is used here to refer to any of the technologies for the scope of this paper.

Once the appropriate vendor and virtualization approach has been determined, these technologies can be leveraged to benefit network operators. Using this technology to create a digital representation of physical architecture is one of the most useful applications for OT network operators.

Virtualization presents an agile way to prototype different network topologies. There is no need to procure specific hardware to test how an architecture will function. Topologies can go from conception to proof of concept in the time it takes to initialize a virtual machine and provision the intended configuration. This enables engineers to more easily develop potential network architectures to fit their needs, reducing investment in equipment that does not suit their requirements as expected.

Introducing an OT digital twin network alongside an OT production network produces increased flexibility, agility, and efficiencies in simulations and testing while reducing the risk to OT production networks that can be caused by network changes. OT digital twin deployments can be implemented on a handful of servers, replacing the multiple racks of physical equipment required to simulate the same scale.

OT digital twin networks can deploy many simulations as needs arise without additional physical work, such as rewiring. A baseline digital twin that mirrors the production network may be used for each simulation as a starting point, eliminating the need for time-consuming individual network equipment resets. The baseline digital twin should be synchronized with the network regularly to confirm accurate simulation and testing scenario results before rolling out changes into the OT production network.

The ability to host multiple simulation and testing scenarios also enables a unique training environment that can be used to educate OT network engineers on the operations of the production network. Training can include break-fix scenarios to enhance the technical skill sets of OT network teams and increase the ability to maintain and understand the production network. This advanced training capability improves knowledge in OT production network operations and reduces the time to identify, fix and verify problems on the OT production network.

A project that required the complete replacement of routing hardware to support necessary software upgrades serves as an example of an OT digital twin implementation. This project had critical limitations in physical hardware access, as well as software compatibility issues that would have severely hindered the necessary prework and testing required to perform the work, resulting in extended outages and system instability.

Virtualization was used to streamline the workflow for this project, which needed to replace complex in-service routing equipment while mitigating risks. Updates to the configurations — including syntax changes specific to the new software and complete port remapping due to physical differences on the new hardware — could begin well before receiving physical materials, closing the gap between the project’s inception and production execution.

By the time physical equipment had been received and installed for factory acceptance testing, full configurations were already completed and ready to load onto their physical counterparts. This not only shortened the project’s overall timeline, but also reduced the testing time for the equipment. The configurations proven on the digital twin were successfully applied to the physical hardware without issue, allowing testing to begin almost immediately after receiving the equipment.

Virtualization was again valuable between acceptance testing and production execution because the target router configurations significantly changed between the project phases. The configurations were proved in a virtual environment before being reapplied to the equipment prior to production deployment. This reduced the amount of work needed on-site and lowered the project execution risk.

In this project, 14 interconnected routers were successfully virtualized to create a digital twin of a core routing network. This physical configuration would have taken up five to six server racks of physical router equipment, but it was successfully implemented on a single server occupying less than one-tenth of a rack. It is worth noting that while the baseline hardware requirements for starting up virtualized routers and performing basic routing functions are low, the runtime requirements can quickly rise as topology becomes more complex and throughput needs increase.

Increasing efficiency and mitigating risks on a production upgrade are valuable applications for OT network operators, but the technology’s potential extends beyond that. Arbitrarily defined network topologies can be quickly created, tested and tweaked. A digital twin of the network can also be created, and production changes can be tested in a virtual environment, mitigating the risks of pushing a production change. Any topology can be implemented in virtual space if the underlying hardware can provide sufficient computing resources and memory.

The ongoing modernization of OT networks in the electric utility sector is leading to larger and more intricate infrastructures. Concurrently, ISPs are discouraging the use of T1 lines by raising lease costs or phasing out this transport medium as a service, contributing to the expansion of OT networks in the industry. The use of virtualization to optimize network planning for electric utility OT WANs is a critical tool to efficiently and effectively operate these evolving networks.

Virtualization, initially introduced with hypervisors, is a widely adopted technology that enhances the scalability and efficiency of server resources. As these server environments and their hosted applications expanded and evolved, they presented new network complexities. Software-defined networking (SDN) emerged as a solution to these challenges, paving the way for the development of virtualized network operating systems.

While these virtualized network operating systems have been used in various industries for over a decade, they were introduced in the electric utility OT WAN space relatively recently. As OT WANs in the electric utility sector progress, they will encounter comparable networking hurdles, necessitating a similar set of solutions for resolution as seen in other industries.

Virtualization presents an agile way to prototype different network topologies. It introduces increased flexibility, responsiveness and efficiencies in simulations and testing while reducing the risk presented to production networks caused by network changes. The ability to host multiple simulation and testing scenarios also introduces advanced training capabilities that will improve knowledge in OT production network operations and reduce the duration required for problem identification, understanding, resolution and confirmation on the OT production network.

Using virtualization to create an electric utility OT WAN digital twin allows network operators to realize the full implications of a change impact in a testing environment. These simulations and tests can be examined, verified and approved in the virtual environment, reducing risk to the production network. Leveraging insights from other industries and implementing them in electric utility OT WANs will be essential for the efficient and effective operation of these advanced networks.