Building Out a Private Network Across Hundreds of Sites

Challenge

Ameren faced a difficult situation in 2015 when local public communication carriers announced they would no longer support the legacy leased copper services that Ameren has relied on to support the operation of its electrical transmission and distribution businesses across its service territory.

Much of Ameren’s mission-critical substation equipment uses legacy interfaces that worked with those leased copper services but would not interoperate with the new service offerings of the public carriers, forcing Ameren to confront a critical decision. Ameren could upgrade many of the devices that support its critical substation equipment and begin leasing new service offerings from the public carriers, continuing to pay monthly fees to the carriers and operating under the limitations of service-level agreements. Or it could build a private telecommunications transport system that would be owned and operated by Ameren and capable of delivering the legacy services while providing long-term flexibility.

The utility decided to build a private fiber-optic network to provide infrastructure to previously unreached sites and deploy a private transport network using multiprotocol label switching (MPLS) technology. The infrastructure investment, combined with the new technology, would give Ameren the flexibility to connect to its existing equipment without requiring expensive equipment upgrades while also providing a pathway for expanded service offerings that would require more capacity and faster speeds.

Designing, procuring and implementing the private network solution that would reach approximately 650 sites across two states would be a tall order. Ameren has a strong team of telecommunications engineers capable of designing and deploying such a network, but the size and breadth of this endeavor required the utility to seek a partner with technical experience in deploying MPLS for electric utilities for this programmatic deployment. Ameren selected Burns & McDonnell on the strength of the company’s practical experience with telecom technologies and long history of working with and understanding utilities and their communication needs and requirements.

Solution

Working alongside Ameren, we led the detailed design and development of construction documents for the MPLS network engineering and system integration. With our support, the project began in 2015 and developed into a multiphase approach deployed over the course of eight years, focusing first on core substation sites in locations that were critical to building out the rest of the network.

While telecom technology vendors often offer engineering support, it tends to be based more on familiarity with residential and commercial installations, not mission-critical facilities like substations. In contrast, we could draw on extensive engineering and construction experience at utility facilities to better inform planning and designs for the network installation. Our specialized knowledge meant we understood strict utility requirements for redundancy, as well as how an installation could impact other highly sensitive and critical equipment.

The project’s success was attributable in part to realizing numerous efficiencies.

We were able to collect extensive site data in a single visit and input it into a geographic information system (GIS) platform, saving time and reducing return trips while building a central repository for detailed information beneficial to both this program and future projects. Putting all the information into that single platform also helped close any potential gaps and see that the design was fully contiguous even as it crossed the fence line of each individual site. It also enabled the grouping of sites to create yearly portfolios that balanced spending in each region, level of effort and available budgets. This approach enabled Ameren to expedite the scoping of 150 site designs by three months.

With numerous internal stakeholders to satisfy, the GIS mapping helped obtain and expedite buy-in and approvals. Using collaborative tools in a multiuser environment enabled us to consolidate and deliver a lot of information: site data, strategic design elements and asset attributes. Having everything in the GIS database allowed all stakeholders to provide comments, check statuses, resolve conflicts and answer questions without waiting for manual distribution of documents and subsequent input from others. This facilitated faster resolution of concerns or questions related to designs.

We delivered additional value and quality through other forward-thinking solutions:

• Using our in-house Integration & Automation Lab, we were able to stage equipment and allow Ameren to experiment with different technologies, testing and validating interoperability solutions prior to procuring and deploying to the project sites. This saved weeks of upfront troubleshooting during deployment.
• We used a central facility to configure and test racks of equipment before they were shipped to the project sites. We connected the racks and configured them in close quarters, executing factory acceptance testing on hundreds of routers and firewalls and thousands of network switches. By resolving network-specific issues in the centralized facility testing, once the devices were shipped to the field, any troubleshooting could be isolated to the physical fiber network, thereby saving iterative problem-solving steps that often consume resources and extensive time during deployment.

Results

After eight years, the Intelligrid network is supporting mission-critical utility applications at substations, microwave sites, gas storage fields, power plants, operating centers and office buildings across most of Illinois and Missouri.

The decision to build a private MPLS network helped simplify a potentially complicated revamp of Ameren’s communications systems. The project team was able to use technology that could accommodate existing equipment, and the decision to use a private network gives Ameren control over the construction, maintenance and repair of its communications systems, rather than depending on common carriers’ prioritization.

The team successfully executed the design, testing, construction, installation and commissioning of the new fiber network without a single recordable safety incident in over a million hours of labor.

Our integrated approach in performing engineering, system integration and construction management within a single team enabled Ameren to benefit from smoother handoffs, improved communication and reduced installation issues, leading to an early completion.

The utility’s customers will continue to benefit from the GIS database, which Ameren is now using to map out locations for distributed energy resource interconnect requests, helping plan where Intelligrid will need to extend in the future.