Modern distributed control systems (DCS) have a variety of features to optimize the safety, performance, controllability and reliability of a generating station. Enhanced graphic display capabilities and alarm tools allow operators to respond to events quickly and access reference information through the control system. Detailed logic diagrams with ancillary information provide technicians with the ability to address issues and keep the unit running when problems arise. Automated patch management and user profile maintenance provides protection against security intrusions and reduces workload on information technology (IT) personnel. Built-in alarm reporting capabilities and remote access allows management to monitor the station remotely and plan for outage activities.
Many of these features weren’t available 15 years ago but are now an integral part of a modern DCS. Upgrading your DCS to use these enhanced features and mitigate the risks of an aging DCS is a decision that requires proper planning. The risks associated with your current DCS need to be evaluated in combination with the potential enhancements of a new system. The total cost of not only the control system procurement but construction, engineering and internal owner costs need to be accurately budgeted to perform the assessment.
Once a thorough control system assessment has been completed that evaluates the risks and analyzes the potential benefits, management can make an informed decision, identifying if an upgrade is justified.
INTRODUCTION
Many generating stations know they need to upgrade their control system, but it can be difficult to develop the supporting justification. Control system upgrades can develop the stigma of an unnecessary cost. This is mainly due to the difficulty of analyzing the risk associated with the existing system as well as proper valuation of the benefits of a new system. Only in extreme cases when the DCS is causing frequent plant trips or when replacement hardware is no longer available is it obvious that an immediate DCS upgrade is necessary. In most cases, an in-depth analysis is necessary to identify the underlying issues and determine the best time to upgrade.
A control system assessment is crucial to identifying the risks with the current system and the potential upgrade paths to all stakeholders. This paper will first review the risks that are frequently an issue with existing control systems and then explore some of the common components of a detailed DCS control system assessment.
JUSTIFICATION REASONING
Every generating station has its unique set of issues. Reviewing the system with the technicians and operators is crucial to identify the key issues they are facing. There are a few main issues that frequently appear in control system assessments, however, outdated hardware, software and control philosophies are three key issues that frequently appear in a control system assessment.
OUTDATED HARDWARE
One of the most common reasons for a control system upgrade is outdated DCS hardware. Computing technology has changed immensely over the last 15 years. Workstations and servers have become more powerful and efficient. DCS networks have adopted Ethernet-based topologies and communication links are more seamless. Controllers have more efficient processors and substantially more memory.
Outdated controllers, communication hardware and input/output (I/O) modules are an immediate danger that can result in lost generation. Most DCS controllers, old or new, are in a redundant configuration so that a single controller failure does not result in the failure of all of the controlled devices. With outdated controllers, there can be an issue replacing a single failed controller with the modern equivalent. Most instances require the replacement of both controllers at the same time, and in some instances it is necessary to upgrade the communication hardware as well. This can leave the facility in a difficult situation in which it has to run on a single controller until the next outage, substantially increasing risk.
The existing system configuration should be designed such that an I/O module failure will fail in a safe position that will not cause harm to personnel or equipment, but this failure could easily result in a plant trip, depending on the I/O module that fails. By updating hardware that is outdated, the risk of a module failure can be greatly reduced.
Original equipment manufacturer (OEM) supplied parts may not be available, depending on the age of the installed system. Once a station has to resort to purchasing used components on a secondary market to keep the DCS operational, the reliability of the station is put in jeopardy and an immediate control system upgrade should strongly be considered.
OUTDATED SOFTWARE
One of the most significant implications for using the enhanced control features of a modern DCS is the configuration software provided by the DCS vendor. There have been many enhancements in the last 15 years to improve the capability of DCS configuration tools. Configuring logic in a flow diagram layout is faster and more intuitive, and has become an industry standard. Logic macros have been refined to allow quick development and duplication. Graphic design has done away with stick-built graphics and developed enhanced macros that can incorporate the latest high-performance graphic philosophies. Many of these features were not available 15 years ago, and a control system upgrade is required to implement them.
Outdated software from the DCS vendor is not the only problem. Operating system software can also be a major issue. The Windows XP platform that was a requirement for some DCS packages is approaching end of life. Microsoft discontinued support for Windows XP as of April 8, 2014, so control systems that rely on this platform are exposed to increased security risks and availability issues. A control system upgrade brings the servers and workstations onto a modern operating system, reducing the cybersecurity threat.
OUTDATED CONTROL PHILOSOPHIES
The last key issue that frequently appears in a control system assessment is the opportunity cost of the existing system. This can be difficult to identify unless the buyer is aware of all of the latest features and trends in the power generation control system industry. The new features briefly described earlier have allowed for a shift in the way a station is controlled. Transmitters are replacing switches to bring greater operator awareness to the control room. Programmable logic controllers (PLCs) are being replaced with DCS controllers to allow for better coordinated control and operational integration. Startup and shutdown sequences are being optimized to allow the station to be frequently cycled, instead of the original baseload intent. Graphics and alarm management have become hot topics, with many stations converting to high-performance graphics and a highly structured alarm management program.
While a control system upgrade is not necessary to perform some of these changes, a DCS upgrade is the ideal time to incorporate new philosophies into the overall control of a generating station. The I/O list, logics, graphics and alarm list will have to be extracted from the old system and converted to the new system. It is the perfect time to redesign graphics, add logic enhancements, validate the alarm list and clean up the I/O list. Resources from the station will need to be involved in the upgrade project and have availability to provide input and review proposed modifications.
CONTROL SYSTEM ASSESSMENT COMPONENTS
The control system assessment can vary in its structure, but there are some common sections that are frequently included in the assessment. Risk assessments are usually included that identify and rank the risks the station is facing with the current system. A review of each control area should be performed, analyzing the existing conditions and the effects of a control system upgrade. A cost analysis is necessary to forecast the costs of implementing a DCS upgrade, and a project execution plan lays out the scope, assumptions and proposed method of execution for the project. It is useful to include supporting documentation in the final report, such as a project execution schedule, control system architecture, cost estimate details and cash flow forecasting.
QUALITATIVE RISK ASSESSMENT
The qualitative risk assessment provides a method for identifying risks and assigning an associated severity to the risk. The qualitative risk assessment is subjective and open to interpretation, but it is a useful exercise for identifying the risks and assigning a relative severity. To assign a numeric value to the severity, a review of the likelihood of occurring and the associated consequence needs to be performed. See Figure 1 for a sample table of likelihood vs. consequence used to assign each risk a numeric value.
Since control system upgrades need to be planned ahead to coincide with outages, it is important to include multiple years in the analysis. This allows for long-range planning and indicates the severity level to which the problem will grow worse. By comparing the before and after side-by-side (Figures 2 and 3), it becomes clear which risks will be mitigated and which will remain an issue.
QUANTITATIVE RISK ASSESSMENT
The quantitative risk assessment is a useful method for assigning a cost to an aging control system. Depending on the data available and the condition of the existing DCS, a quantitative risk assessment alone can justify a control system upgrade. A quantitative risk assessment uses existing root cause analysis (RCA) forms, generator availability data system (GADS) reporting and operations logs to identify system downtime that can be directly attributed to the DCS. By analyzing the cost of past occurrences, future occurrences can be estimated and it can be determined if a control system upgrade is cost effective.
IMPACTED WORK AREAS
Once the overall risks are identified, the various areas of the plant need to be reviewed to identify the impact of a control system upgrade. The existing DCS-related equipment in each area should be analyzed, then the potential upgrade options and how they would affect the area should be considered. For example, when evaluating the main DCS room, the existing I/O counts should be analyzed to determine if there will be issues fitting new DCS cabinets in the existing space. Based on the density of the existing system, additional cabinets may be needed to accommodate the required I/O in the room, or the DCS may have to have specific compact I/O requirements to accommodate the space. The control room should be analyzed to determine if a new console is necessary or additional lighting modifications should be considered to comply with the latest ergonomics initiatives. PLCs and other datalinks should be evaluated and a consensus should be developed on which PLCs should be incorporated into the DCS and which will remain as PLCs to communicate to the DCS as datalinks.
The challenge is to avoid getting pulled into detailed design of the system during this preliminary assessment phase and to just evaluate the main issues that could cause major schedule, scope or budget issues during execution of the project.
Although this paper focuses only on the DCS portion of a control system upgrade, a thorough review of the instrumentation is commonly included in a control system assessment. Upgrading instrumentation from switches to transmitters provides advanced control capability and operational awareness. Outdated devices can be upgraded to HART-smart devices to take advantage of enhanced calibration capabilities and secondary readings.
COST ANALYSIS
The cost analysis is evaluated by many individuals at all levels in an organization. Decision-makers immediately want to know how much an upgrade will cost. It is important to capture as much of the project costs as possible so that a project can come be budgeted correctly. The primary cost areas for execution of a control upgrade are engineering, procurement, construction, commissioning/startup and internal owner costs.
Engineering is an essential portion of the project and drives the direction of the project. Some generating stations have the capability to staff this internally, but many utilities rely on an engineering firm to manage this role. Engineering works closely with the project manager, technicians and operators to see that the project exceeds expectations and accomplishes the goals set forth in the control system assessment. They develop the majority of the key documents in the project, including the DCS procurement specification, logic enhancement diagrams, graphic markups, alarm management criteria, instrumentation data sheets, electrical schematics, cable schedule, I/O checkout plan and functional test plan. It is important that the engineering time be budgeted to match the level of involvement described in the project execution plan.
Procurement costs are developed by requesting budgetary estimates from potential DCS vendors. If only one vendor is being considered, the specification can be tailored for that vendor. Conversely, many generating facilities prefer or even require bids from multiple bidders during this assessment phase to keep options open and pricing competitive. A substantial amount of detail should be put into the specification of the request for proposal (RFP). The more detail that is provided in the RFP, the more accurate the DCS vendor’s budgetary estimate will be. If a +/- 10% estimate is requested, a detailed specification, control system architecture, I/O count by controller, cabinet details, field service and admin expectations are required so the vendor can adequately price the project.
Construction costs can vary depending on the state of the existing control system and the proposed modification detailed in the Impacted Work Areas section. Some projects require substantial field wiring changes, while other upgrades only involve work inside the cabinet. To develop an accurate construction cost, a full takeoff including material and labor should be done for each work area. Allowances should be added for each area to cover scope not identified in the high-level estimate. These allowances should be considered as part of the base price instead of contingency, since there will most likely be additional scope discovered during detailed design that is not included in the high-level preliminary assessment. Once the complete construction estimate is developed, it is best to review the estimate with an electrical contractor familiar with control system upgrades. Labor cost for cable installation, terminations and demolition work can vary greatly, so it is essential to get verification of the metrics used.
I/O checkout and commissioning is a substantial effort for a control system upgrade. Most generating stations require that the I/O be tested from the field device back to the control system, instead of at an intermediary junction box. The manpower should be accurately budgeted and take into account that work hours will most likely be extended beyond the standard work week.
The internal costs for the station can vary depending on its involvement in the project and its staff availability. Different organizations handle owner’s costs via different methods, but it is essential to assign an estimate to the internal costs for the generating station staff. Since staff is available at the plant, the perception can be to lump a large portion of this cost in with overhead, but in most cases this misrepresents the actual execution of the project and causes the project to go over budget. If someone at the plant level, be it operations, technicians or engineering, is working on the project providing supervision, direction or review, they are taking time away from their other duties at the plant. Time needs to be allotted for individuals to review documents and attend meetings during the design phase. The DCS factory acceptance test is a substantial time commitment that involves engineers, technicians and operators. During I/O checkout and startup, technicians need to be assigned to the project to avoid being pulled off to address the other miscellaneous plant items that require attention during an outage. By properly accounting for the internal cost to the utility during the upfront assessment, the project can execute smoothly with reduced risk of budget overrun.
PROJECT EXECUTION PLAN
The project execution plan is an important part of the control system assessment because it identifies the high-level scope for the project. It provides backing for the budgetary cost by defining the activities and assumptions made. Detail should be provided for each major activity.
Engineering is one of the areas that needs the most definition in the project execution plan. Detail must be provided so that all individuals involved are aware of the proposed engineering enhancements to the system. For example, alarming can refer to only copying over the existing DCS alarms into the new system, or it can mean developing a whole new alarm management philosophy that involves a substantial time commitment from operators, technicians and management. Other examples requiring varying levels of detail might include the extent of graphics optimization or logic configuration enhancements. By clearly defining the assumptions in the project execution plan, the basis for the budgetary cost and project schedule becomes clear.
Procurement starts with a detailed specification for the DCS. Depending on whether the DCS is going to be bid to multiple vendors or sole-sourced to one vendor, the DCS specification should be developed for the contracting approach. The DCS specification should at a minimum include project requirements, DCS performance requirements, control system architecture and I/O counts by cabinet. Additional information should be provided as needed to clearly define the project scope. A pre-bid meeting with a site walkdown is customary for the bidders to review the site details. Once the contract is awarded, a kickoff meeting should occur on-site to review the scope and schedule. Throughout the project, design review meetings and regular conference calls with the DCS vendor should occur to monitor project status. The DCS factory acceptance test will involve engineers, technicians and operators to check the configuration from all aspects.
A construction specification is developed later in the project once the electrical and mechanical scope has been clearly defined. During control system upgrades, the main focus is on keeping the outage duration as short as possible. Pre-outage construction work is essential to reducing the outage duration and should be done to the greatest extent possible. Any assumptions made in the cost analysis, such as the contractor being required to work day and night shifts, should be clearly stated in the project execution plan.
A well-designed I/O checkout team should involve an individual from each discipline of the project, including technicians, engineers and operators. Technicians know the plant the best and can easily locate the field instrument and perform the checks. The engineer has the most experience with the new electrical design and will work with the technician in the field testing the instruments and resolving issues. This is also the ideal time to give operators experience with the new graphics and functionality of the control system by verifying points at the DCS console for the field team. A DCS field service engineer should also be on-site to fix any issues with the network, graphics, logics or I/O database identified by the team.
Once I/O checkout is complete for a system, functional testing and tuning can commence. Each system should be tested to make sure the logic was converted as intended and any logic enhancements perform as expected. The tuning parameters can be carried over from the previous control system, but minor differences in control system execution most often require that a re-tune of the entire system be performed.
SUPPORTING DOCUMENTATION
It is beneficial to provide supporting information for those who want to dive into the details. A control system architecture is useful to provide a layout of the complete system. Details of the cost estimate should be provided that show breakout pricing for each phase of the project. A project execution schedule is crucial to lay out the project and show constraints such as upcoming planned outages. Finally, a cash flow analysis is useful for long-range forecasting.
CONCLUSION
The topics described above highlight frequent issues, but every control system is unique. It is important to listen to the concerns of all stakeholders to refine the key issues and address their impact. This uniqueness also carries over to the components of the control system assessment. The main components were discussed in this paper, but it is important to know the audience and modify the structure of the assessment as needed so that the control system assessment can become a useful tool for future planning. Depending on the situation, additional emphasis can be placed on the technical side of the assessment. Other times the assessment can be primarily used for long-range forecasting, so the cost analysis has the most detail.
The overall objective of a control system assessment is to provide a thorough review of the existing system and provide potential upgrade solutions. With this information, the analysis can be performed to weigh the risks against the potential benefits and determine the right time to upgrade the control system.
