Olugbenga Gideon, Thomas Ulrich, Roger Lew, Benjamin Barton, Zethnouneay Dubois (2024). Early-Stage Usability Testing of Thermal Power Dispatch Simulator Using Novice Operators. In: Human Factors and Simulation. AHFE Open Access, vol. 139, pp. 75–85. https://doi.org/10.54941/ahfe1005026
Flexible Plant Operations and Generation (FPOG) allow nuclear power plants (NPPs) to exploit alternative, non-electric revenue streams while ensuring their sustained role as dependable and environmentally friendly sources of baseload electrical power. The surplus thermal energy produced by NPPs during periods of low electricity demand can be directed to industrial processes through a thermal power dispatch (TPD) system such as high temperature steam electrolysis (HTSE) hydrogen production. Previous work at the Idaho National Laboratory (INL) involved developing and implementing a TPD system. An early-stage test using students as operator surrogates (n=12) was conducted using a modified GSE Generic Pressurized Water Reactor (GPWR) simulator on a desktop computer display. The study is the first to evaluate the usability of the dual-train TPD design and operating procedures toward identifying HSI issues common to students and expert users. Participants completed a startup and shutdown scenario with the control system in a manual and auto-ramp mode for a 2x2 factor design. The qualitative data analysis identified issues within three themes: information display, perceptual organization, and procedural confusion. The most notable issues were the use of small font sizes, the non-salient nature of essential dynamic features, and the ineffective groupings of interface elements. Using student participants to identify usability issues at this early stage of the dual-train TPD development is a proactive and cost-effective approach that will enable a full-scope study using expert operators.
Flexible plant operations and generation (FPOG) offer nuclear power plants (NPPs) the chance to leverage alternative, non-electric revenue streams while ensuring their continued role as reliable and clean sources of baseload electri- cal power. The excess thermal energy generated from NPPs during periods of low electricity demand can be channeled to industrial processes via a thermal power dispatch (TPD) system. Hydrogen production via high-temperature steam electrolysis (HTSE) is a lucrative and promising use case based on tech- nical and economic feasibility. Ongoing work performed at Idaho National
Laboratory (INL) previously developed and implemented TPD system mod- els within a GSE Solutions Generic Pressurized Water Reactor (GPWR) simulator to support human operator-in-the-loop (HOIL) scenario-based evaluations (Ulrich et al., 2021). Current work aims to evaluate a new dual- train TPD design that extracts steam after the high-pressure turbine and eliminates many control complications in lieu of higher quality steam. How- ever, the lower quality steam is suitable for HTSE applications. As a precursor to the full-scope HOIL dynamic TPD system integration study planned for mid-2024, an early-stage usability testing of the TPD human-system inter- face (HSI) and procedures was conducted using the TPD modified GPWR simulator on a desktop computer display.
The ability to rely on non-experts to stand in for nuclear operators to assess some aspects of the human-system integration (HSI) and operational procedures dramatically increases the flexibility of human factors analysis of critical HSI concerns. The U.S. Nuclear Regulatory Commission (NRC) adapted Lackey’s (2014) “equal but different” principle for using a novice population for nuclear plant main control room (MCM) research. The “equal but different” principle attempts to simplify complex tasks to enable novices to perform them and experience equivalent workload conditions as experts would. The approach has been used successfully with Rancor microworld studies (Ulrich et al., 2017). Even extensively trained students are less pro- ficient in their overall system knowledge and procedural expertise than experienced operators. However, previous work has shown that with suffi- cient training within the microworld, students can gain a working knowledge of the system and the corresponding procedures to operate the TPD simulator successfully for the purposes of usability evaluations.
This study is the first usability test conducted using the dual-train TPD design with auto-ramp control, compared to the previous single-train design with manual control. The dual-train design affords a smaller design option and provides system redundancy for potential malfunction in one train. Using student participants to identify usability issues in the early stage of the development is a proactive and cost-effective approach to enable the full-scope study requiring expert operators. The aim of this study was to eval- uate the usability of the dual-train TPD design and operating procedures to identify early HSI issues common to naive and expert users. Participants per- formed operator-like tasks using the adapted HSI and procedures. The task involves executing the 2 TPD scenarios in manual and auto-ramp control modes.
THERMAL POWER DISPATCH SIMULATOR HUMAN-SYSTEM INTERFACE The TPD HSI simulator mimics the actual TPD interface on a 32” 4K UHD monitor, providing operators with a dedicated instrument and control clus- ter to simplify monitoring and control tasks. The TPD HSI is a dual-train design, each consisting of two segments: the piping and instrumentation dia- gram (PID) display and the control panel for performing control actions (see Figure 1). The PID is divided into two parts: extraction steam loop (XSL) and
Early-Stage Usability Testing of Thermal Power Dispatch Simulator 77
delivery steam loop (DSL). The XSL extracts thermal energy from the nuclear power plant while the DSL delivers steam to the HTSE with heat exchangers that act as barrier between the secondary plant and external delivery piping and hydrogen plant. The layout of the PID has a horizontal alarm annun- ciator bank at the top of the window, vertical train status indicators on the flanking external edges, and a single-operator alarm control group situated in the center of the screen. The control panel contains controllers corresponding to the XSL and DSL.
Operating Scenarios and Control Modes The TPD has two operating scenarios and two control modes. The operating scenarios include Hot Standby to Online (HSB -> ONL) and Online to Hot Standby (ONL -> HSB). Simply put, Hot Standby to Online refers to increas- ing extraction steam from its Hot Standby rate of 2.5 lb/s to its online rate of 50 lb/s, while Online to Hot Standby refers to returning extraction steam flow to its standby rate of 2.5 lb/s. The TPD is placed Hot Standby when steam is not required in the HTSE to curtail the dispatch of electrical and thermal energy.
Figure 1: TPD human-system interface (HSI).
The scenarios may be operated in manual or auto-ramp control modes. In manual control, the operator executes a series of control action steps and ver- ifies system behavior in line with the corresponding procedure’s documented setpoint values. In the auto-ramp control, the system automatically executes the action steps, and the operator performs verification as oversight in line with the U.S. NRC’s NPP automation Level 5 (O’Hara & Higgins, 2020). The combination of operating scenarios and control modes resulted in four operating conditions (see Table 1).
Adapted Operating Procedures The TPD operating procedures were adapted from the general GPWR sim- ulation procedures. The TPD-adjusted procedure consists of four sections: prerequisites, precautions and limitations, initial conditions, and procedure. The current study used student participants, hence simplification was favored over matching the explicit format of the GPWR procedures to keep format and language as simple as possible. The content of the entire section on pre- caution and limitation was excluded, given that students participants were not expected or needed to understand the operating limits and boundaries communicated therein. In its place, a general statement that participants had to read aloud, stating, “All precautions and limitations have been read and well understood.”
METHODS
The usability study was conducted at a north-western university in the U.S. The study is a 2x2 design with scenario and control modes as the independent variables (see Table 2). Basic qualitative psychological usability measures were used as dependent variables.
Participants Participants were recruited from the student population at a north-western university in the U.S. Participants received nominal compensation for full participation. The protocol was approved by the Institutional Review Board prior to beginning.
Trials The trials consisted of the 2 TPD operating scenarios in manual and auto- ramp control modes. The operating scenarios include Hot Standby to Online (HSB -> ONL) and Online to Hot Standby (ONL -> HSB). The task was split into two sessions (on two separate days) of no more than one hour each over one week. The maximum time limit between sessions was capped to ensure participants retained the knowledge gained from the first session on how to control the TPD system. In each session, participants completed a manual and auto-ramp control scenario using trains A and B. The order of presentation of operating scenarios, control modes, and trains were counterbalanced to yield 4 study conditions (see Table 2).
Early-Stage Usability Testing of Thermal Power Dispatch Simulator 79
Table 2. Study conditions.
Table available in the original PDF paper.
Note. Circled numbers represent the order in which trials were completed in each session Participants were randomly assigned to conditions while forcing equal sample sizes. Participants maintained a think-aloud protocol through the entire period of task execution. The think-aloud protocol involved partici- pants reading each step of the procedure out loud. Also, they verbalized their understanding of the step and what aspect of the HSI they were looking at, including any thoughts or challenges they were experiencing while the exper- imenter took notes. The study was implemented on a single-screen 32” 4K UHD monitor desktop computer display lacking some of the physical realism of a full-scope simulator. Of course, the physical realism of a full-scope sim- ulator is required to understand complicated process integrations of the TPD with the nuclear plant. However, the emphasis of the current usability studies is to identify HSI issues. Therefore, given the underlying model in the desktop computer is the same as in the full-scope simulator, limiting participants’ view to the TPD HSI only, presents no risk of missing any information necessary for operating the TPD due to the lack of physical realism. The only concern may be the difference between students and expert operators, which was miti- gated by adapting the tasks using the “equal but different”principle for using a novice population for nuclear plant main control room (MCM) research so that students can perform them and experience equivalent workload con- ditions as experts would.The TPD system HSI was connected through an application programming interface to the GPWR simulator to populate each elements dynamic value throughout the simulator. To familiarize participants with the layout of the HSI and operating procedures, participants watched a recorded video showing an operator walkthrough one of the TPD scenario.
Measures Debrief sessions followed the completion of a scenario set. The debriefs were conducted to elicit participants’ usability measures along 4 dimensions: visual elements, situation awareness, conceptual understanding, and proce- dural understanding. A questionnaire was used to administer the debriefs and captured participants’ corresponding usability measures using the Microsoft Word speech-to-text feature.
The visual elements measures assessed participants’ bare minimum requirements to make sense of the display. Four usability measures assessed visual elements: labels, organization, graphic displays, and consistency.
Labels measure the ease of identifying labels and reading digital values on the PID and control panel. There are three levels of situation awareness: percep- tion of elements in the environment, comprehension of the current situation, and projection of future status (Endsley, 1995). The first level of situation awareness was the focus of this study. Emphasis was on whether (or not) participants noticed changes in the PID and what changes they did notice. Conceptual understanding involves gaining basic insight into the fundamen- tal ideas, principles, and relationships that underlie the TPD HSI. This form of understanding goes beyond mere memorizing procedure steps and instead focuses on grasping the deeper relationships and connections among the dif- ferent elements of the PID. Procedural understanding focused on measuring the extent to which participants understand the procedures for executing corresponding operating scenarios.
Procedure Participants completed informed consent at the beginning of the first session. Participants then watched the video demonstration showing a walkthrough of one TPD scenario in manual control. The experimenter then guided the participants through a walkthrough while performing a think-aloud proto- col and cued them to bring up relevant issues. Participants were randomly assigned to conditions. Participants were asked to complete tasks in the assigned condition’s first session in two modes - manual and auto-ramp con- trol modes - with the order counterbalanced. Participants provided an overall session debrief via a questionnaire after completing both scenarios. Partici- pants returned for the second session on a mutually agreed date and com- pleted the second set of scenarios and its corresponding debrief (see Table 3).
The data were analyzed to identify usability issues. The findings and issues from this section describe insights gathered from participants’ responses to the self-report questionnaires, think-aloud protocol, and observations cap- tured by the experimenter during the study. The findings and issues were grouped into three themes: information display, perceptual organization, and procedural confusion. The themes are further segregated to reflect the corre- sponding trial order (i.e., trial 1 or 2). A simple nominal count of the number of participants who reported an issue was done (see Figure 2).
Early-Stage Usability Testing of Thermal Power Dispatch Simulator 81
Figure 2: Usability issues.
Information Display The information display category addresses the effectiveness of presented information. The operator’s information requirements dictate how such information is represented. Important information representation considera- tions must be factored into a system’s display to support task performance. Participants raised two major issues related to information. The issues include small font sizes and non-salient feedback on dynamic display/control feature status.
Small font sizes. Participants reported a challenge viewing the values and text on the PID and the control panel. A total of 11 and 10 out of 12 partici- pants raised issues related to small font sizes, especially on the control panel in the first and second trials, respectively. The text fields on the control panel were commonly reported to contain small font sizes (see Figure 3). Partic- ipants were also observed squinting and moving closer to the screen when they entered values into the fields of the control panel.
Figure 3: Control panel depicting the reported issue of small font sizes.
Non-salient feedback of dynamic display/control feature status. Partici- pants complained about non-salient features to indicate the status of dynamic elements, such as control buttons when pressed (see Figure 4). Participants need to receive salient feedback when a button is clicked (as in clicking the “Auto-Ramp” button) and when the status of a dynamic display changes. The non-salience of features led to confusion about the current operational modes and status of buttons on the interface.
The significant negative operational effect reported by participants and gathered from observation due to non-salient feedback is a recurring mode error in auto-ramp control. To operate the TPD system in auto-ramp, par- ticipants click the “Auto-Ramp” button to ramp up or ramp down steam flow and verify all subsequent steps occur automatically. Due to non-salient feedback indicating the current control mode of the system, some partici- pants forgot the system was in auto-ramp and began enacting action steps in the operating procedures instead of merely verifying the actions occurred automatically.
Figure 4: Control panel depicting reported non-salient status indicator after a “manual” button click.
Perceptual Organization Perceptual organization involves the spatial arrangement of the display com- ponents to enable comprehension and task performance based on gestalt grouping principles. In the current study, participants reported 3 issues relat- ing to perceptual organization, namely ineffective grouping, distal element attributes, and clutter.
Ineffective grouping. Grouping is the process by which elements in an interface are brought together into a common unit. The TPD interface has elements of the PID mapped to their corresponding controls on the control panel. However, no form of grouping shows what controls are mapped to elements of the PID. Participants reported difficulty in identifying what com- ponents of the PID were being controlled by the manipulations on the control panel as they shifted attention back and forth between the control panel and the PID. For example, there is no form of grouping that shows the relation- ship between the extraction flow controller and its corresponding elements on the PID (see Figure 5). A simple box drawn around PID elements and their associated controls might minimize this difficulty.
Early-Stage Usability Testing of Thermal Power Dispatch Simulator 83
Figure 5: TPD HSI depicting reported ineffective grouping between extraction the flow controller and its corresponding elements on the PID.
Distal element attributes. Information integration becomes ineffective when display elements are spatially located far away from their associated attributes. For example, instruments on the PID have trend lines, names of trend lines, and instrument readings, all as attributes. Participants pointed out the distal and inconsistent location of these essential attributes with respect to different instruments (see Figure 6), resulting in confusion when referring to those attributes. Adopting a correspondence approach where controls are located near associated indicators, and related indicators clustered together will minimize such challenges.
Figure 6: PID depicting reported distal and inconsistent location of essential attributes with respect to corresponding instruments.
Clutter. Clutters impede visual attention, which hinders selective attention, especially in a search field. Executing tasks in TPD involves searching for various elements on the interface, which can be easily hampered by clutter. Participants reported clutter arising from the four face plates (drain receiver level controller, kettle reboiler level controller, steam pressure controller, and drain tank level controller) on the control panel that were never used. Accord- ing to the participants, these faceplates acted as distractions to completing their tasks on the “extraction flow controller” - the only face plate used across all scenarios. A toggle button to hide faceplates not currently in use by participants may help reduce visual clutter.
Figure 7: Control panel depicting reported faceplates irrelevant to the task completion.
Procedural Confusion Some participants experienced phonological similarity confusion due to the similarity in the naming convention used in the components of the TPD as they verbalized procedure steps involving FCV/LCV, XSL/DSL, and LT/FT. It is unknown if such a challenge will apply to expert operators who are very conversant with a three-way communication protocol. Nevertheless, having the operators verbalize component names in full instead of abbreviated forms may differentiate one component from another, minimizing verbal confusion.
CONCLUSION
The current work is the first usability test conducted using a dual-train TPD design with auto-ramp control, compared to the previous single-train design with manual control. The dual-train design affords a smaller design option and provides system redundancy for potential malfunction in one train. The aim was to identify early HSI and procedure-related concerns common to novice and expert users of the dual-train design. Several issues were identified and categorized into three themes: information display, perceptual organiza- tion, and procedural confusion. The issues reported by participants suggest the need for improvement in the subsequent design iterations. The first is using larger font sizes on the PID and control panels, the second is making the status of essential dynamic features more salient, and the third is adopting
Early-Stage Usability Testing of Thermal Power Dispatch Simulator 85
a consistent and effective spatial arrangement of display components using Gestalt grouping principles. Lastly, there may be a need to minimize clut- ter and confusion arising from similarity in naming conventions. The main takeaway is that using student participants to identify usability issues at this early stage of the dual-train TPD development is both a proactive and cost- effective approach to enable full-scope study requiring expert operators. The usability issues identified can be addressed early in the development lifecycle, allowing the full-scope study to focus on the expertise required to operate the system rather than HSI confusion. Given that the current study was con- ducted using students, it may be necessary to refine the findings within the context of experienced operators before implementing improvement in the full-scope simulator.
REFERENCES
Endsley, M. R. (1995). Towards a theory of situation awareness in dynamic systems.
Human Factors, 37, 32–64. Lackey, S., Reinerman-Jones, L. E., Salcedo, J. (2014). Equal but different: 5 research strategies for improving conclusions drawn from novice populations. Paper presented at the MODSIM World Conference, Hampton, VA. O’Hara, J. & Higgins, J. (2020). Adaptive Automation: Current Status and Chal-
lenges. Washington, D. C.: U. S. Nuclear Regulatory Commission. Ulrich, T. A., Lew, R., Werner, S., & Boring, R. L. (2017, September). Rancor: a gamified microworld nuclear power plant simulation for engineering psychology research and process control applications. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 61, No. 1, pp. 398–402). Sage CA: Los Angeles, CA: SAGE Publications. Ulrich, T. A., Lew, R., Hancock, S. G., Westover, T. L., Medema, H. D., Boring PhD,
R. L.,... & Minard, N. (2021). Dynamic Humanin- the-Loop Simulated Nuclear Power Plant Thermal Dispatch System Demonstration and Evaluation Study (No. INL/EXT-21- 64329-Rev000). Idaho National Lab. (INL), Idaho Falls, ID (United States).
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Human Factors and Simulation
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-964867-15-1
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional