Intelligent Process Visualization through Nuclear Operation Process Modeling, Reasoning, and Object Detection from Field Videos (Final Report)

G. Edward Gibson, Pingbo Tang, Alper Yılmaz, Ronald L. Boring, Thomas T. Myers, Jinding Xing, Pengkun Liu, Shehan Perera, Boresh Pouyan2022 · DOI

Abstract

This report details the development and evaluation of the Intelligent Context-Aware Safety Information Display (ICAD), an Augmented Reality (AR) system designed to enhance the safety and efficiency of Nuclear Power Plant (NPP) field operations. By integrating Natural Language Processing (NLP), computer vision, and sensor-based reasoning, the ICAD system bridges the "attention gap" between physical workspaces and complex operational procedures. The project demonstrates the technical feasibility of proactive visualization, achieving high accuracy in object detection (95.3%) and procedure extraction (>80%), while providing resilient indoor navigation for workers in complex industrial environments.

Introduction

Nuclear Power Plant (NPP) field operations are characterized by high-stakes tasks performed in complex, often uncertain environments. Currently, field workers rely heavily on Paper-Based Procedures (PBPs), requiring them to constantly switch focus between physical equipment, digital sensor data, and printed manuals. This cognitive load is a significant contributor to human error, which accounts for over 60% of reported incidents in the nuclear industry. Common errors include the misidentification of components, sequencing mistakes, and failure to recognize environmental hazards.

To address these challenges, the DOE NEET Project 19-16790 developed the ICAD-AR system. The goal is to provide "proactive visualization"—delivering the right safety and operational information at the right time directly within the worker's field of view. This system transforms static procedures into dynamic, context-aware guidance, reducing the need for constant communication with supervisors and minimizing the risk of operational errors.

The ICAD Framework

The ICAD system is built upon a five-pillar technical framework designed to model, reason, and visualize NPP processes in real-time:

1. Natural Language Processing (NLP)

To convert unstructured paper procedures into machine-readable formats, the team utilized a BERT-CRF (Bidirectional Encoder Representations from Transformer with Conditional Random Fields) model. This model extracts critical entities—such as actions, components, and process variables—to build computer-interpretable knowledge graphs that drive the AR overlays.

2. Sensor Log Analysis and Workflow Prognostics

Context-awareness requires understanding the plant's state. The system employs k-Nearest Neighbor (KNN) classifiers, augmented by SMOTE (Synthetic Minority Oversampling Technique), to analyze sensor logs. This allows the system to predict the next required control action and inform workers of necessary "waiting times" or safety windows based on real-time plant conditions.

3. Worker Localization and Navigation

In GPS-denied indoor environments like NPP mechanical rooms, tracking worker movement is difficult. The system uses Visual Inertial Odometry (VIO) for localization. To combat "drift error"—the cumulative deviation in path estimation—the team integrated a Non-Recursive Bayesian Filter (NRBF), ensuring that navigation guidance remains accurate over time.

4. Object Detection and Safety Checking

The system utilizes the YOLOv5 algorithm to identify task-specific and safety-critical objects (e.g., valves, tanks, sensors). This ensures that workers are interacting with the correct equipment and are alerted to nearby hazards, such as radiation zones or high-pressure areas.

5. AR Integration

The AR platform serves as the synthesis layer, overlaying the extracted knowledge, navigation paths, and object labels onto the physical workspace through AR glasses.

Experimental Setup and Datasets

Due to the sensitive nature of NPP data, the project utilized several high-fidelity proxies for training and validation:

  • Rancor Reactor Simulator: Used for collecting control history logs and training action prediction models.
  • Duke Energy Flow Loop: A physical training facility used for mapping Piping and Instrumentation Diagrams (P&ID) to physical objects.
  • Mechanical Rooms (OSU and CMU): Provided 3D point cloud data and video for simulating complex industrial navigation and visual aliasing challenges.
  • Water Treatment Plant: Served as a testbed for the generalizability of object detection models across different industrial contexts.

Key Results

The ICAD-AR prototype demonstrated strong technical performance across all core modules:

  • Procedure Extraction: The NLP model successfully extracted over 80% of critical information from standard PBPs.
  • Action Prediction: The sensor log analysis model achieved an accuracy of 0.883 in predicting the next control action.
  • Object Detection: YOLOv5 achieved an average accuracy of 95.3% at processing speeds exceeding 30 frames per second, enabling real-time interaction.
  • Navigation Resilience: The NRBF integration significantly reduced trajectory estimation errors, making the system resilient to the visual aliasing common in environments with repetitive piping and valve structures.

Discussion and Future Outlook

While the ICAD system proves the feasibility of context-aware AR in nuclear environments, several challenges remain. "Visual aliasing"—the difficulty of distinguishing between identical-looking components—remains a hurdle for computer vision in dense mechanical rooms. Furthermore, while the technical accuracy is high, future work must focus on human factors and User Experience (UX) testing. It is critical to ensure that the AR interface provides clarity without introducing new distractions or increasing the operator's cognitive load.

Ultimately, the integration of process reasoning with computer vision represents a significant step toward the "digital twin" of the field worker, where the system understands not just where the worker is, but what they are doing and what they need to stay safe.

Summary updated

Abstract

This report is a deliverable for the “Final Report” task of DOE NEET Project 19-16790, "Context-Aware Safety Information Display for Nuclear Field Workers." This project's overall goal is to test the hypothesis that integrating computer vision and process reasoning methods will enable proactive visualization of the safe operation and maintenance processes of Nuclear Power Plants (NPP) for field workers. Augmented Reality (AR) glasses adopting such proactive safety information visualization techniques can significantly increase personnel safety and reduce the NPP’s operating costs. The current practice of monitoring NPPs requires workers to switch between digital models, data, and physical workspaces in identifying relevant but potentially occluded objects and in assessing the risks of operation and maintenance processes. On the other hand, frequently changed field conditions require field workers to report to supervisors for real-time guidance. Such guidance is essential to ensure that changing conditions will not invalidate or endanger the work order and other ongoing processes that may jeopardize NPP operations. Additionally, incorrect recognition of equipment objects can result in communication errors and safety problems. AR techniques can assist engineers in viewing the physical workspaces with objects labeled with detailed operation procedures and safety reminders during field operations. The project team developed an “Intelligent Context-Aware Safety Information Display” (ICAD) for supporting Nuclear Power Plant (NPP) field workers in achieving safe and efficient execution of a series of operational tasks in uncertain and changing workspaces of an NPP. Before designing the ICAD-AR prototype, the project team synthesized NPP operational knowledge models through literature review studies, surveys, interviews with domain experts, and knowledge modeling. The project team conducted an extensive study of the operational procedures of various NPPs, and digital technologies that can support the safe and efficient execution of those procedures in different NPP operational contexts. This literature review helped the project team conduct surveys and interviews with nuclear engineers and field workers to identify three categories of information. The NPP knowledge modeling efforts reveal that the three categories of information identified have different levels of importance in a typical procedure of carrying out a series of tasks to achieve a specific NPP operation goal (e.g., shutdown, mode changes). These three categories of information include 1) Workspace dynamics – the changing spatial arrangements of workspaces, tools, protection equipment, and supporting materials, 2) Workflow prognostics – the dynamic dependencies between different parts of an NPP that functionally support and influence each other in terms of safety and efficiency, and 3) Hazards – objects and spaces that contain hazardous materials or physical conditions that can pose risks to workers or mechanical systems. The project team has profiled the importance levels of these categories of information into a knowledge model. This knowledge model specifies what types of information are more critical for a given task in a given workspace so that computers can automatically identify critical objects and sensors in a scene for delivering context-ware safety information to field workers through AR devices. Significant research development of this project results in technical research outcomes and a prototyping system that illustrates the technical feasibility of establishing an ICAD-AR system supporting the proactive safety information display for nuclear field workers. This final report summarizes the project team’s technological achievements in the past three years. Overall, the project team completed the development and integration of five techniques into a prototype ICAD Augmented Reality (ICAD-AR) system and demonstrated the developed system’s real-time execution in a mechanical room. The project team completed the analysis of using this prototype in other types of workspaces based on 3D image data and digital design models collected from two additional workspaces (a water treatment plant and a flow loop training facility). The integrated techniques include 1) Natural Language Processing (NLP) algorithms supporting the generation and updates of nuclear fieldwork process models based on text analysis of work packages and operation manuals; 2) sensor log analysis for predicting control actions in given sensor reading contexts; 3) computer vision algorithms for automatic localization and navigation of workers; 4) object detection algorithms for identifying task-related objects and correlated sensors for safety checking; 5) AR technique as a platform for supporting the integration. The testing results of these five techniques have shown that 1) the sensor log analysis model can predict the next control action with an accuracy of 0.883; 2) the trained natural language processing model can extract more than 80% of the critical information from paper-based procedures (PBPs); 3) the navigation algorithm with the integration of Visual Inertial Odometry (VIO) and Non-Recursive Bayesian Filter methods make operator’s trajectory estimation resilient to drift error; 4) the computer vision algorithm can detect task-specific and safety-critical objects with an average accuracy of 95.3%. The project team used work procedures collected from a flow loop training facility and two datasets collected from two mechanical rooms simulating the workspaces of NPPs to demonstrate the technical capabilities of the developed ICAD-AR prototype. The demonstration validated the technical feasibility of establishing the ICAD-AR system for nuclear field workers and identified the challenges in 1) automatic text analysis of work packages; 2) use of limited samples of sensor logs for predicting the proper timings of control actions; 3) reliably tracking workers and their task progress in mechanical rooms with many similar objects.
BibTeX
@misc{gibson2022intelligent,
  author = {G. Edward Gibson and Pingbo Tang and Alper Yılmaz and Ronald L. Boring and Thomas T. Myers and Jinding Xing and Pengkun Liu and Shehan Perera and Boresh Pouyan},
  title = {{Intelligent Process Visualization through Nuclear Operation Process Modeling, Reasoning, and Object Detection from Field Videos (Final Report)}},
  year = {2022},
  doi = {10.2172/1906971},
}
Figure