Abstract
This report details the development and evaluation of the Intelligent Context-Aware Safety Information Display (ICAD), an Augmented Reality (AR) system designed to enhance the safety and efficiency of Nuclear Power Plant (NPP) field operations. By integrating Natural Language Processing (NLP), computer vision, and sensor-based reasoning, the ICAD system bridges the "attention gap" between physical workspaces and complex operational procedures. The project demonstrates the technical feasibility of proactive visualization, achieving high accuracy in object detection (95.3%) and procedure extraction (>80%), while providing resilient indoor navigation for workers in complex industrial environments.
Introduction
Nuclear Power Plant (NPP) field operations are characterized by high-stakes tasks performed in complex, often uncertain environments. Currently, field workers rely heavily on Paper-Based Procedures (PBPs), requiring them to constantly switch focus between physical equipment, digital sensor data, and printed manuals. This cognitive load is a significant contributor to human error, which accounts for over 60% of reported incidents in the nuclear industry. Common errors include the misidentification of components, sequencing mistakes, and failure to recognize environmental hazards.
To address these challenges, the DOE NEET Project 19-16790 developed the ICAD-AR system. The goal is to provide "proactive visualization"—delivering the right safety and operational information at the right time directly within the worker's field of view. This system transforms static procedures into dynamic, context-aware guidance, reducing the need for constant communication with supervisors and minimizing the risk of operational errors.
The ICAD Framework
The ICAD system is built upon a five-pillar technical framework designed to model, reason, and visualize NPP processes in real-time:
1. Natural Language Processing (NLP)
To convert unstructured paper procedures into machine-readable formats, the team utilized a BERT-CRF (Bidirectional Encoder Representations from Transformer with Conditional Random Fields) model. This model extracts critical entities—such as actions, components, and process variables—to build computer-interpretable knowledge graphs that drive the AR overlays.
2. Sensor Log Analysis and Workflow Prognostics
Context-awareness requires understanding the plant's state. The system employs k-Nearest Neighbor (KNN) classifiers, augmented by SMOTE (Synthetic Minority Oversampling Technique), to analyze sensor logs. This allows the system to predict the next required control action and inform workers of necessary "waiting times" or safety windows based on real-time plant conditions.
3. Worker Localization and Navigation
In GPS-denied indoor environments like NPP mechanical rooms, tracking worker movement is difficult. The system uses Visual Inertial Odometry (VIO) for localization. To combat "drift error"—the cumulative deviation in path estimation—the team integrated a Non-Recursive Bayesian Filter (NRBF), ensuring that navigation guidance remains accurate over time.
4. Object Detection and Safety Checking
The system utilizes the YOLOv5 algorithm to identify task-specific and safety-critical objects (e.g., valves, tanks, sensors). This ensures that workers are interacting with the correct equipment and are alerted to nearby hazards, such as radiation zones or high-pressure areas.
5. AR Integration
The AR platform serves as the synthesis layer, overlaying the extracted knowledge, navigation paths, and object labels onto the physical workspace through AR glasses.
Experimental Setup and Datasets
Due to the sensitive nature of NPP data, the project utilized several high-fidelity proxies for training and validation:
- Rancor Reactor Simulator: Used for collecting control history logs and training action prediction models.
- Duke Energy Flow Loop: A physical training facility used for mapping Piping and Instrumentation Diagrams (P&ID) to physical objects.
- Mechanical Rooms (OSU and CMU): Provided 3D point cloud data and video for simulating complex industrial navigation and visual aliasing challenges.
- Water Treatment Plant: Served as a testbed for the generalizability of object detection models across different industrial contexts.
Key Results
The ICAD-AR prototype demonstrated strong technical performance across all core modules:
- Procedure Extraction: The NLP model successfully extracted over 80% of critical information from standard PBPs.
- Action Prediction: The sensor log analysis model achieved an accuracy of 0.883 in predicting the next control action.
- Object Detection: YOLOv5 achieved an average accuracy of 95.3% at processing speeds exceeding 30 frames per second, enabling real-time interaction.
- Navigation Resilience: The NRBF integration significantly reduced trajectory estimation errors, making the system resilient to the visual aliasing common in environments with repetitive piping and valve structures.
Discussion and Future Outlook
While the ICAD system proves the feasibility of context-aware AR in nuclear environments, several challenges remain. "Visual aliasing"—the difficulty of distinguishing between identical-looking components—remains a hurdle for computer vision in dense mechanical rooms. Furthermore, while the technical accuracy is high, future work must focus on human factors and User Experience (UX) testing. It is critical to ensure that the AR interface provides clarity without introducing new distractions or increasing the operator's cognitive load.
Ultimately, the integration of process reasoning with computer vision represents a significant step toward the "digital twin" of the field worker, where the system understands not just where the worker is, but what they are doing and what they need to stay safe.
Summary updated