Automatic Imagery Data Analysis for Proactive Computer-Based Workflow Management during Nuclear Power Plant Outages

Pingo Tang, Alper Yılmaz, Nancy J. Cooke2018 · DOI

Abstract

This report summarizes the findings of DOE NEUP Project 15-8121, which investigated the use of real-time imagery-based object tracking and human behavior modeling to improve the efficiency of nuclear power plant (NPP) outages. Typical NPP outages involve thousands of maintenance activities and a large workforce operating under tight schedules with zero-tolerance for accidents. Delays in these environments are extremely costly, averaging approximately $1.5 million in losses per day. This research established methods for real-time object tracking, human factor modeling, and spatiotemporal analysis to detect productivity anomalies and improve coordination between workers and control centers. By integrating computer vision with agent-based simulations, the project demonstrated that automated imagery-driven methods can proactively enhance workflow efficiency and safety in complex maintenance environments.

The Challenge of NPP Outages

Nuclear power plant outages are high-stakes logistical marathons occurring every 18 to 24 months. These periods are dedicated to refueling and critical repairs, involving a massive temporary workforce and thousands of simultaneous activities. Because these environments operate on razor-thin schedules, any delay is catastrophic for both the utility budget and the stability of the power grid.

Currently, outage management relies heavily on manual inspections and verbal reports. These traditional methods are inherently slow and prone to "human-related issues," such as miscommunication and a lack of real-time situational awareness. There is a critical need for systems that can monitor human behavior in real-time, predict how minor delays might cascade through a complex schedule, and provide proactive alerts to managers before small issues become major setbacks.

Methodology: The Virtual-Physical Loop

The researchers addressed these challenges through an integrated framework that connects digital modeling with real-world physical behaviors. This "Virtual-Physical" loop allows for continuous monitoring and proactive adjustment of maintenance workflows.

The integrated framework linking virtual digital models with physical world data through a close-loop control system.

As illustrated in the framework above, the system consists of three primary components:

  1. Computer Vision (CV): To track workers in crowded indoor spaces without privacy-invasive sensors, the team developed a deep-learning approach using single fixed cameras. They utilized a 2D human pose predictor to identify joints and introduced "Anthropometric Planes"—virtual parallel planes based on average human height—to project 2D video coordinates onto a 3D layout map, solving the common "loss of depth" problem in single-camera setups.
  2. Human Factors (HF) Analysis: The team analyzed a decade of Licensee Event Reports (LERs) from the Nuclear Regulatory Commission. They discovered that nearly 44% of incidents were rooted in team cognition and communication failures rather than technical glitches, highlighting the need for better behavioral monitoring.
  3. Agent-Based Simulation: Using the NetLogo platform, the researchers built a "digital twin" of the maintenance workflow. This allowed them to test "what-if" scenarios, such as the impact of automated communication systems or specific protocol changes on the overall schedule.

Key Insights and Results

The study revealed that the human supervisor is often a communication bottleneck. Manual reporting chains (worker to supervisor to coordinator to control center) are often too slow for the 10-minute accuracy required in modern outages.

Efficiency Gains

In laboratory settings, replacing a human-mediated reporting system with an automated "blackboard" system—where task status updates automatically trigger notifications for the next team—reduced average workflow duration from 79.97 minutes to 68.39 minutes, a 14.5% improvement.

The "Early-Call" Strategy

One of the most effective strategies identified was the "early-call" protocol: requiring workers to notify supervisors 15–20 minutes before completing a task. Simulation results showed this could reduce overall schedule delays by approximately 3.1% to 3.2%. While seemingly small, this represents tens of thousands of dollars saved daily in a million-dollar-per-day operation.

Tracking Performance

The multi-worker tracking algorithm achieved an average precision of 70%. However, recall was lower (38%) due to the extreme difficulty of tracking individuals in highly occluded, "packed" indoor environments typical of NPPs. The analysis also identified Mode 5 (Cold Shutdown) as the most error-prone phase for human factors in nuclear plants.

Conclusion and Future Work

This research demonstrates the significant potential of integrating computer vision and simulation for NPP outage control. While technical hurdles remain—specifically regarding "ID switching" when workers cross paths and reflections from stainless steel surfaces—the framework provides a clear path toward reducing waste and ensuring safety.

Future work will focus on increasing the robustness of tracking algorithms in compact indoor workspaces and integrating more complex human behavior models, such as "forgetting curves," to predict when fatigue or distraction might lead to critical safety omissions.

Summary updated

Abstract

This report is being submitted for the task “Final Report” of DOE NEUP Project 15-8121 “Automatic Imagery Data Analysis for Proactive Computer-Based Workflow Management during Nuclear Power Plant Outages.” Typical nuclear power plant (NPP) outages always involve thousands of maintenance and refueling activities and a large number of workers in limited workspaces, while having tight schedules and zero-tolerance for accidents. During an outage, thousands of workers will be working in various workspaces across the NPP. High outage costs and expensive delays (approximately 1.5 million dollars of loss per day of delay) in NPP maintenance demand tight outage schedules. In packed workspaces, an automatic system that monitors human behaviors in real-time and provides insights about current and pending schedule deviations from the plan is critical for ensuring: 1) effective collaboration among workers and worker teams from different trades; 2) less waste of time and resources due to the lack of situational awareness; and 3) proactive outage project control. The overall goal of this project is to test the hypothesis that real-time imagery-based object tracking and spatial analysis, as well as human behavior modeling of outage participants, will significantly improve the efficiency of outage control while lowering the rates of accidents and incidents. Three objectives of this project are: 1) Establish real-time object tracking and spatiotemporal analysis methods that automatically assess the productivity of field activities and detect anomalous spatiotemporal relationships among activities that cause inefficiencies and risks; 2) Establish realtime human tracking and human factor modeling methods for automatically diagnosing unexpected actions of and interactions between outage participants, those which cause inefficient collaborations between Advanced Outage Control Center (AOCC), satellite outage centers, NPP workers, and maintenance service providers; and 3) Test the proposed automated object tracking, human behavior modeling, and spatiotemporal analysis methods in outage control case studies in order to characterize the effectiveness of automated imagery-data-driven methods in proactively improving the efficiency and safety of workflows in outage coordination and risk management. Recent studies of detailed human behavior monitoring on construction sites have examined the potential of applying advanced computer vision algorithms in detecting and tracking anomalous workers (i.e., workers who do not wear hard hats or safety vests) for ensuring job site safety. Some studies of human factor studies revealed the importance of modeling detailed interactions between individuals within and across teams in better understanding the impact of human in proactive project control. Other studies in the construction domain have developed computational simulation frameworks that formalize detailed spatiotemporal interactions between tasks in order to simulate the impacts of individual tasks on the performance of workflows. While integrated analyses that combine human factor assessment, as well as image processing and simulation, are in demand for effective decision-making, limited studies have examined the potential of such integrated analyses in NPP outage control. This research project has examined an automatic outage monitoring and control system that integrates human factor analysis, computer vision techniques, and simulation methods in order to enable engineers to better understand the interactions between humans, resources, and workflows during outage processes. The project aims at providing NPP maintenance agencies insights into more efficient use of limited resources in extending the life of a nuclear plant, as well as reducing waste while ensuring sufficient generation of electricity. This study is significant for the safety of nuclear plants, sustainable electricity generation for livable communities, and cost savings for maintaining electricity infrastructures in the United States.
BibTeX
@misc{tang2018automatic,
  author = {Pingo Tang and Alper Yılmaz and Nancy J. Cooke},
  title = {{Automatic Imagery Data Analysis for Proactive Computer-Based Workflow Management during Nuclear Power Plant Outages}},
  year = {2018},
  doi = {10.2172/1491996},
}
Figure