Abstract
The widespread adoption of AI in industry is often hampered by its limited robustness when faced with scenarios absent from training data, leading to prediction bias and vulnerabilities. To address this, we propose a novel streaming inference pipeline that enhances data-driven models by explicitly incorporating prior knowledge. This paper presents the work on an industrial AI application that automatically counts excavator workloads from surveillance videos. Our approach integrates an object detection model with a Finite State Machine (FSM), which encodes knowledge of operational scenarios to guide and correct the AI's predictions on streaming data. In experiments on a real-world dataset of over 7,000 images from 12 site videos, encompassing more than 300 excavator workloads, our method demonstrates superior performance and greater robustness compared to the original solution based on manual heuristic rules.
1. Introduction
While deep learning models for object detection, such as YOLO, achieve high accuracy in controlled laboratory settings, their performance often degrades in real-world industrial environments. Construction sites, characterized by heavy dust, poor lighting, and complex occlusions, present significant challenges. In these settings, AI models frequently encounter "hallucinations" or misidentifications—for instance, mistaking a person or a piece of equipment for a target vehicle.
This paper addresses the specific challenge of automatically counting excavator workloads from surveillance video, a task critical for equipment leasing and operational tracking. Traditional heuristic-based counting methods are often brittle, failing to distinguish between actual work cycles and transient noise. The core challenge is bridging the gap between raw pixel detection and the high-level logical sequence of industrial operations.
2. Methodology
We propose a two-stage streaming inference pipeline that bridges the gap between raw pixel-level detection and high-level operational logic.
2.1 Event Identification
The first stage utilizes a YOLOv9 model to detect three primary classes: Trucks, Vertical Buckets, and Horizontal Buckets. From these raw detections, the system identifies five specific events based on spatial relationships and temporal changes, such as a bucket approaching a truck (calculated by the decreasing distance between the two largest detected objects).
2.2 FSM-Driven Inference
The core of our approach is the integration of a Finite State Machine (FSM) that encodes the "common sense" business logic of an excavator's workload. The FSM acts as a contextual filter, ensuring that detected events follow a logical sequence: Digging → Carrying → Approaching → Possibly Unloaded → Unloaded.
By maintaining a current state, the system can reject detections that deviate from the expected operational flow. For example, an "unloading" event is ignored if the system is currently in the "Digging" state, effectively filtering out false positives caused by environmental noise. A workload is only officially counted when the FSM reaches the final Unloaded state and then resets to Start.
3. Experimental Evaluation
The system was evaluated using a comprehensive real-world dataset:
- Training Set: Approximately 16,000 images from diverse construction sites.
- Evaluation Set: Over 7,000 images from 12 distinct site videos, capturing more than 300 complete workloads.
- Environmental Diversity: The data includes challenging conditions such as heavy dust, rain, nighttime operations, and varying camera angles (4–12m height).
4. Key Results
The FSM-driven approach significantly outperformed traditional manual heuristic rules across all key metrics:
- Precision: Increased from 0.91 to 0.96, demonstrating a marked reduction in "ghost" workload counts.
- F1-Score: Improved to 0.94, a 2% increase over the baseline.
- Stability: The system reduced the number of false workload counts from 34 (in a loose heuristic system) to just 13.
- Generalization: The FSM logic remained stable across 12 different locations without requiring site-specific tuning, a major advantage over manual rules.
5. Conclusion
By pairing deep learning's spatial recognition capabilities with the temporal logic of a Finite State Machine, we have developed a more reliable and robust industrial AI system. This hybrid approach provides the "common sense" necessary to navigate messy, real-world environments where pure data-driven models often fail. Future work will focus on automatically learning state transitions and extending the system to handle multi-machine scenarios.
Paper Citation: Zhang, Y., Pei, Z., et al. "Boosting AI Reliability with an FSM-Driven Streaming Inference Pipeline: An Industrial Case." Preprint Version, 2026.
Summary updated