Towards Autonomous Mining via Intelligent Excavators

Hooman Shariati, Anuar Yeraliyev, Burhan Terai, Shahram Tafazoli, Mahdi RamezaniWorkshops @ CVPR 2019

Abstract

Mining productivity has faced a significant decline over the last decade, while safety remains a paramount concern due to the high risks associated with mobile equipment. This paper introduces a robust, real-time perception system for mining excavators designed to operate in extreme environments. By employing a multi-frame LSTM-YOLO architecture, the system achieves over 90% accuracy in detecting critical components like bucket teeth and payload materials. This technology provides a foundational layer for situational awareness, paving the way for future autonomous mining operations.

Introduction

The mining industry operates in high-stakes environments where safety and productivity are often in conflict. Data from the Mine Safety and Health Administration (MSHA) indicates that nearly 40% of mining fatalities involve mobile equipment. Simultaneously, global mining productivity has dropped by 28% in the last ten years.

A major operational hurdle is the limited visibility from excavator cabins. Operators often cannot clearly see the bucket's contents or the condition of Ground Engaging Tools (GET), such as bucket teeth. A broken metal tooth entering a downstream crusher can cause millions of dollars in damage and create significant safety hazards. This research addresses the need for an intelligent monitoring system capable of real-time detection under brutal conditions, including heavy dust, extreme vibration, and poor lighting.

Methods: Multi-Frame Temporal Integration

To overcome the limitations of single-frame object detection in noisy environments, the authors propose a multi-frame approach.

  • Backbone Architecture: YOLOv2 was selected as the primary feature extractor due to its superior speed compared to alternatives like SSD on the target hardware.
  • Temporal Modeling: Convolutional LSTM layers are integrated to "remember" information across video frames. This allows the model to maintain detection even when objects are momentarily obscured by dust or glare.
  • Operational Efficiency: The system processes video at 10 FPS. Downsampling from 30 FPS provides a larger "temporal receptive field," allowing the LSTM to observe actions over a longer duration without increasing computational overhead.
  • Ruggedized Deployment: The model is deployed on industrial embedded hardware (Intel CPU and Nvidia GTX1050Ti) designed to withstand extreme temperatures (-40°C to +55°C) and mechanical shocks.
Perception system overlays demonstrating the identification of bucket regions and teeth lines across various shovel types and environmental conditions.

Datasets and Evaluation

The system was trained and tested on a proprietary dataset from diverse global mining sites. Approximately 3,000 human-labeled images were used per shovel type (Hydraulic, Bucyrus, and P&H). To evaluate generalization, the researchers used four distinct test sets ranging from known sites to "hard" examples featuring extreme dust, glare, or low contrast.

Key Results and Insights

The multi-frame model consistently outperformed single-frame baselines, particularly in challenging conditions.

  • Performance Metrics: The system achieved a Mean Average Precision (mAP) of 0.953 on standard sets and 0.883 on the most difficult examples.
  • Temporal Robustness: The LSTM's ability to aggregate features across 10 frames (1 second) proved crucial for maintaining a "lock" on bucket teeth during intermittent occlusions.
  • Generalization: The model successfully generalized to unseen equipment types by learning the underlying semantics of shovel operations rather than just memorizing specific visual patterns.
  • Physical Analytics: By mapping pixel dimensions to known physical bucket widths, the system automatically calculates rock fragmentation sizes and tooth wear levels without manual calibration.
Object detection results showing bounding boxes for teeth (green), bucket boundaries (blue), and payload (red/cyan) across different equipment and lighting scenarios.

Limitations and Future Work

While the system handles intermittent occlusion well, persistent obstructions—such as thick mud on the lens—remain a challenge. Future iterations will focus on self-diagnostic capabilities to alert operators when imaging conditions are too poor for reliable detection. Additionally, the researchers plan to expand the detection vocabulary to include more specific components like lip-shrouds and integrate the system with high-definition maps for full excavator autonomy.

Summary updated

BibTeX
@inproceedings{shariati2019towards,
  author = {Hooman Shariati and Anuar Yeraliyev and Burhan Terai and Shahram Tafazoli and Mahdi Ramezani},
  title = {{Towards Autonomous Mining via Intelligent Excavators}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
}

Loading related content...

Figure