Research on a Cooperative Grasping Method for Heterogeneous Objects in Unstructured Scenarios of Mine Conveyor Belts Based on an Improved MATD3

Rui Gao, M. Liu, Jingyi Du, Yifan Bao, Xudong Wu, Jiahui LiuSensors 2025 · DOI

TL;DR

This paper introduces an improved Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm, dubbed P-MATD3, for robust cooperative grasping of diverse foreign objects on mine conveyor belts. By incorporating a multi-factor reward function, prioritized experience replay, and a sequential optimization strategy for slender objects, the method significantly boosts grasping success rates and efficiency in challenging, unstructured environments.

What Problem Does This Paper Address?

Underground coal mines rely on conveyor belts to transport materials. However, these belts often become contaminated with foreign objects like coal gangue, anchor bolts, and wood pieces. These contaminants disrupt operations and pose safety risks, necessitating their removal. The challenge lies in the heterogeneity of these objects – they vary greatly in shape, size, and orientation – and the unstructured, unpredictable nature of the mining environment. Traditional robotic control methods struggle with this complexity due to their reliance on precise models and fixed parameters, leading to poor adaptability and robustness. This paper aims to develop a cooperative grasping system that can reliably handle these diverse objects in such challenging conditions.

Methods

The core of the proposed solution is an enhanced MATD3 algorithm, referred to as P-MATD3. MATD3 is a deep reinforcement learning (DRL) algorithm designed for multi-agent cooperative tasks, building upon the TD3 algorithm to mitigate issues like value function overestimation and policy instability. The authors introduce three key improvements to MATD3:

  • Multi-Factor Reward Function: This function is designed to guide the robotic arms more effectively. It incorporates factors like end-effector distance to the target, collision avoidance, guidance towards the target, and task completion, aiming to accelerate convergence and optimize grasping trajectories in continuous action spaces.
  • Prioritized Experience Replay (PER): PER is integrated to improve sample efficiency. Instead of sampling experiences uniformly, PER prioritizes experiences with higher temporal-difference (TD) errors, meaning it focuses on learning from more informative transitions. This accelerates convergence to optimal policies and enhances adaptability to dynamic environments.
  • Sequential Cooperative Optimization Strategy: For slender objects (like anchor rods), a specific strategy is developed. This strategy breaks down the grasping and placement into sequential stages (grasp, pull-up, transfer, place) with tailored interpolation and inverse kinematics calculations to ensure stable and reliable manipulation.

The system uses a dual-arm setup, with each arm acting as an independent agent within the MATD3 framework. The state space includes joint positions, end-effector and target point displacements, and a task completion flag. The action space consists of joint angle increments for each robotic arm.

Datasets

The experiments were conducted in the PyBullet simulation environment. The paper does not mention the use of external datasets for training or evaluation. Instead, it focuses on simulating the complex scenario of foreign object removal from a conveyor belt. The objects themselves are represented by bounding box models, and their positions are randomly generated within defined regions to ensure diversified and accessible targets during training.

Key Insights and Ideas

  • Cooperative Grasping in Unstructured Environments is Achievable: The research demonstrates that DRL, particularly with the proposed enhancements, can effectively tackle the complex problem of cooperative grasping in highly unstructured and unpredictable environments like mine conveyor belts.
  • Reward Shaping is Crucial for Efficiency: The multi-factor reward function is a key enabler for faster convergence and more precise trajectory planning, highlighting the importance of carefully designing the learning signal in DRL.
  • Prioritization Accelerates Learning: PER significantly boosts learning efficiency by allowing the agents to learn more effectively from critical experiences, leading to faster policy optimization.
  • Specialized Strategies for Object Types Matter: The sequential optimization strategy for slender objects underscores that a one-size-fits-all approach is insufficient. Tailoring manipulation strategies to object characteristics is vital for robust performance.
  • Robustness to Noise is Essential for Real-World Deployment: The study's validation under Gaussian noise demonstrates that the proposed method maintains stability even with sensor inaccuracies, a critical step towards real-world applicability.

Key Results

The P-MATD3 algorithm demonstrated significant improvements over baseline MATD3 and MADDPG algorithms:

  • Single-Arm Grasping:
    • Grasping success rate increased by 7.1% (vs. MATD3) and 9.94% (vs. MADDPG).
  • Number of steps to reach the pre-grasping point reduced by 11.44% (vs. MATD3) and 12.77% (vs. MADDPG).
  • Dual-Arm Cooperative Grasping:
    • Success rates increased by 5.58% (vs. MATD3) and 9.84% (vs. MADDPG).
  • Step counts decreased by 11.6% (vs. MATD3) and 18.92% (vs. MADDPG).
  • Robustness: The algorithm maintained high stability and a notable success rate even with Gaussian noise up to 0.1, indicating good performance under moderate environmental uncertainties.
  • Training Stability: Ablation studies showed that the combined P-MATD3 approach (reward function + PER) achieved superior stability and performance compared to using individual components or the base MATD3 algorithm.

Limitations and Future Work

  • Simulation to Reality Gap: The authors acknowledge that the current work is based on simulation. A significant future challenge is bridging the gap between simulation and real-world deployment, which involves addressing issues like sensor noise, actuator limitations, and dynamic uncertainties not fully captured in simulation.
  • Kinematic vs. Dynamic Planning: The current focus is on kinematic planning. Future work could explore incorporating dynamic aspects of robotic arm control, which are crucial for real-world scenarios involving nonlinear friction, motor saturation, and latency.
  • Perception Enhancements: While the current method is effective, future research could benefit from advanced perception architectures (e.g., YOLO and GG-CNN) to enhance robustness and generalization, particularly for precise manipulation.
  • Real-World Perturbations: Further validation with real-world perturbations beyond Gaussian noise is suggested to thoroughly test the algorithm's generalization capabilities.
  • Safety in Dual-Arm Cooperation: The inherent safety risks of dual-arm cooperation, such as collisions, are noted as challenges that simulations may not fully represent. Future work could focus on more robust safety mechanisms.

Summary updated

Abstract

Underground coal mine conveying systems operate in unstructured environments. Influenced by geological and operational factors, coal conveyors are frequently contaminated by foreign objects such as coal gangue and anchor bolts. These contaminants disrupt conveying stability and pose challenges to safe mining operations, making their effective removal critical. Given the significant heterogeneity and unpredictability of these objects in shape, size, and orientation, precise manipulation requires dual-arm cooperative control. Traditional control algorithms rely on precise dynamic models and fixed parameters, lacking robustness in such unstructured environments. To address these challenges, this paper proposes a cooperative grasping method tailored for heterogeneous objects in unstructured environments. The MATD3 algorithm is employed to cooperatively perform dual-arm trajectory planning and grasping tasks. A multi-factor reward function is designed to accelerate convergence in continuous action spaces, optimize real-time grasping trajectories for foreign objects, and ensure stable robotic arm positioning. Furthermore, priority experience replay (PER) is integrated into the MATD3 framework to enhance experience utilization and accelerate convergence toward optimal policies. For slender objects, a sequential cooperative optimization strategy is developed to improve the stability and reliability of grasping and placement. Experimental results demonstrate that the P-MATD3 algorithm significantly improves grasping success rates and efficiency in unstructured environments. In single-arm tasks, compared to MATD3 and MADDPG, P-MATD3 increases grasping success rates by 7.1% and 9.94%, respectively, while reducing the number of steps required to reach the pre-grasping point by 11.44% and 12.77%. In dual-arm tasks, success rates increased by 5.58% and 9.84%, respectively, while step counts decreased by 11.6% and 18.92%. Robustness testing under Gaussian noise demonstrated that P-MATD3 maintains high stability even with varying noise intensities. Finally, ablation and comparative experiments comprehensively validated the proposed method's effectiveness in simulated environments.
BibTeX
@inproceedings{gao2025research,
  author = {Rui Gao and M. Liu and Jingyi Du and Yifan Bao and Xudong Wu and Jiahui Liu},
  title = {{Research on a Cooperative Grasping Method for Heterogeneous Objects in Unstructured Scenarios of Mine Conveyor Belts Based on an Improved MATD3}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25226824},
}
Figure