Compliant Motion Planning Integrating Human Skill for Robotic Arm Collecting Tomato Bunch Based on Improved DDPG

Yifan Zhang, Yajun Li, Qingchun Feng, Jiahui Sun, Chuanlang Peng, Li Gao, Liping ChenPlants 2025 · DOI

TL;DR

This paper introduces DDPG-Z, a novel reinforcement learning approach that integrates human demonstration data to train robotic arms for delicate tomato bunch collection. By incorporating human skills into a DDPG framework with specialized pre-training and reward mechanisms, the system achieves more compliant, accurate, and efficient fruit handling than standard DRL methods.

What Problem Does This Paper Address?

Harvesting delicate fruits like tomatoes presents a significant challenge for robots. Traditional robotic harvesting often involves simply picking and dropping fruit bunches into a container, which can lead to damage due to impact. Human harvesters, however, employ a "compliant" and "dexterous" collection technique, carefully placing bunches to avoid damage and ensure orderly arrangement. Current path planning methods for robots struggle to replicate these nuanced human skills, often focusing on collision-free paths rather than the subtle movements required for delicate handling. This paper aims to bridge this gap by enabling robotic arms to learn and reproduce these human-like compliant collection behaviors.

Methods

The core of this work is an enhanced Deep Deterministic Policy Gradient (DDPG) algorithm, dubbed DDPG-Z. The key innovations include:

  • Human-Robot Skill Transfer: Demonstration paths are acquired by manually guiding a robotic arm to collect tomato bunches. These paths are then processed and used to train the DDPG agent.
  • Pre-training with Demonstration Data: The agent is first "pre-trained" offline using the collected demonstration data. This helps it learn the fundamental compliant movements before interacting with the environment.
  • Expert Reward Regression (ERR) Loss: A specialized loss function is introduced during pre-training to stabilize the learning process and better align the agent's policy with human expert behavior.
  • Dynamic Step-Length Return (DSR): This mechanism dynamically adjusts the lookahead horizon for rewards during online training. It starts with shorter horizons to focus on immediate rewards and gradually increases to consider longer-term outcomes, balancing learning speed and long-term performance.
  • Prioritized Experience Replay (PER): This standard technique is applied to both demonstration and interaction data, allowing the agent to learn more efficiently by prioritizing more informative experiences.

The DDPG-Z model uses four neural networks (actor, critic, and their respective target networks) and maintains two replay buffers: one for demonstration data and one for interaction data.

Datasets

The paper utilizes a custom dataset generated through kinesthetic teaching (manual guidance) of a robotic arm performing tomato bunch collection. This dataset consists of spatial trajectories recorded from the robotic arm's end-effector. The experiments were conducted in a realistic simulation environment built using MuJoCo, which models the physics of the robotic arm, fruit bunch, and basket.

Key Insights and Ideas

  • Demonstrations are Crucial for Dexterity: The study strongly suggests that incorporating human demonstration data is paramount for teaching robots complex, compliant manipulation tasks, significantly outperforming methods relying solely on exploration.
  • Pre-training is More Than Just Initialization: The ERR loss and pre-training phase are not just about providing a starting point; they actively shape the agent's initial policy towards human-like dexterity, leading to faster and more stable convergence.
  • Balancing Short-Term and Long-Term Rewards is Key: The DSR mechanism highlights the importance of dynamically adjusting reward horizons to effectively learn complex tasks, preventing premature convergence to suboptimal local policies.
  • Simulation-to-Reality Gap is Manageable: The paper demonstrates that a well-tuned simulation environment can effectively train a policy that translates to good performance on a real robotic system.

Key Results

In a realistic simulation environment, DDPG-Z demonstrated significant improvements:

  • Convergence Speed: 25% faster convergence compared to the "No Demo" model.
  • Post-Convergence Reward: 10.3% higher average reward after convergence compared to the "No Demo" model.
  • Destination Accuracy: 51.3% improvement in destination accuracy compared to the "No Demo" model.
  • Success Rate: Achieved a 100% collection success rate in evaluation episodes, significantly outperforming other DRL baselines (DDPG, SAC, TD3) which often failed to converge or achieved very low success rates (e.g., 15% for DDPG).
  • Path Similarity: The paths planned by DDPG-Z closely matched the human demonstration paths, exhibiting smoother trajectories and more precise contact points and destinations.
  • Real-World Performance: Tests with a real robotic arm showed placement effects consistent with simulation results, demonstrating dexterous and orderly placement without collisions or free falls.

Limitations and Future Work

The authors acknowledge that the current planning process is open-loop. This means the model plans paths based on pre-set positions without real-time feedback on dynamic changes in the environment. This can lead to issues like mutual extrusion between bunches if the robotic arm's accuracy or bunch placement isn't perfect.

Future work will focus on:

  • Real-time Feedback Mechanisms: Incorporating visual information to allow the robot arm to adjust its placement position dynamically.
  • Enhanced Adaptability and Flexibility: Addressing limitations related to movement accuracy and bunch size variations to further improve performance in complex, real-world scenarios.

Summary updated

Abstract

Dexterous manipulation and gradual placement are crucial for preserving fruit integrity during harvesting. Addressing the limitations of conventional path planning methods in learning manual compliant skills, we propose a novel method for tomato bunch collection that integrates human-robot skill transfer with Deep Deterministic Policy Gradient (DDPG). In our method, a demonstrator manually guided the robotic arm using an existing tomato collection mechanism, with spatial trajectories recorded as demonstration paths. We then developed an enhanced DDPG-Z model that incorporates human skill replay for pre-training, expert reward regression loss to stabilize pre-training, and dynamic step-length returns to balance short- and long-term rewards. Subsequently, the agent was trained to minimize the deviations of key points between the demonstration paths and actual paths, effectively approximating human operations. In a highly realistic simulation environment, our method achieved a 25% improvement in convergence speed, a 10.3% increase in post-convergence reward, and a 51.3% boost in destination accuracy compared to the case without the demonstrations, whereas classical models such as DDPG, SAC (Soft Actor-Critic), and TD3 (Twin Delayed Deep Deterministic Policy Gradient) failed to converge within the prescribed episodes. This work provides valuable insights for enhancing the compliant operational performance of agricultural robots.
BibTeX
@inproceedings{zhang2025compliant,
  author = {Yifan Zhang and Yajun Li and Qingchun Feng and Jiahui Sun and Chuanlang Peng and Li Gao and Liping Chen},
  title = {{Compliant Motion Planning Integrating Human Skill for Robotic Arm Collecting Tomato Bunch Based on Improved DDPG}},
  booktitle = {Plants},
  year = {2025},
  doi = {10.3390/plants14050634},
}
Figure