Automating hydraulic excavators is a significant challenge in the field of heavy machinery. These systems are characterized by multi-input multi-output (MIMO) configurations and highly complex nonlinear dynamics. Factors such as massive inertia, gravitational effects, hydraulic time delays, and state-dependent dead zones make precise trajectory tracking extremely difficult. Traditional control methods, such as PID or Model Predictive Control (MPC), often struggle to handle these nonlinearities without prohibitively complex mechanistic models. Conversely, model-free Reinforcement Learning (RL) methods, while capable of mapping nonlinearities, are notoriously data-inefficient, requiring millions of interaction steps that are impractical and potentially hazardous for heavy equipment like a 7.5-ton excavator.
To address these challenges, we introduce EfficientTrack, a model-based learning framework designed for high-precision and high-efficiency trajectory tracking. EfficientTrack avoids the need to learn global open-loop dynamics from scratch by focusing on the closed-loop dynamics of the system.
The EfficientTrack Framework
The core innovation of EfficientTrack lies in its hierarchical structure, which leverages existing control stability while adding a layer of intelligent adjustment.
The system consists of three primary components:
- Underlying Controller: A standard Proportional-Derivative (PD) controller manages low-level joint movements, providing a baseline of stability.
- Closed-Loop Dynamics Model: A neural network is trained to predict future excavator states based on historical observations and the reference trajectories provided to the PD controller.
- Trajectory Adjustment Policy: A second neural network serves as a "whisperer" to the PD controller. It analyzes the desired trajectory and outputs adjusted reference points, effectively "tricking" the PD controller into compensating for physical lags and inherent errors.
Multi-Step Gradient Backpropagation
A key technical contribution is the use of multi-step gradient backpropagation for training. Unlike standard RL, which relies on high-variance value function estimations, EfficientTrack backpropagates tracking errors directly through the frozen dynamics model.
This approach provides a dense, direct gradient signal, allowing the policy to learn precise corrections. To ensure robustness, Gaussian noise is injected into reference trajectories during data collection, preventing the model from overfitting to idealized paths and enabling it to recover from real-world disturbances.
Experimental Results
EfficientTrack was evaluated using a high-fidelity simulator and a real-world XCMG 7.5-ton XE75 excavator equipped with IMU and GNSS/INS sensors.
- Simulation Performance: In comparative benchmarks, EfficientTrack outperformed state-of-the-art methods including PPO, SAC, and TD-MPC. It achieved a Mean Absolute Error (MAE) of $0.32^\circ$ with only 2.45 hours of interaction time. In contrast, PPO required 171 hours of interaction to achieve a significantly higher MAE of $4.45^\circ$.
- Real-World Precision: On physical hardware, the baseline PD controller exhibited a significant end-effector error of 79.21 cm. EfficientTrack reduced this error to just 6.53 cm after three rounds of continual learning. Remarkably, the system maintained sub-10 cm accuracy even when trained on as little as 15 minutes of real-world data.
- Robustness: The policy demonstrated excellent generalization, maintaining a low MAE of $0.68^\circ$ when performing actual digging tasks with loaded buckets, without requiring any additional retraining.
Conclusion
EfficientTrack demonstrates that learning closed-loop dynamics is a highly effective strategy for automating heavy machinery. By combining the stability of traditional controllers with the nonlinear modeling capabilities of deep learning, the framework achieves unprecedented precision and data efficiency. Future work will focus on extending this approach to more complex, dynamic tasks and diverse excavator models to prove its generalizability across diverse construction scenarios.
Summary updated