Abstract
This paper introduces a framework for the automatic evaluation of excavator operators by learning reward functions directly from expert demonstrations. By modeling machine dynamics and safety constraints as probabilistic distributions, the authors generate a dense reward signal. This signal facilitates real-time feedback for human trainees and enables Reinforcement Learning (RL) agents to achieve human-like safety and efficiency in simulated maneuvers. The approach addresses the challenges of subjective human evaluation and the difficulty of manual reward engineering in complex robotics tasks.
Introduction
Operating heavy machinery like excavators is a high-skill task that traditionally requires expensive, one-on-one training with expert instructors. However, human evaluation is notoriously subjective, noisy, and prone to bias. From an automation perspective, training an AI to operate an excavator via Reinforcement Learning (RL) faces the "reward engineering" challenge. Sparse rewards (e.g., "did you reach the goal?") lead to slow convergence, while dense rewards (mathematical formulas for every micro-action) are incredibly difficult to design manually without causing unintended or unsafe behaviors. This research seeks to bridge this gap by "learning" what a good operator looks like from data, rather than coding it by hand.
Methodology
The authors propose a two-stage approach to automate evaluation and reward generation: representation learning and policy optimization.
Representation Learning (The "Expert Model")
Instead of simple imitation, the framework learns two independent probabilistic distributions from expert data:
- Dynamic Distribution ($q_D$): An LSTM-based encoder-decoder architecture processes a window of 32 time-steps (including torque, power, and fuel consumption). It learns to predict future states, effectively capturing the mechanical limits and operational "rhythm" of an expert.
- Safety/Infraction Distribution ($q_I$): A 2-layered Multi-Layer Perceptron (MLP) predicts the likelihood of infractions—such as collisions with poles or environment boundaries—based on the current state. This creates a smooth "safety landscape" that allows agents to anticipate collisions before they occur.
Reward Formulation and Optimization
The composite reward function is defined as: $$R = R_{Goal} + R_{Dynamic} + R_{Safety}$$ The $R_{Dynamic}$ and $R_{Safety}$ components are calculated using Kullback–Leibler (KL) divergence, measuring how much the current operator's actions deviate from the learned expert distributions. The authors use the Proximal Policy Optimization (PPO) algorithm to train an RL agent in a simulated environment, using these learned rewards to guide the agent toward expert-level performance.
Experimental Setup
The study utilizes the Vortex Simulator, a high-fidelity physics engine for construction equipment. Trajectories were collected from 40 different users, with 7 classified as "experts" based on a heuristic scoring system. The observation space includes a 19-dimensional vector covering linear/angular positions and velocities of the bucket, stick, boom, and swing actuators, alongside engine metrics.
Key Results and Insights
The authors compared an RL agent trained only on the task goal against an agent trained with the learned Dynamic and Safety rewards:
- Safety Improvement: The task-only agent averaged 35 infractions per episode. In contrast, the agent using the learned safety reward reduced this to near zero (approximately 1 infraction).
- Operational Efficiency: The "Dynamic+Safety" agent matched expert levels of torque (46.44%) and fuel consumption (5.12%) much more closely than the baseline task-only agent, which tended to over-rev the engine (51.73% torque).
- Dynamics as a Regularizer: A key finding is that the "Dynamic" reward acts as a constraint that prevents the RL agent from exploring physically unrealistic actions, leading to faster convergence compared to agents trained on task goals alone.
- Human-Like Trajectories: Qualitatively, agents trained with learned rewards followed smoother paths that closely replicated the "style" of expert human operators.
Conclusion and Future Work
This framework demonstrates that learned reward functions can effectively bridge the gap between human expertise and automated control. The same models used to train AI can serve as objective "automated coaches" for human trainees. Future work aims to develop a single, unified representation (a "foundation model" for excavators) to handle multiple constraints more efficiently and to test the transferability of these learned rewards from simulation to real-world hydraulic excavators.
Summary updated