Automatic Evaluation of Excavator Operators using Learned Reward Functions

Pranav Agarwal, Marek Teichmann, Sheldon Andrews, Samira Ebrahimi KahouarXiv 2022 · DOI · arXiv

Abstract

This paper introduces a framework for the automatic evaluation of excavator operators by learning reward functions directly from expert demonstrations. By modeling machine dynamics and safety constraints as probabilistic distributions, the authors generate a dense reward signal. This signal facilitates real-time feedback for human trainees and enables Reinforcement Learning (RL) agents to achieve human-like safety and efficiency in simulated maneuvers. The approach addresses the challenges of subjective human evaluation and the difficulty of manual reward engineering in complex robotics tasks.

Introduction

Operating heavy machinery like excavators is a high-skill task that traditionally requires expensive, one-on-one training with expert instructors. However, human evaluation is notoriously subjective, noisy, and prone to bias. From an automation perspective, training an AI to operate an excavator via Reinforcement Learning (RL) faces the "reward engineering" challenge. Sparse rewards (e.g., "did you reach the goal?") lead to slow convergence, while dense rewards (mathematical formulas for every micro-action) are incredibly difficult to design manually without causing unintended or unsafe behaviors. This research seeks to bridge this gap by "learning" what a good operator looks like from data, rather than coding it by hand.

Methodology

The authors propose a two-stage approach to automate evaluation and reward generation: representation learning and policy optimization.

Representation Learning (The "Expert Model")

Instead of simple imitation, the framework learns two independent probabilistic distributions from expert data:

  1. Dynamic Distribution ($q_D$): An LSTM-based encoder-decoder architecture processes a window of 32 time-steps (including torque, power, and fuel consumption). It learns to predict future states, effectively capturing the mechanical limits and operational "rhythm" of an expert.
  2. Safety/Infraction Distribution ($q_I$): A 2-layered Multi-Layer Perceptron (MLP) predicts the likelihood of infractions—such as collisions with poles or environment boundaries—based on the current state. This creates a smooth "safety landscape" that allows agents to anticipate collisions before they occur.
Network architecture and resulting safety distribution for predicting total infractions.

Reward Formulation and Optimization

The composite reward function is defined as: $$R = R_{Goal} + R_{Dynamic} + R_{Safety}$$ The $R_{Dynamic}$ and $R_{Safety}$ components are calculated using Kullback–Leibler (KL) divergence, measuring how much the current operator's actions deviate from the learned expert distributions. The authors use the Proximal Policy Optimization (PPO) algorithm to train an RL agent in a simulated environment, using these learned rewards to guide the agent toward expert-level performance.

Experimental Setup

The study utilizes the Vortex Simulator, a high-fidelity physics engine for construction equipment. Trajectories were collected from 40 different users, with 7 classified as "experts" based on a heuristic scoring system. The observation space includes a 19-dimensional vector covering linear/angular positions and velocities of the bucket, stick, boom, and swing actuators, alongside engine metrics.

The Vortex Simulator environment used for data collection and RL training, showing the excavator and task-related obstacles.

Key Results and Insights

The authors compared an RL agent trained only on the task goal against an agent trained with the learned Dynamic and Safety rewards:

  • Safety Improvement: The task-only agent averaged 35 infractions per episode. In contrast, the agent using the learned safety reward reduced this to near zero (approximately 1 infraction).
  • Operational Efficiency: The "Dynamic+Safety" agent matched expert levels of torque (46.44%) and fuel consumption (5.12%) much more closely than the baseline task-only agent, which tended to over-rev the engine (51.73% torque).
  • Dynamics as a Regularizer: A key finding is that the "Dynamic" reward acts as a constraint that prevents the RL agent from exploring physically unrealistic actions, leading to faster convergence compared to agents trained on task goals alone.
  • Human-Like Trajectories: Qualitatively, agents trained with learned rewards followed smoother paths that closely replicated the "style" of expert human operators.

Conclusion and Future Work

This framework demonstrates that learned reward functions can effectively bridge the gap between human expertise and automated control. The same models used to train AI can serve as objective "automated coaches" for human trainees. Future work aims to develop a single, unified representation (a "foundation model" for excavators) to handle multiple constraints more efficiently and to test the transferability of these learned rewards from simulation to real-world hydraulic excavators.

Summary updated

Abstract

Training novice users to operate an excavator for learning different skills requires the presence of expert teachers. Considering the complexity of the problem, it is comparatively expensive to find skilled experts as the process is time-consuming and requires precise focus. Moreover, since humans tend to be biased, the evaluation process is noisy and will lead to high variance in the final score of different operators with similar skills. In this work, we address these issues and propose a novel strategy for the automatic evaluation of excavator operators. We take into account the internal dynamics of the excavator and the safety criterion at every time step to evaluate the performance. To further validate our approach, we use this score prediction model as a source of reward for a reinforcement learning agent to learn the task of maneuvering an excavator in a simulated environment that closely replicates the real-world dynamics. For a policy learned using these external reward prediction models, our results demonstrate safer solutions following the required dynamic constraints when compared to policy trained with task-based reward functions only, making it one step closer to real-life adoption. For future research, we release our codebase at https://github.com/pranavAL/InvRL_Auto-Evaluate and video results https://drive.google.com/file/d/1jR1otOAu8zrY8mkhUOUZW9jkBOAKK71Z/view?usp=share_link .
BibTeX
@misc{agarwal2022automatic,
  author = {Pranav Agarwal and Marek Teichmann and Sheldon Andrews and Samira Ebrahimi Kahou},
  title = {{Automatic Evaluation of Excavator Operators using Learned Reward Functions}},
  year = {2022},
  eprint = {2211.07941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.07941},
}
Figure