FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers' Preference Elicitation

Hanfang Lyu, Yuanchen Bai, Xin Liang, Ujaan Das, Chuhan Shi, Leiliang Gong, Yingchi Li, Mingfei Sun, Ming Ge, Xiaojuan MaarXiv 2024 · DOI · arXiv

TL;DR

FARPLS is a feature-augmented labeling system designed to help humans provide high-quality preference data for training robots. By automatically highlighting key trajectory features and dynamically ordering tasks, it significantly improves labeling consistency and user engagement without increasing cognitive load.

What Problem Does This Paper Address?

Teaching robots to perform tasks that align with human values often uses Reinforcement Learning from Human Preferences (RLHP). In this setup, humans watch pairs of robot videos and select which one they prefer. However, robot trajectories are complex and high-dimensional.

Non-expert labelers often struggle with three main challenges:

  1. Difficulty in forming criteria: It's hard to know what "good" looks like without seeing a wide range of examples first.
  2. Overlooking details: Subtle but critical events, like a minor collision or a slightly unstable grasp, are easily missed in a standard video player.
  3. Mental fatigue: Comparing hundreds of similar-looking videos is tedious, leading to inconsistent labels and "noisy" data that degrades robot training.

Methods

The authors developed FARPLS (Feature-Augmented Robot trajectory Preference Labeling System) to bridge the gap between raw video data and human sense-making. The system's architecture is built around three core design requirements:

  • Feature Augmentation & Visualization: The system automatically extracts 11 features across three categories: Safety (e.g., collisions, contact force), Efficiency (e.g., speed, path length), and Task Quality (e.g., smoothness, grasp position). It visualizes these as distribution charts, highlighting "outliers" to draw the labeler's attention to distinctive characteristics.
  • Keyframe Extraction: FARPLS identifies critical moments (like the exact frame of a collision) and provides "jump-to" buttons, allowing labelers to quickly review and compare specific events across videos.
  • Dynamic Prompting Strategy: Instead of random pairs, the system initially presents high-variance pairs to help users establish their internal preference criteria. It then balances difficulty and novelty based on how many "clusters" of trajectories the user has seen.
  • Consistency Monitoring: The system periodically re-prompts previously labeled pairs to check for consistency, providing real-time feedback and encouraging messages to keep users focused.
Figure 2: This figure demonstrates an overview of our design pipeline: from identified challenges, we establish the design requirements to address each challenge, then design corresponding features to meet those requirements, and finally, we integrate those features into concrete components of our labeling system.

Datasets

The researchers introduced PickPlaceCans, a dataset based on the robosuite simulation framework. It consists of 636 successful pick-and-place trajectories, including proficient human demonstrations, multi-human demonstrations, and machine-generated paths. Each trajectory is augmented with the 11 extracted features and corresponding keyframes.

Key Insights and Ideas

  • The Dunning-Kruger Effect in Data Labeling: A major finding was that labelers using standard interfaces were just as confident as those using FARPLS, yet their actual labeling consistency was significantly lower. Humans often don't realize what details they are missing until a system explicitly points them out.
  • Criteria Formation as a "Cold Start" Problem: Labelers need a "calibration" phase. By showing diverse, high-contrast examples early on, FARPLS helps users build a more robust and stable set of rules for what makes one trajectory better than another.
  • Engagement Over Speed: Interestingly, FARPLS users took about 40% longer to complete their tasks. However, they reported higher engagement and lower boredom. The extra time wasn't "wasted"; it was spent in active analysis rather than passive viewing.

Key Results

The system was evaluated in a between-subjects study with 42 participants, each labeling 105 pairs of trajectories.

  • Higher Consistency: The FARPLS group achieved a mean consistency score of 0.89, significantly higher than the 0.79 achieved by the baseline group ($p < 0.01$).
  • Improved Sense-making: Subjective ratings for "feature support" (5.76 vs 4.10) and "comparison support" (5.86 vs 4.76) were significantly higher for FARPLS.
  • Stable Cognitive Load: Despite the additional information (charts, buttons, feedback), users did not report a significant increase in mental effort compared to the simple baseline interface.
The FARPLS user interface juxtaposes two videos with auxiliary feature distributions and keyframe buttons to assist comparison.

Limitations and Future Work

While promising, the study has a few limitations:

  • Task Complexity: The evaluation was limited to a relatively simple pick-and-place task. Future work needs to test if these benefits hold for multi-step or collaborative human-robot tasks.
  • Downstream Validation: The study focused on label quality (consistency). The next step is to train actual reward models on this data to see if the improved labels lead to measurably better robot performance.
  • Algorithmic Integration: The authors suggest integrating active learning to reduce the total number of pairs a human needs to label, further mitigating the workload.

Summary updated

Abstract

Preference-based learning aims to align robot task objectives with human values. One of the most common methods to infer human preferences is by pairwise comparisons of robot task trajectories. Traditional comparison-based preference labeling systems seldom support labelers to digest and identify critical differences between complex trajectories recorded in videos. Our formative study (N = 12) suggests that individuals may overlook non-salient task features and establish biased preference criteria during their preference elicitation process because of partial observations. In addition, they may experience mental fatigue when given many pairs to compare, causing their label quality to deteriorate. To mitigate these issues, we propose FARPLS, a Feature-Augmented Robot trajectory Preference Labeling System. FARPLS highlights potential outliers in a wide variety of task features that matter to humans and extracts the corresponding video keyframes for easy review and comparison. It also dynamically adjusts the labeling order according to users' familiarities, difficulties of the trajectory pair, and level of disagreements. At the same time, the system monitors labelers' consistency and provides feedback on labeling progress to keep labelers engaged. A between-subjects study (N = 42, 105 pairs of robot pick-and-place trajectories per person) shows that FARPLS can help users establish preference criteria more easily and notice more relevant details in the presented trajectories than the conventional interface. FARPLS also improves labeling consistency and engagement, mitigating challenges in preference elicitation without raising cognitive loads significantly
BibTeX
@misc{lyu2024farpls,
  author = {Hanfang Lyu and Yuanchen Bai and Xin Liang and Ujaan Das and Chuhan Shi and Leiliang Gong and Yingchi Li and Mingfei Sun and Ming Ge and Xiaojuan Ma},
  title = {{FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers' Preference Elicitation}},
  year = {2024},
  eprint = {2403.06267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.06267},
}
Figure