TL;DR
FARPLS is a feature-augmented labeling system designed to help humans provide high-quality preference data for training robots. By automatically highlighting key trajectory features and dynamically ordering tasks, it significantly improves labeling consistency and user engagement without increasing cognitive load.
What Problem Does This Paper Address?
Teaching robots to perform tasks that align with human values often uses Reinforcement Learning from Human Preferences (RLHP). In this setup, humans watch pairs of robot videos and select which one they prefer. However, robot trajectories are complex and high-dimensional.
Non-expert labelers often struggle with three main challenges:
- Difficulty in forming criteria: It's hard to know what "good" looks like without seeing a wide range of examples first.
- Overlooking details: Subtle but critical events, like a minor collision or a slightly unstable grasp, are easily missed in a standard video player.
- Mental fatigue: Comparing hundreds of similar-looking videos is tedious, leading to inconsistent labels and "noisy" data that degrades robot training.
Methods
The authors developed FARPLS (Feature-Augmented Robot trajectory Preference Labeling System) to bridge the gap between raw video data and human sense-making. The system's architecture is built around three core design requirements:
- Feature Augmentation & Visualization: The system automatically extracts 11 features across three categories: Safety (e.g., collisions, contact force), Efficiency (e.g., speed, path length), and Task Quality (e.g., smoothness, grasp position). It visualizes these as distribution charts, highlighting "outliers" to draw the labeler's attention to distinctive characteristics.
- Keyframe Extraction: FARPLS identifies critical moments (like the exact frame of a collision) and provides "jump-to" buttons, allowing labelers to quickly review and compare specific events across videos.
- Dynamic Prompting Strategy: Instead of random pairs, the system initially presents high-variance pairs to help users establish their internal preference criteria. It then balances difficulty and novelty based on how many "clusters" of trajectories the user has seen.
- Consistency Monitoring: The system periodically re-prompts previously labeled pairs to check for consistency, providing real-time feedback and encouraging messages to keep users focused.
Datasets
The researchers introduced PickPlaceCans, a dataset based on the robosuite simulation framework. It consists of 636 successful pick-and-place trajectories, including proficient human demonstrations, multi-human demonstrations, and machine-generated paths. Each trajectory is augmented with the 11 extracted features and corresponding keyframes.
Key Insights and Ideas
- The Dunning-Kruger Effect in Data Labeling: A major finding was that labelers using standard interfaces were just as confident as those using FARPLS, yet their actual labeling consistency was significantly lower. Humans often don't realize what details they are missing until a system explicitly points them out.
- Criteria Formation as a "Cold Start" Problem: Labelers need a "calibration" phase. By showing diverse, high-contrast examples early on, FARPLS helps users build a more robust and stable set of rules for what makes one trajectory better than another.
- Engagement Over Speed: Interestingly, FARPLS users took about 40% longer to complete their tasks. However, they reported higher engagement and lower boredom. The extra time wasn't "wasted"; it was spent in active analysis rather than passive viewing.
Key Results
The system was evaluated in a between-subjects study with 42 participants, each labeling 105 pairs of trajectories.
- Higher Consistency: The FARPLS group achieved a mean consistency score of 0.89, significantly higher than the 0.79 achieved by the baseline group ($p < 0.01$).
- Improved Sense-making: Subjective ratings for "feature support" (5.76 vs 4.10) and "comparison support" (5.86 vs 4.76) were significantly higher for FARPLS.
- Stable Cognitive Load: Despite the additional information (charts, buttons, feedback), users did not report a significant increase in mental effort compared to the simple baseline interface.
Limitations and Future Work
While promising, the study has a few limitations:
- Task Complexity: The evaluation was limited to a relatively simple pick-and-place task. Future work needs to test if these benefits hold for multi-step or collaborative human-robot tasks.
- Downstream Validation: The study focused on label quality (consistency). The next step is to train actual reward models on this data to see if the improved labels lead to measurably better robot performance.
- Algorithmic Integration: The authors suggest integrating active learning to reduce the total number of pairs a human needs to label, further mitigating the workload.
Summary updated