Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation

Nathaniel Dennler, Stefanos Nikolaidis, Maja J. Mataric

TL;DR

Researchers at USC have developed CLEA (Contrastive Learning from Exploratory Actions), a framework that learns what users value in robot behavior by watching how they explore a robot's capabilities. Instead of forcing users to perform tedious labeling tasks, CLEA harvests the "free" data generated when users browse through behavior options, outperforming standard self-supervised methods in predicting human preferences across visual, auditory, and kinetic modalities.

The Challenge of Preference Learning

To make robots truly useful, they need to understand human preferences, often modeled as a "reward function." This function relies on features—numerical representations of behavior like "how fast is it moving?" or "how 'dog-like' does this look?"

Traditionally, these features come from three sources, each with significant drawbacks:

  1. Hand-crafted features: Limited by engineer intuition and often miss subtle human nuances.
  2. Self-supervised features: (e.g., Autoencoders) Good at capturing physical data but often ignore the semantic qualities humans actually care about.
  3. Human-labeled features: Accurate but require "proxy tasks" (e.g., rating 100 random videos), which are tedious and discourage users.

CLEA asks a fundamental question: Can we learn user-aligned features from the natural interactions users already have when they first get a robot?

The CLEA Framework

The core innovation of CLEA is treating a user’s "exploratory search" as an implicit labeling process. When a user customizes a robot, they naturally click on behaviors that look interesting or relevant (Explored Data) and skip over those that don't (Ignored Data).

The CLEA pipeline: from exploratory search to learned features and personalized customization.

CLEA uses a Symmetric Triplet Loss to train a feature-generating network:

  • Positive pairs: Two behaviors from the "Explored" set are pushed closer together in the latent space.
  • Negative pairs: An "Explored" behavior is pushed away from an "Ignored" behavior.

By doing this, the model learns a latent space where the distance between behaviors reflects human-perceived similarity rather than just raw pixel or joint-angle similarity. The authors tested this across three modalities: visual (video frames), auditory (spectrograms), and kinetic (joint angles).

Exploratory Search as Labeling

We often think of a user clicking around an interface as "messy" data. CLEA argues that exploration is an intuitive reasoning process. What a user chooses to ignore is just as informative as what they choose to test.

Users naturally categorize robot behaviors into 'Explored' and 'Ignored' sets during initial interaction.

By piggybacking on the "exploratory search" phase—which users perform anyway to see what the robot can do—developers can skip the "proxy task" phase entirely. This provides a "free lunch" for interaction design, where semantic alignment is achieved without additional user burden.

Key Results and Insights

The authors evaluated CLEA against several baselines (Random, Pretrained, AE, VAE) using a dataset of 5,192 videos, 867 sounds, and 2,125 head movements collected from 25 training users and 42 evaluation users.

  • Superior Accuracy: CLEA+AE achieved the highest Test Preference Accuracy (TPA) in visual and auditory modalities, proving it is best at predicting which behaviors a new user would prefer.
  • Simplicity and Minimality: CLEA-based features allowed a simple linear reward model to perform as well as complex neural networks, even when the feature space was compressed to just 8 dimensions.
  • Semantic vs. Physical Alignment: While standard self-supervised models (AE/VAE) group behaviors based on visual structure (e.g., "both have a circle"), CLEA groups them based on intent (e.g., "both look happy").
  • Explainability: Top-ranked signals in CLEA space were significantly more similar to "exemplar" behaviors from the training set, facilitating "explanation by example."

Limitations and Future Work

While promising, CLEA faces challenges with summary representations; finding the right way to "summarize" complex behaviors (like quadruped movement) so a user can quickly decide to explore or ignore them remains an open question. Additionally, for kinetic data, standard reconstruction losses sometimes conflict with CLEA's semantic goals—for instance, a "scared" gesture and an "excited" gesture might have similar joint movements but opposite meanings. Balancing these physical and social interpretations is a key area for future research.

Summary updated

Abstract

People have a variety of preferences for how robots behave. To understand and reason about these preferences, robots aim to learn a reward function that describes how aligned robot behaviors are with a user's preferences. Good representations of a robot's behavior can significantly reduce the time and effort required for a user to teach the robot their preferences. Specifying these representations-what “features“ of the robot's behavior matter to users-remains a difficult problem; Features learned from raw data lack semantic meaning and features learned from user data require users to engage in tedious labeling processes. Our key insight is that users tasked with customizing a robot are intrinsically motivated to produce labels through exploratory search; they explore behaviors that they find interesting and ignore behaviors that are irrelevant. To harness this novel data source of exploratory actions, we propose contrastive learning from exploratory actions (CLEA) to learn trajectory features that are aligned with features that users care about. We learned CLEA features from exploratory actions users performed in an open-ended signal design activity $(N=25)$ with a Kuri robot, and evaluated CLEA features through a second user study with a different set of users $(N=42)$. CLEA features outperformed self-supervised features when eliciting user preferences over four metrics: completeness, simplicity, minimality, and explainability.
BibTeX
@misc{dennler2025contrastive,
  author = {Nathaniel Dennler and Stefanos Nikolaidis and Maja J. Mataric},
  title = {{Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation}},
  year = {2025},
  eprint = {2501.01367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.01367},
}
Figure