Uncertainty-Aware Deployment of Pre-trained Language-Conditioned Imitation Learning Policies

Bo Wu, Bruce D. Lee, Kostas Daniilidis, Bernadette Bucher, Nikolai Matni

Abstract

Large-scale robotic "foundation models" trained on diverse datasets have shown significant promise in generalization across tasks and platforms. However, these models often exhibit overconfidence, leading to brittle performance and failures when deployed in novel environments or in the presence of distractors. This paper introduces a simple, training-free deployment strategy designed to mitigate these issues. By combining temperature scaling for post-hoc calibration with a neighborhood-based action selection mechanism, the proposed method improves success rates and robustness across multiple state-of-the-art imitation learning architectures without requiring expensive model re-training.

Introduction

The emergence of language-conditioned imitation learning policies, such as Perceiver-Actor (PerAct), RVT, and CLIPort, has enabled robots to perform complex manipulation tasks from natural language instructions. Despite their capabilities, these models frequently lack well-calibrated uncertainty. In practice, a model may assign a high probability to an incorrect action, such as grasping an object at an unstable edge or being distracted by irrelevant visual noise. This lack of reliability poses a significant barrier to real-world deployment. This research addresses the challenge by making pre-trained models "aware" of their own uncertainty during the inference phase.

Methodology

The authors propose a two-step, post-hoc modification to the standard deployment pipeline that requires no changes to the underlying model weights.

1. Target Task Calibration

To address miscalibration, the researchers employ temperature scaling. This technique involves finding a scaling factor ($T$) that adjusts the model's output logits. By utilizing a small set of expert demonstrations (as few as 25) from the specific target task, the model's predicted confidence is aligned with the actual likelihood of action success. This ensures that the probability scores generated by the model are representative of its true performance accuracy.

2. Uncertainty-Aware Action Selection

Standard deployment typically follows a "greedy" approach, selecting the single action with the highest confidence score. This paper proposes a Neighborhood-based Action Selection method. Instead of relying on an isolated peak, the algorithm calculates a new score for candidate actions by summing the confidence scores of all similar actions within a defined radius ($ au$).

This approach favors regions of the action space where the model is consistently confident. By aggregating local confidence, the policy becomes more conservative and precise—for instance, preferring the center of an object where many neighboring points are also valid, rather than an edge where a slight deviation would lead to failure.

Experimental Evaluation

The methodology was evaluated across three state-of-the-art models and two primary benchmarks:

  • RLBench: Used to test Perceiver-Actor (PerAct) and RVT models across 18 manipulation tasks, including "open drawer" and "push buttons."
  • Ravens Benchmark: Used to evaluate CLIPort on tasks such as block stacking and kit assembly.
  • Distractor Robustness: The authors introduced modified Ravens tasks containing irrelevant objects to evaluate the models' ability to ignore out-of-distribution noise.

Key Insights

Calibration and Model Architecture

The study revealed that calibration needs vary by model. CLIPort, which was trained on 1,000 demonstrations per task, was found to be relatively well-calibrated out of the box. In contrast, PerAct and RVT, trained on 100 demonstrations per task, exhibited significant overconfidence. For these models, temperature scaling was essential, drastically reducing the Expected Calibration Error (ECE). For example, in the "Light Bulb" task, PerAct's ECE dropped from 0.64 to 0.07.

Robustness to Distractors

Visual distractors often create isolated "spikes" of high confidence in a model's internal heatmap. Because these spikes lack a supportive neighborhood of high-confidence actions, the proposed neighborhood-summing algorithm effectively filters them out. In the "stack-block" task, the baseline CLIPort model's performance dropped from ~95% to 79.6% when distractors were present. With uncertainty-aware selection, performance was restored to 97.1%.

Results

The proposed deployment strategy yielded consistent improvements:

  • RLBench Success Rates: PerAct saw an increase from 38.2% to 41.4%, while RVT improved from 60.2% to 62.3%.
  • Precision: The neighborhood aggregation naturally encouraged more robust grasping and placement by favoring central, high-density confidence regions.

Conclusion and Future Work

This work demonstrates that the reliability of robotic foundation models can be significantly enhanced through simple post-hoc calibration and smarter action selection. While the method cannot correct a model that has failed to learn the task entirely, it provides a vital layer of robustness for deployment. Future research may explore learning the similarity metrics ($d$) and thresholds ($ au$) directly from data, or extending these uncertainty-quantification techniques to the domain of dynamics forecasting.

Summary updated

Abstract

Large-scale robotic policies trained on data from diverse tasks and robotic platforms hold great promise for enabling general-purpose robots; however, reliable generalization to new environment conditions remains a major challenge. Toward addressing this challenge, we propose a novel approach for uncertainty-aware deployment of pre-trained language-conditioned imitation learning agents. Specifically, we use temperature scaling to calibrate these models and exploit the calibrated model to make uncertainty-aware decisions by aggregating the local information of candidate actions. We implement our approach in simulation using three such pre-trained models, and showcase its potential to significantly enhance task completion rates. The accompanying code is accessible at the link: https://github.com/BobWu1998/uncertainty_quant_all.git
BibTeX
@inproceedings{wu2024uncertainty,
  author = {Bo Wu and Bruce D. Lee and Kostas Daniilidis and Bernadette Bucher and Nikolai Matni},
  title = {{Uncertainty-Aware Deployment of Pre-trained Language-Conditioned Imitation Learning Policies}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802849},
}
Figure