VICTR: Value-Informed In-Context Learning for Vision-Language-Action Models

Luca Macesanu, Tianluo Zhang, Aditya Patil, Archit Sharma, Kevin Angers, Christopher M. Clark, Chen Feng, Juexiao ZhangarXiv 2026 · arXiv

TL;DR

VICTR introduces a novel framework for robot in-context learning that retrieves demonstration snippets based on task progress rather than just visual similarity. By using a learned "In-Context Value Function Estimator" to align current observations with corresponding stages in a reference video, VICTR more than doubles the success rate on unseen, out-of-domain manipulation tasks compared to standard baselines.

What Problem Does This Paper Address?

In-context learning (ICL) has revolutionized how we interact with Large Language Models, allowing them to adapt to new tasks simply by seeing a few examples. Researchers are now trying to bring this "few-shot" adaptability to robotics through Vision-Language-Action (VLA) models.

The standard approach, known as Retrieval-Augmented ICL (RICL), involves giving a robot a "pool" of demonstrations and having it retrieve the most visually similar snippets to guide its current action. However, visual similarity is a noisy signal. A robot arm approaching a cube looks very similar to an arm that has just finished placing a cube. If the retrieval system gets these stages mixed up, the robot receives the wrong "advice" for its next move. Furthermore, visual retrieval is easily distracted by changes in lighting, background, or the presence of new objects that weren't in the original demonstrations.

Methods

The authors propose VICTR (Value-Informed In-Context Retrieval), which replaces visual similarity with a more robust metric: task-relative progress.

1. In-Context Value Function Estimator (IC-VFE)

The heart of VICTR is the IC-VFE. Unlike traditional value functions that are hard-coded for specific tasks, IC-VFE is a generalist. It takes three inputs:

  • A reference demonstration video (the "shortest demo" in the pool, serving as a clean expert execution).
  • A language instruction.
  • The current observation.

It outputs a progress estimate between 0 (start) and 1 (success). Because it "looks" at a reference video to understand what progress means for that specific task, it can estimate progress even for tasks it has never seen during training.

2. The VICTR Framework

At inference time, the system:

  1. Uses IC-VFE to estimate the progress of the current state.
  2. Searches the demonstration pool for a chunk that has the most similar progress value.
  3. Feeds this "progress-aligned" chunk as context into a pretrained VLA policy ($\pi_{0.5}$) to generate the next action.
VICTR method overview showing the two-stage process of progress-informed retrieval and action generation.

The IC-VFE is built on a Gemma-270M backbone with a custom "hybrid attention" mechanism that allows it to aggregate information across the reference demonstration while remaining causal with respect to the current observation.

Datasets

The researchers collected a new real-robot dataset using a YOR bimanual robot setup.

  • Training: 28 tasks, 1,180 episodes (~7 hours of data).
  • Evaluation: 16 tasks, split into 8 "In-Domain" (seen) and 8 "Out-of-Domain" (OOD).
  • OOD Factors: The OOD tasks tested generalization across novel objects, new skill compositions (e.g., "stack a cube then hit it"), and semantically similar but unseen tasks.
  • Simulation: They also validated the approach on the LIBERO-100 benchmark.

Key Insights and Ideas

  • Progress is Task-Centric: Visual similarity often over-relies on scene geometry. Progress-based retrieval is more "task-centric," focusing on where the robot is in the execution flow rather than just what the scene looks like.
  • The "Shortest Demo" Heuristic: Using the shortest successful demonstration as a reference is a clever proxy for quality. Shorter trajectories usually contain fewer mistakes or "retries," providing a cleaner signal for progress estimation.
  • Visual Similarity is Brittle: In OOD settings, distractor objects or background changes can easily break visual retrieval. Progress estimation, by conditioning on a reference video of the same task, acts as a functional anchor that ignores these irrelevant visual shifts.
Qualitative comparison showing how visual similarity can retrieve suboptimal chunks compared to the stage-aligned chunks retrieved by IC-VFE.

Key Results

VICTR shows its true strength in Out-of-Domain (OOD) scenarios where the robot must generalize to something new:

  • Success Rate: VICTR achieved a 23.8% success rate on OOD tasks, compared to only 6.2% for a fine-tuned $\pi_{0.5}$ baseline and 15.0% for visual-similarity-based RICL.
  • Task Progress: Even when it didn't fully succeed, VICTR made significantly more progress (55.5%) than the baseline (28.9%).
  • Value Estimation Quality: The IC-VFE outperformed existing online value estimators (like RECAP and Robometer) in correlating with human-annotated progress curves, especially on unseen tasks.
Main results bar chart showing VICTR's superior performance on OOD tasks compared to baselines.

Limitations and Future Work

  • Absolute Performance: While a 4x improvement over the baseline is impressive, a 25% success rate on new tasks is still far from "production-ready" reliability.
  • Context Bottleneck: To fit a full demonstration video into the model's context window, the authors had to use aggressive spatial pooling. This loses fine-grained visual details, which might be critical for high-precision tasks (e.g., threading a needle).
  • Environment Diversity: All real-world evaluations were done in a single physical lab environment. Testing across genuinely novel environments (different rooms, different lighting) remains a future challenge.
  • Human Demonstrations: Currently, the system relies on robot-to-robot demonstrations. Extending this to allow robots to learn in-context from human videos would be a major step forward.

Summary updated

Abstract

Recent advances in robot foundation models have highlighted the potential of in-context learning (ICL) for adapting robot behavior from demonstrations without further fine-tuning. Previous work has used retrieval-based post-training to equip pretrained vision-language-action (VLA) models with the capability for ICL by teaching them to condition action predictions on retrieved visually-relevant demonstration chunks. However, the effectiveness of retrieval hinges on selecting context relevant to the current execution stage, which visual similarity alone may not capture. We introduce VICTR, a framework enabling in-context learning in VLAs via progress-informed context retrieval built around an In-Context Value Function Estimator (IC-VFE). IC-VFE conditions on a reference demonstration video, a language instruction and the current observation to estimate task-relative progress. By conditioning progress estimation on a reference demonstration, IC-VFE can align the current state with a demonstrated execution even for tasks unseen during training. VICTR uses the estimated progress to select demonstration chunks at corresponding stages of execution and supplies them to a $π_{0.5}$ policy post-trained for in-context action generation. On unseen out-of-domain real-world manipulation tasks, VICTR raises the success rate from 6.2% for a fine-tuned $π_{0.5}$ baseline to 23.8% and more than doubles the task progress rate, from 28.9% to 55.5%. Project page with code, data and videos: https://ai4ce.github.io/VICTR/
BibTeX
@misc{macesanu2026victr,
  author = {Luca Macesanu and Tianluo Zhang and Aditya Patil and Archit Sharma and Kevin Angers and Christopher M. Clark and Chen Feng and Juexiao Zhang},
  title = {{VICTR: Value-Informed In-Context Learning for Vision-Language-Action Models}},
  year = {2026},
  eprint = {2611.08698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08698},
}
Figure