Enhancing Zero-Shot VLM Reward Models Through Structure-Aware Fine-Tuning

Pyrros Koussios, Chenhao Li, Xin Chen, Andreas KrauseSubmission to ICLR 2026 · Rejected

TL;DR

Foundation Vision-Language Models (VLMs) can serve as zero-shot reward models for reinforcement learning via image-text similarity, but their reward signals are notoriously noisy and brittle. This paper introduces Structure-Aware Fine-Tuning (SAFT), a lightweight, online LoRA-adaptation method that denoises VLM reward landscapes using self-supervised structural priors (such as spatial invariances and Lipschitz continuity) without requiring any ground-truth rewards or continuous human labels.

What Problem Does This Paper Address?

Designing dense, well-shaped reward functions in reinforcement learning (RL) is notoriously difficult. While using pretrained VLMs (like CLIP) to score observations against textual goal descriptions (e.g., "the red and blue object are in the same place") bypasses manual reward engineering, off-the-shelf VLMs often fail in practice. They lack fine-grained 3D spatial grounding, produce noisy and inconsistent reward landscapes, and can collapse under minor visual perturbations.

Standard remedies like Preference-based RL (PbRL) or Reinforcement Learning from Human Feedback (RLHF) alleviate this, but demand hundreds to thousands of expensive, ongoing human annotations. SAFT tackles how to stabilize and align VLM reward models online during RL training using domain-level structural symmetries rather than human feedback.

Methods

SAFT freezes the base VLM and attaches lightweight LoRA adapters (only ~180k parameters, or 0.2% of the model) to the image encoder. During RL rollouts, these adapters are updated online using one of two self-supervised auxiliary objectives:

  1. Contrastive Augmentation Loss (CAL) — Invariance: Exploits environment symmetries where different visual states share semantic equivalence. It minimizes variance among positive task-preserving transformations (e.g., horizontal flips in CartPole, rotation in reaching tasks) while maximizing variance against negative states (e.g., vertical flips or temporally distant frames): $$\mathcal{L}_{\text{CAL}} = \beta \, \text{std}\big(r(o_t), r(o_t^+)\big) - (1 - \beta) \, \text{std}\big(r(o_t), r(o_t^-)\big)$$
  2. Reward Lipschitz Regularization (RLR) — Proportionality: Encourages smooth, locally consistent reward landscapes when proximity in state space correlates with goal progress. Over a sliding observation window $W$, it penalizes disproportionate spikes between normalized observation distances and normalized reward differences: $$\mathcal{L}_{\text{RLR}} = \sum_{i=1}^W \sum_{j \neq i} \left( \frac{\|\tilde{r}_i - \tilde{r}_j\|_2}{\|\tilde{o}_i - \tilde{o}_j\|_2 + \epsilon} - 1 \right)^2$$

Neither loss requires ground-truth rewards or human labels during online adaptation.

Datasets & Environments

The vision backbone is an 86M parameter ViT-B-16 from OpenCLIP pretrained on LAION-2B.

Evaluation is conducted across four continuous control and robotic manipulation environments:

  • Classic Control (Gymnasium): CartPole (supports both CAL and RLR) and MountainCar (uses RLR with an added action penalty).
  • Robotic Manipulation (Isaac Lab / Orbit):
  • Reach (Franka arm): End-effector position matching with top-down tracking; uses CAL with rotational augmentations ($\pm 10^\circ$ to $\pm 40^\circ$).
  • ReposeCube (Allegro hand): In-hand manipulation requiring a specific cube face oriented upward; uses CAL with rotational augmentations and a $180^\circ$ opposing-view hard negative.

Key Insights and Ideas

  • Self-supervised priors beat naive smoothing: Simply averaging rewards over positive augmentations at inference does not match SAFT (Figure S10). Updating model representations via negative contrastive pairs actively denoises the latent geometry and prevents representational collapse.
  • Rescuing weak and small models: A standout finding is the "critical regime" (Figure 8, S9). Off-the-shelf ViT-B-16 models often completely fail on tasks like CartPole. SAFT stabilizes the reward signal enough to bridge the gap, enabling a small 86M-parameter model to succeed where it previously collapsed.
  • Combining priors doesn't always stack: In CartPole, testing CAL and RLR simultaneously showed that CAL performed best alone; adding RLR provided no cumulative benefit over CAL (Figure S13).
  • Human-in-the-loop elimination: A one-time specification of domain augmentations (e.g., rotational invariance) replaces thousands of interactive human pairwise comparison queries.

Key Results

  • Closer Ground-Truth Alignment: Measured via Equivalent-Policy Invariant Comparison (EPIC) distance (lower is better):
  • CartPole: Dropped from 0.4306 $\rightarrow$ 0.2782 ($\pm 0.0205$)
  • MountainCar: Dropped from 0.3106 $\rightarrow$ 0.2330 ($\pm 0.0074$)
  • Reach: Dropped from 0.3443 $\rightarrow$ 0.3129 ($\pm 0.0069$)
  • ReposeCube: Dropped from 0.2963 $\rightarrow$ 0.2605 ($\pm 0.0098$)
  • Sample Efficiency: Across all benchmarks, SAFT demonstrated significantly faster policy convergence and higher asymptotic returns compared to frozen base VLMs and goal-baseline regularized baselines (Figure 6).
  • Annotator Query Savings: Compared to an equivalent Bradley-Terry PbRL baseline, SAFT saved:
  • $5,608 \pm 309$ binary queries on CartPole
  • $670 \pm 27$ queries on MountainCar
  • $13,516 \pm 1,682$ queries on Reach
  • $8,755 \pm 1,711$ queries on ReposeCube

Limitations and Future Work

  • Manual Prior Selection: Augmentations for CAL, choices between CAL and RLR, and the temporal window $W$ for RLR are currently engineered per environment rather than learned automatically.
  • L2 Distance in Observation Space: RLR relies on Euclidean distance in observation space, which works well for state-like features or clean inputs but degrades in high-dimensional, visually noisy scenes.
  • Frozen Text Encoder: Only the vision encoder's LoRA adapters are updated; the text encoder remains completely frozen. The authors suggest future extensions could fine-tune text embeddings using contrastive goal descriptions and prompt negations.
  • Failure Under Complete Misalignment: If a zero-shot model completely lacks the basic perceptual capability to detect task-relevant objects, SAFT cannot recover the policy, as it relies on refining an existing (even if noisy) signal.

Summary updated

Abstract

Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as zero-shot reward models, computing text–observation similarity to bypass manual reward engineering. Although promising, these rewards are noisy, brittle, and misaligned with ground-truth objectives. We introduce Structure-Aware Fine-Tuning (SAFT), a lightweight, LoRA-based method that adapts frozen VLM reward models online using simple structural priors. SAFT enforces invariances and proportionality in the reward signal via augmentations and auxiliary losses, yielding smoother and more consistent reward landscapes. Experiments across classic control and robotic manipulation tasks show faster policy convergence, substantially improved alignment with ground-truth rewards, and elimination of the extensive human annotation effort that Preference-based Reinforcement Learning (PbRL) would otherwise require. These results establish structure-aware fine-tuning as a simple path toward stable, text-conditioned reinforcement learning.
BibTeX
@misc{koussios2026enhancing,
  author = {Pyrros Koussios and Chenhao Li and Xin Chen and Andreas Krause},
  title = {{Enhancing Zero-Shot VLM Reward Models Through Structure-Aware Fine-Tuning}},
  year = {2026},
  eprint = {2608.03875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.03875},
}
Figure