TL;DR
Foundation Vision-Language Models (VLMs) can serve as zero-shot reward models for reinforcement learning via image-text similarity, but their reward signals are notoriously noisy and brittle. This paper introduces Structure-Aware Fine-Tuning (SAFT), a lightweight, online LoRA-adaptation method that denoises VLM reward landscapes using self-supervised structural priors (such as spatial invariances and Lipschitz continuity) without requiring any ground-truth rewards or continuous human labels.
What Problem Does This Paper Address?
Designing dense, well-shaped reward functions in reinforcement learning (RL) is notoriously difficult. While using pretrained VLMs (like CLIP) to score observations against textual goal descriptions (e.g., "the red and blue object are in the same place") bypasses manual reward engineering, off-the-shelf VLMs often fail in practice. They lack fine-grained 3D spatial grounding, produce noisy and inconsistent reward landscapes, and can collapse under minor visual perturbations.
Standard remedies like Preference-based RL (PbRL) or Reinforcement Learning from Human Feedback (RLHF) alleviate this, but demand hundreds to thousands of expensive, ongoing human annotations. SAFT tackles how to stabilize and align VLM reward models online during RL training using domain-level structural symmetries rather than human feedback.
Methods
SAFT freezes the base VLM and attaches lightweight LoRA adapters (only ~180k parameters, or 0.2% of the model) to the image encoder. During RL rollouts, these adapters are updated online using one of two self-supervised auxiliary objectives:
- Contrastive Augmentation Loss (CAL) — Invariance: Exploits environment symmetries where different visual states share semantic equivalence. It minimizes variance among positive task-preserving transformations (e.g., horizontal flips in CartPole, rotation in reaching tasks) while maximizing variance against negative states (e.g., vertical flips or temporally distant frames): $$\mathcal{L}_{\text{CAL}} = \beta \, \text{std}\big(r(o_t), r(o_t^+)\big) - (1 - \beta) \, \text{std}\big(r(o_t), r(o_t^-)\big)$$
- Reward Lipschitz Regularization (RLR) — Proportionality: Encourages smooth, locally consistent reward landscapes when proximity in state space correlates with goal progress. Over a sliding observation window $W$, it penalizes disproportionate spikes between normalized observation distances and normalized reward differences: $$\mathcal{L}_{\text{RLR}} = \sum_{i=1}^W \sum_{j \neq i} \left( \frac{\|\tilde{r}_i - \tilde{r}_j\|_2}{\|\tilde{o}_i - \tilde{o}_j\|_2 + \epsilon} - 1 \right)^2$$
Neither loss requires ground-truth rewards or human labels during online adaptation.
Datasets & Environments
The vision backbone is an 86M parameter ViT-B-16 from OpenCLIP pretrained on LAION-2B.
Evaluation is conducted across four continuous control and robotic manipulation environments:
- Classic Control (Gymnasium): CartPole (supports both CAL and RLR) and MountainCar (uses RLR with an added action penalty).
- Robotic Manipulation (Isaac Lab / Orbit):
- Reach (Franka arm): End-effector position matching with top-down tracking; uses CAL with rotational augmentations ($\pm 10^\circ$ to $\pm 40^\circ$).
- ReposeCube (Allegro hand): In-hand manipulation requiring a specific cube face oriented upward; uses CAL with rotational augmentations and a $180^\circ$ opposing-view hard negative.
Key Insights and Ideas
- Self-supervised priors beat naive smoothing: Simply averaging rewards over positive augmentations at inference does not match SAFT (Figure S10). Updating model representations via negative contrastive pairs actively denoises the latent geometry and prevents representational collapse.
- Rescuing weak and small models: A standout finding is the "critical regime" (Figure 8, S9). Off-the-shelf ViT-B-16 models often completely fail on tasks like CartPole. SAFT stabilizes the reward signal enough to bridge the gap, enabling a small 86M-parameter model to succeed where it previously collapsed.
- Combining priors doesn't always stack: In CartPole, testing CAL and RLR simultaneously showed that CAL performed best alone; adding RLR provided no cumulative benefit over CAL (Figure S13).
- Human-in-the-loop elimination: A one-time specification of domain augmentations (e.g., rotational invariance) replaces thousands of interactive human pairwise comparison queries.
Key Results
- Closer Ground-Truth Alignment: Measured via Equivalent-Policy Invariant Comparison (EPIC) distance (lower is better):
- CartPole: Dropped from 0.4306 $\rightarrow$ 0.2782 ($\pm 0.0205$)
- MountainCar: Dropped from 0.3106 $\rightarrow$ 0.2330 ($\pm 0.0074$)
- Reach: Dropped from 0.3443 $\rightarrow$ 0.3129 ($\pm 0.0069$)
- ReposeCube: Dropped from 0.2963 $\rightarrow$ 0.2605 ($\pm 0.0098$)
- Sample Efficiency: Across all benchmarks, SAFT demonstrated significantly faster policy convergence and higher asymptotic returns compared to frozen base VLMs and goal-baseline regularized baselines (Figure 6).
- Annotator Query Savings: Compared to an equivalent Bradley-Terry PbRL baseline, SAFT saved:
- $5,608 \pm 309$ binary queries on CartPole
- $670 \pm 27$ queries on MountainCar
- $13,516 \pm 1,682$ queries on Reach
- $8,755 \pm 1,711$ queries on ReposeCube
Limitations and Future Work
- Manual Prior Selection: Augmentations for CAL, choices between CAL and RLR, and the temporal window $W$ for RLR are currently engineered per environment rather than learned automatically.
- L2 Distance in Observation Space: RLR relies on Euclidean distance in observation space, which works well for state-like features or clean inputs but degrades in high-dimensional, visually noisy scenes.
- Frozen Text Encoder: Only the vision encoder's LoRA adapters are updated; the text encoder remains completely frozen. The authors suggest future extensions could fine-tune text embeddings using contrastive goal descriptions and prompt negations.
- Failure Under Complete Misalignment: If a zero-shot model completely lacks the basic perceptual capability to detect task-relevant objects, SAFT cannot recover the policy, as it relies on refining an existing (even if noisy) signal.
Summary updated