TL;DR
Rank-Then-Act (RTA) is a two-stage framework that enables agents to learn complex control policies using nothing but expert video demonstrations—no manual reward engineering required. By training a Vision-Language Model (VLM) to recover the temporal order of shuffled video frames and using rank correlation as a reward signal, RTA creates a robust, scale-invariant learning signal that generalizes across diverse tasks and environments.
What Problem Does This Paper Address?
Reinforcement Learning (RL) typically relies on "extrinsic rewards"—numerical signals defined by developers to tell an agent when it’s doing a good job. However, designing these rewards is notoriously difficult and brittle. If the reward is too simple, the agent might "game" the system; if it’s too sparse (only rewarding the final goal), the agent may never find it through random exploration.
While using expert videos as a reference is a popular alternative, current methods struggle with three main issues:
- Trivial Shortcuts: Models often learn that "later frames are better" based on simple visual cues (like a timer on screen) rather than actual task progress.
- Calibration Drift: Absolute "progress scores" are hard to normalize across different tasks or even different attempts at the same task.
- Computational Cost: Using a massive VLM to generate rewards at every single step of an RL simulation is often too slow and expensive.
Methods
RTA splits the problem into two distinct stages:
Stage 1: The Ranker (Offline)
The authors fine-tune a VLM (specifically Qwen2.5-VL-7B) to act as a "progress scorer." They feed it shuffled clips from expert videos where the first frame is "anchored" (fixed at the start). The model must reason about the visual semantics to assign a progress rank to each shuffled frame.
- The Secret Sauce: They use Group Relative Policy Optimization (GRPO) to maximize the Spearman rank correlation between the model’s predicted ranks and the actual ground-truth timestamps. Shuffling the frames forces the model to understand what is happening in the video to determine the order, rather than relying on chronological shortcuts.
Stage 2: The Actor (Online)
The VLM is frozen and used as a reward model for an RL agent (using PPO or DrQv2).
- Correlation as Reward: Instead of asking the VLM for a single scalar "goodness" score, RTA looks at a sliding window of recent frames. It calculates the Spearman correlation between the VLM's predicted ranks and the actual time indices.
- Scale Invariance: Because correlation only cares about the order of events, the reward is naturally bounded between -1 and 1 and is invariant to the absolute scale of the VLM’s internal scoring.
Datasets
The researchers evaluated RTA across both discrete and continuous control domains:
- Discrete (PyBoy): They used human playthroughs of the puzzle game Catrap and the platformer Kirby’s Dream Land.
- Continuous: They used expert trajectories for PointMaze and the MetaWorld robotics suite (tasks like opening doors, pressing buttons, and moving hammers).
- Generalization Data: To test "zero-shot" capabilities, they even trained the scorer on a pool of 70 diverse GameBoy playthroughs scraped from YouTube.
Key Insights and Ideas
- Structure Over Magnitude: The core "aha moment" is that we don't need a perfectly calibrated reward model. We just need a model that understands the sequence of a task. Rank correlation provides a dense, shaped reward that is much more stable than regressing absolute values.
- Anchoring is Essential: By fixing the first frame as a reference point, the model learns to measure progress relative to the start of the task, preventing it from getting confused by local appearance changes.
- Cross-Domain "Super-Scorers": Surprisingly, a scorer trained on a mix of YouTube videos or different game levels can provide effective rewards for a completely new level. This suggests that "temporal ordering" is a generalizable skill that VLMs can leverage across environments.
- The "Goldilocks" Window: The authors found that the size of the sliding window used for the reward is critical. Too small, and the signal is noisy; too large, and the signal becomes "diluted" by averaging.
Key Results
- Superior Performance: In Catrap, RTA achieved a 100% success rate on Level 2 and significantly outperformed prior state-of-the-art video-reward methods (like Rank2Reward and VLM-RM) on more complex levels.
- Kirby Breakthrough: RTA was the only method to achieve a non-zero success rate on the Kirby benchmark without using any environment-provided rewards.
- Robust Continuous Control: On PointMaze-UMaze and MetaWorld tasks, RTA consistently matched or beat baselines, showing that the ordinal progress signal works just as well for robotic manipulation as it does for retro games.
- Efficiency: RTA proved to be more compute-efficient than its competitors, achieving better results in less wall-clock time when compared under a matched budget.
Limitations and Future Work
- Expert Dependency: The method still requires a small set of high-quality expert videos to train the initial scorer.
- VLM Latency: Even with sparse querying (every 15 steps), VLM inference adds overhead. The authors suggest that future work could involve distilling these large scorers into smaller, faster "student" models.
- Non-Monotonic Tasks: While RTA handles some backtracking, tasks that are highly cyclic or require repeating the same visual state multiple times might still confuse a simple rank-based reward. The authors propose multi-scale temporal aggregation or explicit subgoal modeling as potential solutions.
Summary updated