Grana: Compositional Reward Modeling for Long-Horizon Robot Manipulation

Zi-ang Cao, Kaiyuan Zheng, Yinzhen Xu, Haonan Duan, Jiasheng Gu, Fengyuan Hu, Nan Zhu, Kun Lei, Kevin Lin, Ruijie Zheng, Shenyuan Gao, Fangqi Zhu, Zhe Zhang, Radha Poovendran, Linxi Fan, Guanzhi Wang, Yuke ZhuarXiv 2026 · arXiv

In the world of robot manipulation, "long-horizon" tasks—like assembling a model airplane or setting a table—are the final frontier. While modern robots are getting better at simple "pick and place" actions, they often lose their way during multi-minute sequences. A key reason is the reward model: the internal "scorekeeper" that tells the robot how well it's doing.

Most current reward models are either too specific (trained for one task) or too vague (giving a single score for a complex video). Enter Grana, a new vision-language reward model from NVIDIA and university partners that uses a "divide and conquer" approach to track progress with unprecedented accuracy.

Overview of the Grana framework, showing short-horizon pretraining, subtask decomposition, and various downstream applications like offline RL and behavior cloning.

TL;DR

Grana is a vision-language reward model that decomposes long-horizon robot tasks into open-vocabulary subtasks to track continuous progress. By combining language-conditioned subtask selection with a novel "action-quality" supervision protocol, it achieves superior generalization and significantly boosts the success rates of downstream robot policies.

What Problem Does This Paper Address?

Tracking progress in long-horizon manipulation is notoriously difficult for two reasons:

  1. Generalization: Most models rely on fixed action vocabularies or task-specific classifiers. If you want the robot to do a new task, you often have to retrain the reward model from scratch.
  2. Imperfect Execution: Robots mess up. They drop screws, misalign parts, and have to try again. Simple "elapsed time" metrics fail here because time keeps moving forward even when progress has stalled or reversed.

The authors argue that a good reward model should be able to identify which subtask is currently being attempted and how much of that specific subtask is complete, all while being flexible enough to handle new tasks described in plain English.

Methods

Grana’s core innovation is its compositional approach to reward modeling.

1. Compositional Progress Prediction

Instead of predicting one global score, Grana uses a shared Vision-Language Model (VLM) backbone (Qwen3.5-4B) to:

  • Select the Subtask: Given a list of language descriptions (e.g., "Place top wing," "Insert the screw"), it identifies which one is currently unfinished.
  • Estimate Local Progress: It predicts a continuous score from 0 to 1 for that specific subtask.
  • Compose Global Progress: Total progress is calculated by adding the credit for all previously completed subtasks to the current local progress.
The Grana architecture: a VLM processes multi-view observations to select a subtask and predict its local progress, which is then composed into a global task progress estimate.

2. Structured Progress Supervision

To train this, the authors moved away from asking humans to provide arbitrary numerical scores. Instead, they developed a protocol where annotators mark:

  • Subtask Boundaries: When one step ends and the next begins.
  • Action Quality (AQ): Whether the robot's current actions are Nominal (good), Suboptimal (struggling/misaligned), or Regressed (failure/setback).

These sparse labels are deterministically converted into a dense, "sawtooth" progress curve that drops to zero upon failure, providing a clear signal for the robot to learn recovery behaviors.

Comparison of different annotation methods. Grana's "Subtask + AQ" approach captures failures and recoveries that simpler methods miss.

Datasets

  • Pretraining: A massive corpus of 200 diverse manipulation tasks, mostly short-horizon (e.g., opening a toolbox, folding a shirt).
  • Zero-Shot Evaluation: 50 held-out tasks to test generalization.
  • Long-Horizon Benchmarks: Five complex tasks including Airplane Assembly (9 subtasks), Red Car Assembly (10 subtasks), and Pupgo Car Assembly (up to 18 subtasks).
  • Public Benchmark: Evaluated on the RBM-EVAL-OOD trajectory ranking task.

Key Insights and Ideas

  • Language is the Universal Interface: By conditioning on open-vocabulary descriptions, Grana can be applied to entirely new tasks without changing its architecture. You just give it a new list of strings.
  • Short-Task Pretraining Transfers to Long-Horizon: One of the paper's most important findings is that learning "what progress looks like" on 200 simple tasks significantly improves the model's ability to adapt to complex, multi-minute tasks.
  • Decomposition Beats Monoliths: Predicting progress for a 10-second subtask is a much easier learning problem than predicting it for a 5-minute assembly. The compositional logic handles the "long-term memory" so the neural network doesn't have to.

Key Results

  • Zero-Shot Superiority: Grana achieved a 20.7% lower progress error (RMSE) than the strongest baseline (RoboMeter) on 50 unseen tasks without any fine-tuning.
  • Data Efficiency: In long-horizon adaptation, Grana trained on just 15 trajectories outperformed baselines trained on 100 trajectories.
  • Policy Boost: Using Grana's rewards for offline Reinforcement Learning led to a success rate of 65-67% on held-out tasks, compared to ~40-50% for baselines.
  • Test-Time Planning: When used to guide a "Model Predictive Control" (MPC) planner on a tractor assembly task, success jumped from 2/10 to 9/10.
Figure 1: Overview of Grana. Left: diverse short-horizon tasks for pretraining. Center: subtask descriptions condition a progress predictor, and sparse annotations define dense progress supervision. Each disc in the granum represents a subtask; the offset disc marks the selected subtask. Right: offline RL, progress-weighted cloning after long-task adaptation, and cross-embodiment adaptation.

Limitations and Future Work

The authors note that their current composition logic assumes that once a subtask is finished, it stays finished. If a robot accidentally knocks over a previously placed wing while trying to insert a screw, the model might not automatically "un-credit" that progress unless the subtask selection logic is specifically designed to loop back.

Future work could involve more dynamic subtask graphs that allow for arbitrary transitions and even more complex "recovery" logic where the model can suggest which previous step needs to be redone.

Summary updated

Abstract

Robot reward modeling faces two challenges: generalizing across tasks and accurately tracking progress in long-horizon manipulation. General-purpose reward models demonstrate broad task transfer, but transfer from short-task pretraining to long-horizon tasks remains underexplored. Long-horizon progress models often rely on task-specific subtask classifiers or fixed action vocabularies, limiting the reuse of learned progress knowledge when adapting to new tasks. To address these challenges, we introduce Grana, a vision-language robot reward model that learns to identify unfinished subtasks and estimate their continuous progress using open-vocabulary subtask descriptions. We train a language-conditioned model to estimate progress in short-horizon tasks and individual subtasks of long-horizon tasks. For long-horizon tasks, we combine the selected subtask's predicted progress with credit for preceding completed subtasks to estimate whole-task progress. General progress prediction requires training the model across diverse tasks with progress supervision that accounts for imperfect execution. Elapsed-time proxies can misrepresent progress when relative subtask durations vary, or failures reverse local progress. We therefore introduce an annotation protocol that converts sparse subtask boundaries and action-quality annotations into dense progress targets, without requiring annotators to assign numerical progress scores. Pretraining with this supervision on 200 diverse, mostly short-horizon manipulation tasks enables zero-shot reward prediction on unseen tasks and improves progress prediction after adaptation to multi-minute, long-horizon tasks. Experiments demonstrate (i) generalizable reward prediction, including adaptation to five long-horizon tasks, (ii) benefits from pretraining and composing subtask progress estimates, and (iii) improved downstream policy performance.
BibTeX
@misc{cao2026grana,
  author = {Zi-ang Cao and Kaiyuan Zheng and Yinzhen Xu and Haonan Duan and Jiasheng Gu and Fengyuan Hu and Nan Zhu and Kun Lei and Kevin Lin and Ruijie Zheng and Shenyuan Gao and Fangqi Zhu and Zhe Zhang and Radha Poovendran and Linxi Fan and Guanzhi Wang and Yuke Zhu},
  title = {{Grana: Compositional Reward Modeling for Long-Horizon Robot Manipulation}},
  year = {2026},
  eprint = {2611.24926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24926},
}
Figure