A robot that cannot see the handle of a closed drawer, or the meat under a grill lid, has two options. It can guess from what its fixed cameras show, or it can move a camera until the relevant part is visible. Learning the second behavior with reinforcement learning is hard because the reward is sparse: the robot usually learns only at the end whether the whole task succeeded, and it gets no signal about which of its many camera moves helped. This proposal argues that the simple perceptual checkpoints evaluation benchmarks already use, such as "is the target in view?" and "is the target unoccluded and localized?", can fix this credit-assignment problem. They should be used to decide which steps of different attempts get compared with each other, not as extra reward.
The method builds on group relative policy optimization (GRPO), a reinforcement learning method that drops the learned value network. It samples several attempts from the same starting point and scores each attempt relative to the group average. Multi-step extensions of GRPO assign credit to individual steps by comparing actions taken from the identical state in different attempts. With continuous camera poses and randomized scenes, identical states almost never recur. We replace "identical state" with "same perceptual milestone level." Milestone levels are checked automatically from simulator geometry. Within each level, steps are compared on how quickly and reliably they lead to the next level and to task success. The result is a step-level learning signal for active viewpoint control with no critic, no learned judge, and no change to the task objective.
Core Claim
The checkpoints of a hierarchical task taxonomy can serve as approximate shared states for computing group-relative advantages in continuous active perception. Doing so beats both plain trajectory-level GRPO and reward shaping with the same checkpoints. It beats plain GRPO because it gives step-level credit. It beats shaping because it never pays the policy for merely reaching a checkpoint, which removes the incentive to hover on a visible target without finishing the task.
Three lines of prior work point at this gap from different sides. Group-in-Group Policy Optimization for LLM Agent Training computes step-level "micro" advantages by hashing identical environment states across rollouts. The research file notes that this falls back to plain GRPO when exact matches are rare, which is our regime. Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks shows that grouping steps that share a current state but have different histories biases the gradient ("context inconsistency"). It fixes this with groups that share history prefixes of increasing length, weighted adaptively. Both methods supply our machinery, and both run only on text and grid environments.
On the perception side, VIDEOP2R splits a rollout into perception and reasoning segments and rewards perception separately using an expensive LLM judge. That paper motivates us: it shows that outcome-only GRPO suffers advantage collapse and reward hacking. FameMind trains an active frame-requesting video model with outcome-only GRPO over pre-recorded video. That is active perception without physical viewpoints and without intermediate credit.
We differ from all four. Our perceptual checkpoints are geometric and verifiable, and they define comparison groups rather than rewards. The camera moves in a 3D scene rather than selecting frames from a fixed recording.
Research Program
The agent separates looking from acting, following Observe Then Act. That work split a next-best-view agent from a next-best-pose manipulation agent but allowed only a single camera adjustment before acting. Our viewpoint policy may issue a sequence of camera moves before and between manipulation phases. The manipulation policy is pretrained and frozen during the main study. This keeps every effect attributable to viewpoint control. The viewpoint policy is a compact vision model over the current RGB-D view and the task instruction. It outputs one of a discrete set of camera moves (translations, rotations, zoom, and "commit"), so it fits standard GRPO. Discrete moves still accumulate into effectively continuous poses across randomized scenes, so exact state matches stay rare.
The taxonomy has three levels, mirroring the perception, relation, and composition tiers common in hierarchical benchmarks:
- Level 1 is reached when the task-relevant object occupies more than a threshold fraction of the image, computed from the simulator's segmentation mask.
- Level 2 is reached when the task-relevant part (handle, lid edge, peg hole) is unoccluded above a threshold and the frozen manipulation policy's target estimate is within a pose-error tolerance.
- Level 3 is task success.
Levels are monotone within an episode. Every check is a rendering or geometry query, not a model judgment.
For each start state we sample $G$ rollouts. A step $t$ in rollout $i$ belongs to group $\mathcal{G}_{\ell,b}$ if the highest level reached so far is $\ell$ and the elapsed step count falls in time bin $b$. The time bin is a coarse history-length proxy borrowed from the hierarchical grouping idea, which limits context inconsistency. The step advantage is
$$\hat{A}_{i,t} = \hat{A}^{\text{traj}}_i + \lambda \, \frac{y_{i,t} - \text{mean}_{(j,s)\in\mathcal{G}_{\ell,b}}\, y_{j,s}}{\text{std}_{(j,s)\in\mathcal{G}_{\ell,b}}\, y_{j,s} + \epsilon}.$$
The symbols are:
- $\hat{A}^{\text{traj}}_i$ is the ordinary GRPO advantage of rollout $i$'s terminal return (success minus a per-step time cost).
- $\lambda$ weights the step term.
- $y_{i,t}$ is the step's continuation outcome. It is the terminal return when terminal returns vary within the group. When all rollouts in the group fail, so the terminal term carries no signal, it is the negative number of steps until level $\ell+1$ is reached, capped at the horizon.
The fallback is what prevents the all-failure collapse that VIDEOP2R describes. It never adds reward for being at a level. It only ranks steps already at the same level by how fast they progress.
The same three levels are reported as evaluation metrics, as fractions of episodes reaching each level and the steps taken to reach it. This closes the gap the research file identifies, where taxonomies are used only to diagnose models after training and never to train them.
Baselines
Fixed-camera reference. The do-nothing reference is the frozen manipulation policy with fixed cameras: the front camera and the full front, overhead, and wrist combination. Observe Then Act reports its active agent improved success by 26% and efficiency by 28.2% over the C2FARM-plus-front-camera baseline on its occluded RLBench tasks. These are relative figures, so we re-run the fixed-camera reference on our task set rather than quoting an absolute number.
Strongest existing method. The strongest view-planning competitor is TAVP from Learning to See and Act. It renders task-optimal virtual views from a fused point cloud and reports an 86.6% average success rate on RLBench versus 81.6% for ARP, and 67.0% on the out-of-distribution occlusion suite RLBench-OG. It is the strongest because it already achieves task-aware viewing without moving a camera. We run the released recipe on our exact task subset, with the same frozen manipulation policy where possible and the same number of environment interactions. Beating it means higher success on the deep-occlusion subset, where no fixed sensor sees the critical part, without losing on the remaining tasks. Since TAVP's 86.6% is on its own task mix, our comparison uses only our re-run numbers.
Active-agent baselines. We include the original Observe Then Act actor-critic agent with its entropy-reduction and interaction rewards. We also run PPO with a learned critic on our multi-step viewpoint policy.
Critic-free baselines and ablations. These isolate the key ingredient:
- plain trajectory-level GRPO with terminal reward, the FameMind-style recipe moved to physical viewpoints;
- GiGPO with camera poses voxelized into hashable anchors, run at three voxel sizes;
- HGPO with the same voxelized anchors;
- trajectory GRPO whose reward adds bonuses for reaching each milestone level, the reward-shaping alternative.
All critic-free methods share the group size, rollout budget, learning-rate sweep, and number of seeds. No reported result was found on Lacuna for GRPO-trained physical viewpoint selection on RLBench, so the first three critic-free numbers will be new. For calibration on text agents, GiGPO reports more than 12% improvement over GRPO on ALFWorld, and HGPO reports 95.96% ALFWorld success.
Why not just…?
Why not just add the milestones as shaped rewards? This is the most serious alternative, and it is the central ablation. Shaping changes what is optimal. A policy paid for Level 1 can learn to keep the object framed and never commit. The VIDEOP2R finding that process-agnostic rewards invite reward hacking suggests intermediate rewards need care in both directions. We will measure success, the rate of episodes that reach Level 2 but time out, and sensitivity to the bonus magnitude. If tuned shaping matches milestone grouping in success and hovering rate, we will report that grouping offers no advantage beyond needing no bonus tuning.
Why not just discretize camera poses and use GiGPO or HGPO directly? Coarse voxels create matches but merge states that differ in what is visible. Fine voxels keep states distinct but rarely produce matches. The research file notes that GiGPO falls back to macro GRPO when anchors are rare. We sweep voxel sizes and log the fraction of steps that land in a group of size at least two. Milestone grouping should give both a higher match rate and better success. If some voxel size dominates, the milestone construction is unnecessary.
Why not just render virtual views, as TAVP does, and skip camera motion? Virtual views cannot show geometry no fixed sensor captured. Controlled Recognition Bounds argues that passive sensing cannot guarantee recognition under permanent occlusion. We therefore split results into tasks with partial versus total occlusion from the fixed cameras. If TAVP wins on both, physical active sensing is not worth its latency at this scale, and we will say so.
Why not just use PPO with a critic? For a compact viewpoint policy, the memory argument against critics is weaker than for large models, so PPO is a fair competitor here. The PPO baseline settles whether critic-free grouping is preferable on stability and sample efficiency, not merely on memory.
Proposed Experiments
Controlled check: Cartoon Flatland. The first experiment uses Cartoon Flatland from Controlled Recognition Bounds, where regret against an oracle is measurable. There, the entropy-maximizing policy scores regret 1.27, the variance heuristic 1.44, and direct motion toward the target 1.96. Levels are "target in view" and "target identified." Milestone-grouped GRPO should approach the entropy policy's regret without being given an entropy objective. Plain GRPO should lag most on the hardest instances, where all-failure groups are common.
Main test: occluded RLBench. The main test uses the occluded RLBench tasks of RLBench that Observe Then Act selected, including meat-on-grill and drawer tasks, plus RLBench-OG for distribution shift. We report task success, the three level-reach rates, steps per level, and the fraction of zero-gradient groups during training. The decisive checks follow from the claim:
- Milestone grouping must beat trajectory GRPO by the largest margin on tasks where trajectory GRPO's zero-gradient fraction is highest.
- It must match or beat milestone shaping while showing a lower hover rate.
- It must beat the best voxelized GiGPO and HGPO at equal rollout budget.
- An ablation removing the time bin should increase the gradient-direction disagreement between levels, which tests the context-inconsistency concern from HGPO.
Finally, unfreezing the manipulation policy for a short joint phase tests whether level-wise credit still helps when acting and looking are trained together.
References
- Group-in-Group Policy Optimization for LLM Agent Training
- Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
- VIDEOP2R: Video Understanding from Perception to Reasoning
- FameMind: Frame-Interleaved Video Reasoning via Reinforcement Learning
- Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation
- Learning to See and Act: Task-Aware View Planning for Robotic Manipulation
- Controlled Recognition Bounds for Visual Learning and Exploration
- RLBench: The Robot Learning Benchmark & Learning Environment.