Paying for Value, Not Pixels: Regret-Optimal Looking Under Self-Occlusion

When a robot arm reaches for an object, its own wrist and fingers often block the camera just before contact. A common response is to give the robot a movable camera and reward it for "seeing more": more visible pixels, lower scene entropy, more decodable motion. A result from reinforcement learning theory suggests this is the wrong currency. Azar et al. showed that an agent does not need to estimate the whole distribution of possible next states. It only needs to estimate the single number that matters, the expected value of the next state. Dropping the full-distribution estimate removed an unnecessary $\sqrt{S}$ factor from the best known learning guarantees (Minimax Regret Bounds for Reinforcement Learning). Here, regret is the total reward lost while learning, compared with an agent that already knows the optimal policy.

This note argues that a camera move is a transition with an enormous number of possible outcomes. A single look can reveal many hidden scene variables at once, so the number of things the robot might see grows exponentially in how many it reveals. An exploration rule that tries to learn what it will see pays for that whole outcome space. A rule that learns only how much the view will change its expected success pays almost nothing for it. We propose to formalize self-occlusion as a structured, tractable decision problem. We will prove that value-variance exploration bonuses avoid the exponential outcome-space cost that distribution-based bonuses incur, and then turn the resulting look-selection rule into a deep RL agent that we test against entropy-driven and sensorimotor-driven active vision on occlusion-heavy manipulation benchmarks.

Core Claim

Self-occlusion has structure that general partial observability lacks. The robot knows its own geometry and its camera calibration. Given a joint configuration, it can therefore compute exactly which parts of the scene are visible. During the approach phase, the occluded scene is roughly static.

Under these two assumptions (a known visibility function and a static scene with noiseless reveals), the occluded manipulation problem reduces exactly to a fully observed MDP over "robot configuration plus what has been revealed so far." That reduction sidesteps the exponential lower bound for general partially observable problems. In the reduced MDP, the only hard transitions are reveals, whose number of possible outcomes $B$ is exponential in the number of features a look uncovers.

We claim two things. First, an optimistic algorithm with Hoeffding-style bonuses, which bound uncertainty over the full next-state distribution, has a leading regret term that grows with $\sqrt{B}$. Bernstein bonuses on the scalar next-state value, combined with the total-variance argument of Azar et al., remove that dependence. Second, the deep analogue of the Bernstein rule should choose camera moves that resolve value-relevant occlusion and ignore occlusion that is visually large but irrelevant to the task. Entropy-reduction and inverse-dynamics rewards cannot make that distinction.

The Model

The setting has three parts:

  1. Fully observed robot state. Let $q_h$ be the joint configuration of the arm and any camera mount at step $h$.
  2. Hidden scene. Let $z \in \mathcal{Z}^d$ be $d$ discrete scene variables, such as an object's pose bin, whether a hole is free, or whether a drawer is occupied. These are fixed within an episode until contact.
  3. Known visibility. Let $\mathrm{vis}(q) \subseteq \{1,\dots,d\}$ be the set of variables visible from configuration $q$. It is computed from robot meshes and camera extrinsics. It is not learned.

After each move, the agent observes $z_i$ for every $i \in \mathrm{vis}(q_{h+1})$. Because $z$ is static and reveals are noiseless, the record of revealed values $m_h$ is a sufficient statistic. The agent's posterior over the unrevealed variables depends only on $m_h$. The problem is therefore an MDP on states $s_h=(q_h, m_h)$, whose transition kernel is unknown only through the unknown prior over $z$.

A move that newly reveals $k$ variables has up to $B=|\mathcal{Z}|^k$ next states. The bound we target has the form

$$\mathrm{Regret}(T) = \widetilde{O}\big(\sqrt{H\,S_r\,A\,T}\big) \quad\text{versus}\quad \widetilde{O}\big(\sqrt{H\,S_r\,B\,A\,T}\big),$$

where $T$ is total steps, $H$ the horizon, $A$ the number of actions, and $S_r$ the number of reachable $(q,m)$ states. The left side is the Bernstein value-concentration rate. The right side is the Hoeffding confidence-set rate. The $\sqrt{B}$ gap is exactly the extra support-size factor that UCBVI-BF removed, reinterpreted as the cost of a look.

Three caveats apply. $S_r$ can still be large. We state as an open conjecture, which we test empirically and do not claim to prove, that merging reveal states with equal optimistic value shrinks $S_r$ to the value-relevant part. Azar et al.'s lower-order terms also require a large-$T$ regime, $T \ge H^3S^3A$, so the asymptotic separation may be invisible at small budgets. Finally, noisy detections break the exact reduction.

Relation to Prior Work

Two theory papers supply the method and bound the scope.

Azar et al. (Minimax Regret Bounds for Reinforcement Learning) supply the core technique: Bernstein bonuses on $PV^*$ plus the law of total variance. Their analysis assumes full observability, and our reduction is what lets us reuse it.

Shi, Liang and Shroff (Designing Near-Optimal Partially Observable Reinforcement Learning) constrain the design. They prove that general partially observable RL with partial state queries needs $\Omega(A^{H-1}/\epsilon^2)$ episodes. Their tractable algorithms scale with the full unqueried space $|\widetilde{S}|^{d-\tilde{d}}$ and assume any feature subset within budget can be queried freely. We differ in two ways. Which features can be queried is fixed by the arm's kinematics. Our bound targets removal of the exponential outcome factor through value concentration, a question their paper explicitly leaves open beyond the dependence on episode count. Related work removes the same $\sqrt{S}$ factor by other routes, in average-reward problems (Zhang et al.), posterior sampling (Agrawal et al.) and policy optimization (Wu et al.). It confirms that the gap is about concentration technique, not algorithm family.

On the manipulation side, several papers motivate the gap by showing that looking helps while choosing looks with heuristics:

None of these ties camera choice to task value, and none handles the coupling where moving the arm both creates and removes occlusion. AV-ALOHA's observation that the 21-DoF joint action space hurt simple tasks is the empirical symptom our theory predicts. Paying for every visual outcome inflates exploration even when the occlusion does not matter.

Research Program

The work has three pieces that share one design rule.

Theory. We formalize the kinematically-revealing MDP and prove the reduction from the partially observable problem. We then prove the Bernstein-versus-Hoeffding separation in $B$. The proof adapts UCBVI-BF's analysis to transitions whose support is a product over newly revealed variables. We also construct a lower-bound instance showing that the $\sqrt{B}$ factor is real for distribution-based optimism. We will extend to an eye-in-hand variant, where camera pose is a deterministic function of $q$, so look actions and motor actions are the same actions. This is the coupled case SUGARL excludes.

Tabular testbed. We build a discrete reaching-under-occlusion environment with tunable $d$, $|\mathcal{Z}|$ and visibility geometry. There we run UCBVI-BF and a Hoeffding counterpart (UCBVI-CH from the same paper) on the reduced MDP and measure regret as $B$ and the number of task-irrelevant hidden variables grow.

Deep agent. We build an agent whose critic is an ensemble. Its exploration bonus at state $(q,m)$ is a Bernstein-form term: the square root of the ensemble's estimated variance of next-state value, divided by a pseudo-count. Camera and arm actions are scored by this one bonus, with no separate look reward. The visibility mask from robot meshes masks the scene encoder, so the agent knows which variables a configuration would reveal.

Baselines

Fixed camera (do nothing). The naive reference is the same agent and encoder with a fixed camera, or with fixed cameras plus a wrist camera, and no look actions. On Active-Gym's Robosuite NutAssemblySquare, SUGARL reports a hand-coded fixed-view score of 102.9 (Active Vision Reinforcement Learning under Limited Visual Observability). On cluttered FetchPush, Cheng et al. report 61.8% success for a static camera (Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions.). Failing to beat these would mean active looking is not worth its action-space cost in our setup.

SUGARL, the strongest learned competitor on the shared benchmark. It is the strongest active-vision RL method evaluated on a public occluded manipulation benchmark in the file, scoring 108.6 on NutAssemblySquare against 8.42 for a naive joint single policy (Active Vision Reinforcement Learning under Limited Visual Observability). We will run the authors' configuration and our agent with the same backbone RL learner, the same environment-step budget, the same observation crop, and an equal hyperparameter-search budget. We report scores over at least ten seeds. Beating it means a higher mean score with non-overlapping 95% confidence intervals at matched steps, or matching its final score in clearly fewer steps. We also run an eye-in-hand variant, where SUGARL's independence assumption is violated. There we expect the largest gap, and no reported result was found on Lacuna for this setting.

Entropy-reduction looking (Observe Then Act). This is the most direct competitor to our look-selection rule. The file reports only a relative 26% gain over C2FARM with a front camera on its RLBench occlusion tasks (Observe Then Act) and no absolute success rate. We therefore re-implement its $r_e$ term inside our agent, replacing only the bonus, so the comparison isolates the exploration signal.

Ablations. Two ablations isolate the key ingredients. Swapping the Bernstein bonus for a Hoeffding-style bonus with the same pseudo-counts isolates the value-variance ingredient. Replacing the known visibility mask with a learned one isolates the kinematic-geometry ingredient.

Why not just…?

Why not just add more static cameras or render virtual views? RVT (RVT) and ActiveVLA re-render point clouds from virtual viewpoints. Virtual views cannot show points no physical sensor ever saw, such as the inside of a closed drawer or something behind dense clutter. ActiveVLA's two-stage selection also doubles latency, from 0.26 s to 0.53 s per step. AV-ALOHA's static setup reaches 30% on Thread Needle against 52% for active vision. This alternative is a real threat on tasks where the occlusion is only partial. We therefore include a fixed multi-view-plus-wrist configuration as a baseline and report results separately for tasks with complete-void occlusion and tasks with partial occlusion. Our claim is limited to tasks where some looking is necessary.

Why not just use an information-gain or entropy bonus for the camera? This is the most serious objection, because entropy reduction already works empirically (Observe Then Act). Our theory predicts a specific failure: entropy rewards resolving hidden variables that do not affect success, such as distractor poses. The controlled test adds $n$ task-irrelevant occluded distractors. Steps to a fixed success rate should grow with $n$ for the entropy agent and stay roughly flat for ours. If the two curves match, the value-variance argument has no practical value in this regime, and we will say so.

Why not just use generic ensemble-disagreement optimism without the occlusion model? Our deep agent is close to ensemble-based optimism, so this ablation matters. Removing the known visibility mask, and with it the reduction to $(q,m)$, tests whether the structure contributes beyond optimism. The tabular Hoeffding-versus-Bernstein comparison tests whether the form of the bonus matters.

Why not just imitate teleoperated looking? AV-ALOHA learns gaze behavior from VR demonstrations. Its own results show the expanded action space hurting simple tasks, and demonstrations do not address exploration when a new occlusion appears. We treat imitation as complementary. As a secondary experiment, we fine-tune from a small set of demonstrations and ask whether the bonus helps the robot recover from occlusions absent from the demonstrations.

Proposed Experiments

Tabular check (most decisive). Regret curves of UCBVI-BF versus UCBVI-CH on the reduced MDP, sweeping reveal branching $B$ from $2^1$ to $2^{8}$ with $S_r$ held fixed. The theory predicts a Hoeffding-to-Bernstein regret ratio that grows like $\sqrt{B}$. A flat ratio would falsify the central claim. Because UCBVI-BF needs large $T$, we report the budget at which separation first appears. A companion sweep over irrelevant hidden variables tests the value-aggregation conjecture.

Deep check on Active-Gym Robosuite NutAssemblySquare. We compare against SUGARL's reported 108.6 and the 102.9 fixed view, in both a decoupled-camera variant and an eye-in-hand variant.

Deep check on cluttered FetchPush. We compare against the 66.3% active and 61.8% static results of Cheng et al., with the curriculum removed. Removing the curriculum tests whether value-directed looking removes the need for clean-scene pre-training.

RLBench occlusion tasks. On the occlusion-heavy RLBench tasks from Observe Then Act (Open Drawer, Unplug Charger, Take Money Out of Safe, meat_on_grill), we run the entropy-versus-value comparison with the distractor sweep. Metrics are success rate, episode length, and number of camera moves per success.

Generalization. If the irrelevant-distractor gap appears in the tabular setting and on RLBench, the claim that looks should be paid for in value rather than pixels is supported. If it appears only in the tabular setting, the theory holds, but the deep instantiation is the bottleneck.