Robot policies trained by copying human demonstrations (behavior cloning) are capped by the quality of those demonstrations. They are also increasingly too slow to run at every motor tick, so they output "action chunks," short open-loop sequences of future actions that a fast controller plays back while the large model thinks. Recent work breaks the demonstration cap by learning a value function over whole chunks, a critic that scores how good a sequence of actions is. All of it assumes the robot sees the true state. On real hardware, cameras get occluded and sensors lag, and the robot must act on a belief, meaning a probability distribution over what the world might currently be. Under a belief, how long to commit to a chunk is not just a speed-versus-smoothness knob. Every step executed open-loop is a step in which new observations are ignored. That can be free when the robot is confident and costly when a hidden variable is about to matter.
This note proposes a learned value-of-information rule that sets chunk length per decision. We train one belief-conditioned critic that estimates, for each candidate commitment length $k$, the value of executing $k$ actions and then re-observing and replanning. The robot commits to the longest chunk whose value is within a tolerance of the best. The rule plays three roles at once. It is the mechanism that surpasses the demonstrations, because the critic is trained by Bellman backups (bootstrapping value estimates from later ones) over suboptimal and failed data. It is the real-time budget controller, because longer commitments mean fewer expensive replans. It is also a belief-compression device: committing open-loop collapses the branching tree of possible future beliefs. The hypothesis is that this makes the critic easier to fit from limited offline data. That was shown for exact tabular planning but never for learned, continuous, chunked control.
Core Claim
Our central claim is that under partial observability, the gap between "commit to this chunk" and "execute a prefix, then look again" is learnable from offline data by the same chunked temporal-difference machinery that already beats behavior cloning. Using this gap to choose commitment length should achieve three things relative to fixed-horizon chunked critics given the same belief encoder. It should raise success, reduce model invocations per second, and yield lower critic error on held-out data. The last effect should come from a smaller set of reachable beliefs.
The idea transplants a known result into a regime where it has not been tested. Belief-Dependent Macro-Action Discovery in POMDPs using the Value of Information defines value of information as closed-loop value minus open-loop value. It chains open-loop actions whenever this falls below a threshold $\tau$, and proves regret bounded by $\tau$. It also finds that the shrunken reachable belief set lets a fixed-budget solver beat closed-loop planning. That work motivates ours, but it needs an exact transition and observation model and is evaluated only on a 10x10 gridworld. We replace exact point-based planning with a learned chunked critic over a learned belief, trained from offline demonstrations.
Research Program
Belief representation. A small recurrent or causal-transformer encoder maps the history of observations and actions to a belief vector $b_t$. Two prior papers constrain the design. Learning Belief Representations for Imitation Learning in POMDPs (BMIL) regularizes a recurrent belief module by predicting future and past raw observations. The file notes that raw-observation reconstruction is too expensive for fast loops, and BMIL remains an imitation method with no value stitching. Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization (DT-CORL) shows that pretraining a belief model and then freezing it fails under distribution shift, and that joint belief–critic training fixes this. DT-CORL, however, acts one step at a time. We therefore train the encoder jointly with the critic, and regularize it by predicting future latent features rather than pixels.
A family of commitment critics. Let $A = (a_t,\dots,a_{t+H-1})$ be a candidate chunk of maximum length $H$. For each $k$ in a small set $K \subseteq \{h_a,\dots,H\}$, a shared trunk with per-$k$ heads estimates the value of executing the first $k$ actions open-loop and then replanning from the updated belief. The training target is
$$Q_k(b_t, a_{t:t+k-1}) \leftarrow \sum_{i=0}^{k-1}\gamma^i r_{t+i} + \gamma^{k}\max_{j\in K,\,A'\sim\pi_0} Q_j(b_{t+k}, A'_{1:j}),$$
where $\gamma$ is the discount, $r$ the reward, and $\pi_0$ a frozen imitation-trained chunk generator from which the max is approximated with $N$ samples. As in Reinforcement Learning with Action Chunking (Q-chunking), conditioning on the exact executed action sequence makes the multi-step backup free of the usual off-policy bias. This is what lets value propagate across suboptimal demonstrations.
What differs from Q-chunking and from Decoupled Q-Chunking (DQC) is that the heads differ by when re-observation happens, not only by input length. DQC's partial critic optimistically maximizes over completions of a short prefix from the same state. Under a belief, that is still open-loop. Our $Q_k$ bootstraps from the belief after $k$ new observations, so $Q_k$ for small $k$ is a closed-loop value and $Q_H$ is the open-loop one.
Commitment rule. At each replan, draw $N$ chunks from $\pi_0$ and score every (chunk, $k$) pair. Let $Q^\star$ be the best score. The robot executes the longest $k$ whose best chunk satisfies $Q_k \ge Q^\star - \tau$. The difference $Q^\star - Q_H$ is our learned estimate of the value of information. Setting $\tau=0$ recovers pure value maximization. Raising $\tau$ trades a bounded amount of value for fewer replans, mirroring the macro-action paper's regret-for-compute trade.
Execution. Chunks are stitched asynchronously with the inpainting scheme of Real-Time Execution of Action Chunking Flow Policies (RTC), so variable commitment does not produce stop-and-go motion. Following Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning, the heavy encoders run once per replan and only a light head scores candidates. The per-$k$ heads therefore add negligible cost, and the dominant cost is replan frequency.
Analysis of belief compression. To test the belief-dimensionality mechanism directly, we estimate an empirical covering number: how many $\varepsilon$-balls in belief-embedding space are needed to cover beliefs visited at replan points. We measure this under fixed-short, fixed-long, and value-of-information commitment, and relate it to critic error on held-out returns. The covering analysis of A Covering Framework for Offline POMDPs Learning Using Belief Space Metric predicts evaluation error that grows with the size of the belief cover. That paper is purely theoretical, and we use it as the lens for this measurement rather than as a method.
Baselines
The naive reference is the frozen imitation chunk policy $\pi_0$ run at a fixed horizon with RTC stitching and the same belief encoder trained only by imitation. It shows the demonstration ceiling under partial observability. On full-observation LIBERO-10, the comparable imitation policy in Q-Planning starts at 93% success before self-improvement (Beyond Imitation). We expect it to fall well below that once occlusion is introduced.
The strongest existing method is DQC with a history encoder bolted on. DQC separates a long critic horizon from a short policy horizon and is the best reported chunked critic in the file. On OGBench cube-triple-100M it reports 98% success versus SHARSA at 82% and standard Q-chunking at 17%. On cube-quadruple-100M it reports 93% versus SHARSA 67% and Q-chunking 29% (Decoupled Q-Chunking). Those numbers are for fully observed states. We found no reported result on Lacuna for DQC or Q-chunking on partially observable or delayed variants. We will run DQC with our exact jointly trained belief encoder, its best $(h, h_a)$ pair chosen from a grid, and the same data, gradient steps, candidate count $N$, and tuning budget. Beating it means higher success at matched or lower replans per second. Matching success with substantially fewer replans also counts, as a latency result.
Q-Planning over the same frozen $\pi_0$ with a history-conditioned critic is the third competitor, on the LIBERO variants. Its reported full-observation result is 93% to 99% on LIBERO-10.
The key ablation keeps our full critic family and belief encoder but replaces the value-of-information rule with the best single fixed $k$. This isolates adaptive commitment. A second ablation feeds only the current frame, with no belief, to test whether the belief matters at all. A third trains the belief in two stages, as a check on DT-CORL's finding.
Why not just…?
Why not just gate horizon length by prediction disagreement, as Mixture of Horizons does? Mixture of Horizons in Action Chunking shortens execution when its multi-horizon imitation heads disagree and reports up to 2.5x throughput. Disagreement among copies of the demonstrator measures how multimodal the demonstrations are, not how much an unseen variable changes the return. It stays high at harmless forks and can stay low when every demonstrator walks blind into an occluded obstacle. MoH's authors themselves list occlusion and state ambiguity as failure cases. This is a real competitor for the latency part of our claim. We will run MoH-style consensus gating on top of our trained critic and compare against value-of-information gating at equal replan rates. The deciding environments are those where a hidden variable becomes relevant at a predictable point that demonstrations do not flag.
Why not just use short chunks everywhere and let RTC hide the latency? RTC tolerates delays up to 300 ms, so always replanning after a few steps may already be fast enough and maximally reactive. This is a serious threat to the success claim. If short fixed chunks with a belief-conditioned critic match our success, the remaining contribution is compute: fewer model calls per second, relevant for on-robot power and multi-arm scheduling. We will report success and calls per second jointly. We will also test the tabular finding that less-frequent re-observation can improve the learned critic through a smaller belief cover. If that effect does not appear, we will say the method is a compute saver, not a performance gain.
Why not just use critic ensemble disagreement as the replan trigger? Epistemic uncertainty tells you the critic does not know a value. It does not tell you whether looking again would change the action. The two differ when the critic is uncertain but every candidate leads to the same outcome. We include an ensemble-disagreement gate as a baseline at matched replan rates.
Why not just add a history encoder to Q-chunking and stop there? That is the no-adaptation ablation above. If it matches the full method, the value-of-information rule adds nothing.
Proposed Experiments
The primary testbed is OGBench manipulation, specifically cube-triple and cube-quadruple. We chose these because chunked critics are known to beat imitation there (Reinforcement Learning with Action Chunking reports about 86% aggregated over five domains versus 50–60% for baselines). We build two partially observable variants. One masks cube poses from the observation for random intervals. The other adds observation delays of 4–16 steps, the range DT-CORL studies. OGBench warns that value stitching collapses without dataset noise: 99% drops to 6% on pick-and-place. We therefore report results on both its noisy and noise-free datasets, and treat collapse on the noise-free data as a boundary condition rather than a failure to hide. The secondary testbed is LIBERO-10 with injected camera occlusions of objects already seen earlier in the episode. This probes whether the method carries over to visual policies with a frozen imitation generator.
The metrics are success rate, replans per second, and wall-clock latency per replan. The BEAST decoder reports 19 ms single-pass latency and is a reference point for what a chunk generator costs (BEAST). We also report held-out critic error and the empirical belief covering number at replan points.
Three checks decide the question. First, on the occluded cube tasks, value-of-information commitment must beat history-augmented DQC at matched replans, or match it with fewer. Second, the commitment length it chooses should drop at the moments an occluded cube becomes relevant; we will verify this by aligning $k$ with occlusion timing. Third, across commitment strategies, lower covering number should predict lower critic error. That is the learned analogue of the macro-action paper's belief-compression effect.
If the first check fails but the second holds, the rule is interpretable but not useful. If the third fails, the belief-dimensionality story is wrong for learned critics, and we will report it as a negative result.
References
- Belief-Dependent Macro-Action Discovery in POMDPs using the Value of Information
- Learning Belief Representations for Imitation Learning in POMDPs
- Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization
- Reinforcement Learning with Action Chunking
- Decoupled Q-Chunking
- Real-Time Execution of Action Chunking Flow Policies
- Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
- A Covering Framework for Offline POMDPs Learning Using Belief Space Metric
- Mixture of Horizons in Action Chunking
- OGBench: Benchmarking Offline Goal-Conditioned RL
- BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learning