TL;DR
Adaptive Internal Computation (AIC) is a reinforcement learning framework that allows agents to dynamically decide how much "thinking" (internal refinement) to perform for each state. By jointly training action selection and a halting mechanism, AIC achieves higher rewards with significantly lower latency—reducing per-decision computation by up to 44% compared to fixed-depth policies.
What Problem Does This Paper Address?
Most neural network policies are "one-size-fits-all" when it comes to computation: they spend the same amount of processing power on every decision. However, in the real world, some situations are routine (e.g., walking on flat ground) and require minimal thought, while others are complex or unexpected (e.g., recovering from a slip) and benefit from deeper processing.
Iterative policies—which repeatedly refine a state representation before acting—offer a way to scale computation. But using a fixed number of iterations is inefficient: it wastes time on easy states and may not provide enough "thought" for hard ones. The challenge is learning a state-dependent stopping rule that balances performance and computational cost, using only the sparse rewards provided by the environment.
Methods
The authors introduce Adaptive Internal Computation (AIC). The core idea is to treat the stopping depth ($\tau$) and the action ($a$) as a single joint decision made by the policy.
The Architecture
As shown in the architecture diagram below, AIC uses an encoder to produce an initial latent state $z_0$. A shared residual block then iteratively refines this state ($z_1, z_2, \dots, z_{d_{max}}$).
At each refinement step $d$, the model has three "heads":
- Action Head ($\pi_d$): Proposes what to do if the agent stops now.
- Value Head ($V_d$): Predicts the expected return.
- Halting Head ($h_d$): Predicts the probability of stopping at this step.
Training and Objectives
AIC is trained using Proximal Policy Optimization (PPO) on the joint distribution of depth and action. This ensures that the environment's return provides a training signal for both what to do and how long to think. To prevent the model from "thinking" forever, they add an expected-depth penalty to the loss function. Additionally, auxiliary value supervision is applied at every depth, forcing all intermediate representations to predict the same final return, which helps shape a useful latent space early in the refinement chain.
Datasets
The authors evaluated AIC across a diverse 12-task benchmark:
- Continuous Control: MuJoCo locomotion (HalfCheetah, Ant).
- Robotic Manipulation: ManiSkill3 (PickCube, StackCube).
- Navigation: D4RL AntMaze and MiniGrid (DistShift, FourRooms).
- Language: GPT-2 on GLUE-RTE (textual entailment).
- Rare Events: A specialized ManiSkill task involving slips and misalignments.
- Goal-Conditioned RL: OGBench (Cube, Scene, Puzzle).
Key Insights and Ideas
- State-Dependent Allocation is Mathematically Superior: The authors provide a theoretical proof (Theorem 1) showing that if states vary in difficulty, a state-dependent stopping rule will always outperform any fixed or randomized state-independent schedule at the same average computational budget.
- "Thinking" is a Policy Decision: By including the stopping probability in the PPO ratio, the agent learns to stop based on the value of the resulting action, not just a heuristic like "confidence" or "convergence."
- Task-Specific Gains: Ablation studies revealed that for manipulation tasks (like PickCube), the primary benefit comes from the adaptive stopping itself. In contrast, for rare-event tasks, the auxiliary value supervision (which improves the underlying representation) was the dominant factor.
- Zero-Shot Transfer: AIC isn't tied to a specific architecture. The authors successfully attached AIC halting heads to five different diffusion-based policy families (like DiffusionQL and IDQL), showing consistent performance gains.
Key Results
- Efficiency: Across twelve tasks, AIC executed only 3.3 refinement steps on average, despite a maximum cap of 8.
- Speed: It reduced per-decision latency by 31–44% on control and robotics tasks compared to a fixed eight-step policy.
- Performance: AIC's mean reward exceeded the best fixed-depth baseline on 7 out of 12 tasks.
As seen in the performance plots, AIC (red) often matches or exceeds the fixed comparator (blue) while using significantly less computation on average.
Limitations and Future Work
- Training Cost: While AIC is faster at inference, training takes about 1.22x longer than a fixed-depth policy because the model must evaluate all refinement depths to calculate the auxiliary value losses.
- Encoder Bottlenecks: In tasks dominated by a massive frozen encoder (like GPT-2), the time saved by skipping refinement steps is a smaller fraction of the total end-to-end latency.
- Safety in Rare States: There is a risk that an adaptive policy might learn to halt too early in rare, critical states that were under-represented in the training data.
- Scaling: The current experiments use relatively small refinement modules in simulation. Testing AIC on large-scale backbones and real-world physical robots remains an open area for research.
Summary updated