Expensive Controllers, Cheap Options: Using Real Compute Cost to Stop Option Collapse

Hierarchical reinforcement learning splits an agent into a high-level controller that picks a "skill" (an option: a sub-policy that runs for several steps until it decides to stop) and the skills themselves. When trained end to end, these hierarchies tend to fall apart in one of two ways. The controller may terminate every skill after one step, so it is really controlling each primitive action. Or a single skill may take over the whole task. Both failures are called option collapse. Separately, reinforcement learning with large models is expensive largely because the big network must run at every step of every rollout.

This note proposes to make the two problems fix each other. We build the hierarchy so that the controller is a large network and each option is a small, cheap network. We then charge the controller for the compute it actually uses each time it is called. Under this design, re-deciding at every step becomes costly in the quantity people pay for. A single option also cannot quietly absorb the whole task, because each option is too small to do so. The payoff, if it works, is a hierarchy that stays hierarchical using only the task's own reward, with no language-model-generated intrinsic rewards, while spending a fraction of a flat large policy's per-step compute.

Core Claim

The claim has two parts, and each can be tested on its own. First, capacity asymmetry acts as a structural pressure against option collapse. It supplies the division of labor that Wulfmeier et al. argue is missing in single-task hierarchies, where a flat optimum makes modularity redundant. When the options have limited capacity, "one option does everything" is no longer an optimum. Second, pricing the controller's measured cost per call gives a switching penalty whose value is not a free hyperparameter. It is set by a compute budget the practitioner chooses, and it yields wall-clock and FLOP savings that grow with average option duration.

The central bet is that together these two mechanisms remove the negative result in Scalable Option Learning in High-Throughput Environments. In that work, hierarchy trained only on extrinsic reward did no better than flat agents, and gains required intrinsic rewards from Motif. Those Motif rewards bring an annotation pipeline built on a 70B language model and documented reward hacking.

Relation to Prior Work

Two papers motivate the compute-as-deliberation-cost idea without doing it. DAC recasts options as two augmented decision processes that share one critic and can be trained with standard PPO. It reports that termination probabilities keep rising during training, so the agent "chatters," and it suggests a switching cost as future work. The Cost of Commitment in Option-Based Hierarchical RL proves that deliberation cost grows linearly with termination probability. It derives a U-shaped trade-off against model error and checks it on a Ring MDP and CatAndMouse. However, it treats the cost as an abstract scalar $C_{max}$, not as measured FLOPs or latency, and it never uses deep function approximation. We take its U-curve as a prediction to test in deep networks, with the cost measured rather than assumed.

The paper that supplies our method is SOL itself. It packs the controller and all options into one network conditioned on a one-hot policy index, which lets batches of agents running different options execute together on a GPU. It also adds an explicit option-length head and an entropy bonus against one-step options. SOL reaches 43,000 steps per second on NetHack, which is 86% of flat PPO throughput. Its controller and options, however, are the same network and so the same size. That means calling the controller is no more expensive than calling an option, so the architecture offers no compute reason to commit to an option. We keep SOL's batching and bootstrapping machinery and break the symmetry.

Two further lines of work constrain the design. The first is information-theoretic anti-collapse objectives such as HiT-MDP, which uses mutual-information rewards over Transformer option embeddings, and Wasserstein diversity regularizers. These work in MuJoCo but add compute on top of flat RL, so they are the wrong direction for our purpose. The second is RAGEN-2, which shows that flat language-model agents collapse into input-agnostic reasoning templates. It cuts gradient time by 26–41% by skipping rollouts whose rewards do not vary. The authors note that this signal disappears under extreme reward sparsity, which is exactly the regime where hierarchy is supposed to help.

Research Program

The agent has a controller $\pi_H$ with per-call cost $c_H$ and $K$ option policies $\pi_k$, each with per-step cost $c_L$ where $c_L \ll c_H$. All costs are measured in FLOPs, and we also profile wall-clock time on the target hardware. The controller runs only at option boundaries. If $\beta$ denotes the fraction of environment steps on which an option terminates, the expected compute per step is $c_L + \beta\, c_H$. We train with a budget constraint handled by a Lagrange multiplier:

$$\max_{\pi_H,\{\pi_k\}} \; J(\pi) \quad \text{s.t.} \quad c_L + \mathbb{E}_\pi[\beta]\, c_H \le B.$$

Here $J$ is the expected extrinsic return and $B$ is a per-step compute budget. The multiplier $\lambda$ is raised when measured compute exceeds $B$ and lowered otherwise. In effect, it becomes a switching penalty of $\lambda c_H$ per termination, and that penalty is set by the budget instead of by hand.

Implementation reuses SOL's value bootstrapping at option switches and its batched execution. The difference is that options run in a separate small network: a few-layer MLP or small CNN over the local glyph crop. The controller is a much larger recurrent encoder over the full observation and message history. The option network keeps SOL's policy-index conditioning so that batching still holds.

The analysis has three parts. We trace how option duration and option-usage entropy evolve during training, which directly diagnoses both collapse modes. We sweep the capacity ratio $c_H/c_L$ from 1 (symmetric, as in SOL) to roughly 100 to see whether collapse resistance appears as the ratio grows. We sweep $B$ to trace return against compute, and we check whether the curve has the U-shape predicted by the Cost of Commitment analysis.

The second stage moves to language-model agents on RAGEN's Sokoban and FrozenLake. There the controller is the language model, whose autoregressive decoding dominates rollout time. The options are small learned policies over primitive moves, and the language model is called only at option boundaries. This tests whether the same mechanism attacks the generation-in-the-loop bottleneck that token-level fixes such as NLPO's vocabulary masking (Ramamurthy et al.) leave in place.

Baselines

The naive reference point is a flat PPO agent that uses the large controller's architecture at every step. It shows what full compute buys on extrinsic reward alone. It will be run in the SOL codebase with the same frames, the same seeds, and the same number of hyperparameter configurations (twelve per method). The file notes that SOL retains 86% of flat throughput, and that flat baselines score 0% on the PointMaze G-maze.

The strongest existing method is SOL. We run it twice. The first run uses extrinsic reward only, where the paper reports no gain over flat; this is the direct comparison for our claim. The second run is SOL+Motif, as an upper reference that uses external feedback. Its reported numbers are 43,000 steps per second on NetHack and a 70% G-maze success rate against 0% for baselines. The file does not give SOL's NetHack score or its MiniHack returns as numbers (no reported result found on Lacuna for SOL's NetHackScore or MiniHack returns), so we reproduce them ourselves. Beating SOL means one of two outcomes under extrinsic-only reward on equal frames. Either we match or exceed its return at no more than half its per-step FLOPs, or we exceed flat PPO on the MiniHack tasks where SOL-extrinsic does not.

Two ablations isolate our ingredients. The symmetric-with-penalty ablation uses SOL-sized equal networks plus the same Lagrangian switching penalty; it tests whether asymmetry matters beyond the penalty. The asymmetric-without-penalty ablation removes the penalty; it tests whether capacity asymmetry alone prevents takeover by a single option. For the language-model stage, the baseline is RAGEN-2 with SNR-aware filtering, which reports +16.0% on Sokoban over PPO. Beating it means equal or higher success at lower total rollout decoding time.

Why not just…?

Why not just add a tuned switching penalty to a symmetric hierarchy, as DAC suggested? This is the most serious threat. For the chatter failure alone it may be enough, since the Lagrangian penalty is a scalar penalty with an automatic setting. It does nothing, however, against one option taking over the whole task, and in a symmetric network it saves no compute, because the controller costs the same as an option. The symmetric-with-penalty ablation settles this question. If it matches our method on return and on option-usage entropy, the asymmetry claim is wrong, and the contribution shrinks to budget-set penalties plus compute savings.

Why not just call the large flat policy every k steps and repeat its action? Fixed action repeat also cuts controller calls by a factor of $k$ with no hierarchy. It is open-loop, though, and NetHack combat and MiniHack's procedurally generated rooms need reactive control within a macro-action. We include flat-with-repeat at the same $k$ as our measured mean option length, and also a variant with a learned repeat length. If repeat matches us at equal compute, then what our small options learn is not worth having.

Why not just train a small flat policy, which is cheaper still? If a policy at $c_L$ solves the task, the large controller is unnecessary. MiniHack reports that flat baselines drop to zero as rooms grow from 5x5 to 15x15, which suggests that small flat policies fail at long horizons. We test this with compute-matched flat policies sized to our measured per-step FLOPs.

Proposed Experiments

The primary suite is the hard MiniHack tasks (ZombieHorde, TreasureDash, and the 15x15 procedural rooms) plus the full NetHack Learning Environment (Küttler et al.). All runs use extrinsic reward only, five seeds, and the frame budgets used by SOL, and results are plotted against both frames and cumulative FLOPs. PointMaze G-maze serves as a check on deceptive exploration.

The metrics are episode return or NetHackScore and per-step FLOPs, along with wall-clock steps per second. For collapse we track the distribution of option durations, the entropy of option usage over states, and the fraction of steps covered by the most-used option.

The decisive check has three conditions. With extrinsic reward only, our method must beat flat PPO where SOL-extrinsic does not. The symmetric-with-penalty ablation must show collapse that our method avoids. The action-repeat and compute-matched flat baselines must fall below us at equal FLOPs. If the first condition fails, the core bet has failed, and we report asymmetry as a compute-saving technique only. The Sokoban stage is run only if the grid-world results hold.