Teaching Masked Policies What the Mask Knows

Many agents cannot act freely. A robot's planner may only pick a skill whose preconditions hold, and a game unit may only move to an empty cell. The standard fix is an action mask: before the agent samples, every invalid choice gets probability zero. Masks help learning a great deal, but they have a hidden price. The policy never learns where the invalid region is, because a masked action produces no learning signal. So the mask has to be computed for every candidate action at every step, forever. That cost is small in a grid game. It becomes a real overhead in a hierarchical robot controller, where checking one high-level skill can mean running inverse kinematics, a motion planner, or a query to the simulator.

This note proposes a way to have both properties. During training, the mask is applied only to the target distribution, never to the network. We use decoupled policy optimization, a family of methods that first computes an improved action distribution in closed form and then fits the network to it by supervised regression. The target places zero mass on invalid actions. The network, left unmasked, is fit to that target with cross-entropy. This automatically pushes down each invalid action in proportion to how likely the network currently thinks it is. At deployment we check only the action the policy actually proposes, and resample if it fails. The hard validity guarantee is kept, and the number of feasibility checks falls as the policy absorbs the constraints.

Core Claim

The claim has three parts. First, mask-aware decoupled updates teach an unmasked policy to place near-zero probability on invalid actions. The learning pressure falls mainly on the invalid actions the policy is tempted by, and needs no extra loss weight. Second, this internalization turns directly into lower feasibility-checking cost through lazy verification. Third, both effects hold for the high-level decision maker of a hierarchical controller, where feasibility checks and compute budgets are the costly, hard constraints.

The mechanism fits in two lines. Write $\pi_\theta(a\mid s)$ for the unmasked softmax policy, $M_s(a)\in\{0,1\}$ for the validity mask, $Q(s,a)$ for a learned action value, and $\eta$ for a temperature. The target is

$$q(a\mid s)\propto M_s(a)\,\pi_\theta(a\mid s)\exp\big(Q(s,a)/\eta\big).$$

This is the Boltzmann-style target of Maximum a Posteriori Policy Optimisation, restricted to valid actions. We then minimize $-\sum_a q(a\mid s)\log\pi_\theta(a\mid s)$. For any invalid action $b$ with logit $z_b$, the gradient of this loss is exactly $\pi_\theta(b\mid s)$, because $q(b\mid s)=0$ and $q$ sums to one. Invalid logits are therefore pushed down in proportion to their current probability. Masked policy gradients cannot do this, since they zero these gradients by construction.

At deployment, let $\rho(s)$ be the total unmasked probability on invalid actions. Sampling from $\pi_\theta$ and verifying only the sampled action costs an expected

$$\mathbb{E}[\text{checks}] = \frac{1}{1-\rho(s)}$$

feasibility queries (the mean of a geometric distribution, when sampling with replacement). Computing the full mask instead costs $|\mathcal{A}|$ queries. After $k$ failed draws we fall back to the full mask, so no invalid action is ever executed.

Relation to Prior Work

Three lines of work motivate the gap. A Closer Look at Invalid Action Masking in Policy Gradient Algorithms proves that logit masking yields valid policy gradients in which invalid actions receive zero gradient. It also shows that masked agents degrade when the mask is removed, which is the "crutch" effect we target.

The same paper supplies a constraint on our design. Naive masking, which samples from the masked policy but takes gradients against the unmasked one, causes variance and large jumps in the divergence between successive policies. We must explain why our unmasked fit is different. The explanation is that naive masking uses a sampling-based gradient on a distribution the samples did not come from. Our fit is a regression toward an explicit target distribution over the whole discrete action set, so no sampled actions enter the gradient and the mismatch does not arise. This is testable, and we include naive masking as a baseline.

Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking extends masking to convex continuous sets. It reports that exact distributional masking runs up to 170 times slower than unmasked PPO on Walker2D. This is evidence that mask computation itself can dominate cost. We stay in discrete, high-level action spaces, where lazy verification is exact.

The method comes from decoupled optimization. Maximum a Posteriori Policy Optimisation, Supervised Policy Update for Deep Reinforcement Learning and First Order Constrained Optimization in Policy Space all compute a closed-form target and project it onto the network by supervised fitting. None of them injects a state-wise hard mask into the target. FOCOPS handles only expected cumulative cost. What we add is specific: an asymmetry between a masked target and an unmasked student.

The hierarchical setting supplies the regime.

  • HiRT separates a slow planner from a fast controller to escape latency limits.
  • SuSIE shows high-level planners proposing subgoals the arm cannot reach. This is the problem that physical-feasibility masks passed up to the planner would solve.
  • LoTa-Bench already masks LLM planners to a valid skill set, with a 19x planning speedup. It does not train with reinforcement learning, and it does not ask whether the mask can be amortized.
  • RARRL trains an orchestrator to choose between acting and invoking an LLM, using soft latency penalties. Its authors note that soft penalties cannot guarantee hard budgets. We treat "not enough budget left to think" as a cheap hard mask on the THINK action.
  • HiPPO warns that updating low-level skills during training makes the upper level's problem non-stationary. We therefore freeze the low-level skills, so feasibility masks stay stationary.

Research Program

We will build a discrete decoupled learner of the MPO type. The E-step computes $q$ over the full valid action set using a critic trained on replay data. For combinatorial spaces the action is factorized, and the target is computed per component: in µRTS these components are unit choice, action type, and parameters. The M-step minimizes cross-entropy to $q$ under the unmasked network, with a KL trust region as in MPO. Rollouts during training still sample from the masked policy, so exploration keeps the advantages Huang and Ontañón document.

Throughout training we log three quantities: the invalid mass $\rho$, how $\rho$ is distributed over high-value versus irrelevant invalid actions, and the number of feasibility queries under lazy verification.

The hierarchical instance is a high-level policy over a finite skill library in ALFRED, using the skill set of LoTa-Bench, with frozen low-level skills that the simulator executes. The action set is augmented with a THINK action that calls an LLM planner, as in RARRL. There are two mask sources. One is an expensive skill-feasibility mask: simulator preconditions plus an instrumented reachability check whose cost we charge in wall-clock time. The other is a cheap hard budget mask on THINK. The analysis asks three questions:

  • Does $\rho$ shrink specifically on the expensive-mask actions?
  • Does lazy verification then cut total control-loop latency?
  • Does the hard budget mask remove the budget overruns that RARRL's soft penalty permits, without costing task success?

Baselines

The naive reference point is penalty-only PPO, which uses no mask and gives a negative reward for invalid choices. On the 24x24 µRTS map, Huang and Ontañón report an episodic return of 0.50 for this approach, against 40.00 for masked PPO.

The strongest existing method is masked PPO from A Closer Look at Invalid Action Masking in Policy Gradient Algorithms. The file reports a return of 40.00 across all µRTS map sizes. It is the strongest competitor because it already achieves top return with a full mask. It is not beaten on return. It is beaten on cost: matching its return within seeds while needing far fewer feasibility queries per decision, and while degrading less when the full mask is removed. For the mask-removed degradation, no reported result was found on Lacuna. Masked PPO is paired with full-mask deployment and with lazy verification applied to its own unmasked logits.

Naive masking is run to test the gradient-mismatch argument.

The ablation that isolates the key ingredient is masked MPO, in which the mask is also applied inside $\pi_\theta$. It is identical to our method except that invalid logits receive no gradient. Any gap between the two is attributable to the unmasked fit.

Two further competitors, described below, are masked PPO with an auxiliary invalid-mass loss and a separately trained mask classifier.

On ALFRED, the reference numbers are LoTa-Bench's 40.38% task success for GPT-4 and "over 60%" for fine-tuned LLaMA 30B. These come from in-context and supervised planners, so they serve as context rather than head-to-head targets.

For the budget experiments, the baseline is RARRL's soft-penalty orchestrator, reported at 580 tokens per episode against 4,100 for always-on reasoning. We compare hard-masked and soft-penalty orchestrators on the same abstract scenarios.

All methods share the same environment steps, network size, and number of seeds (at least five). Each gets an equal hyperparameter search budget, and each is charged the same per-query cost for feasibility checks.

Why not just…?

Why not just add an auxiliary loss to masked PPO that pushes down unmasked invalid probability? This is the most serious threat. The gradient of $-\log\sum_{a\,\text{valid}}\pi_\theta(a\mid s)$ on invalid logits has the same shape as ours. The decoupled form differs in two ways. It supplies this pressure as part of the single projection objective, with no extra weight to tune. It also reuses replay states whose masks were already computed, which cheapens training when checks are expensive. The auxiliary variant gets its own tuning budget. If it matches our method on $\rho$, query savings, and return, we will report that the contribution lies in the unmasked-student principle and lazy verification, not in decoupling itself. We will not argue that point further.

Why not just train a separate classifier to predict the mask? A mask classifier must learn validity for every action, most of which the policy would never pick. Our gradient concentrates on invalid actions the policy is attracted to. We test this directly by comparing, at equal parameter counts, the queries saved per unit of training compute, and by measuring accuracy on invalid actions the policy actually proposes. If the classifier wins, lazy verification can use it instead, and we will say so.

Why not just compute the full mask and accept the cost, or use penalties? Penalties fail, as the µRTS figures (0.50 versus 40.00) and the quadrotor failures in Excluding the Irrelevant show. Computing the full mask is fine when checks are cheap. That is why the ALFRED experiments charge realistic reachability costs, and why we report savings as a function of per-check cost rather than claim them in general.

Proposed Experiments

In µRTS, we sweep map sizes from 4x4 to 24x24, where the chance that a random action is valid falls from 12.5% to 0.34%. We report episodic return, $\rho$ across training, expected checks per decision under lazy verification, wall-clock time, return with the mask removed, and the divergence between successive policies. The last metric tests whether our fit avoids the instability of naive masking.

In ALFRED, with the LoTa-Bench skill set, we report task success, feasibility queries per episode, total control-loop latency, and the fraction of executed skills that were infeasible. That fraction must be zero for every masked or verified method.

In RARRL's abstract navigation, inspection, and delivery scenarios with budget shocks, we report budget-violation rate, success, and tokens per episode for hard-masked versus soft-penalty orchestrators.

Three results would support the claim:

  1. Our method and the masked-MPO ablation match masked PPO's return.
  2. Only our method, and possibly the auxiliary-loss variant, drives $\rho$ low enough that measured checks per decision approach the $1/(1-\rho)$ prediction, while masked MPO stays near full-mask cost.
  3. The query savings grow with action-space size in µRTS and cut latency in ALFRED without any infeasible execution.

The result that would refute the claim is equal query savings from the auxiliary-loss or classifier baselines. In that case the idea reduces to lazy verification, and we will report it as such.