Lattice-Neighborhood Soft Policies for Multimodal Continuous Control

Maximum entropy reinforcement learning trains an agent to collect reward while keeping its action choices as random as possible. Its best-known continuous-control method, Soft Actor-Critic, buys a tractable entropy term by making the policy a single bell-shaped Gaussian. A single Gaussian cannot put weight on "go left" and "go right" at once, so the method gives up exactly the multimodal exploration that maximum entropy was supposed to provide. The usual alternatives each pay a different price. Discrete policies over binned actions either treat every joint independently or face an exponentially large joint action set. Mixture and energy-based policies are expressive, but their entropy and log-probability are hard to compute.

We propose a policy built from a lattice. A lattice here is a regular, infinitely repeating grid of points, such as the hexagonal grid in 2D or the Gosset lattice $E_8$ in 8D, which is far more evenly spaced than a cubic grid. The construction has three steps. A small network proposes a continuous anchor and a scale for each state. The candidate actions are the anchor plus the scaled nearest lattice point and its first shell of neighbors: 241 points for $E_8$, which are 240 almost perfectly isotropic directions plus the center. The policy is the exact Boltzmann distribution of the critic over these candidates, meaning each candidate is chosen with probability proportional to $\exp(Q/\alpha)$, followed by uniform jitter inside that candidate's lattice cell. Distinct lattice points own disjoint cells, so the resulting continuous action density is multimodal and its log-density and entropy are exact. Computing them takes one critic batch and one lattice rounding step, with no sampling network, no density estimator and no stored codebook.

Core Claim

Let $\Lambda$ be a lattice whose Voronoi cell (the region of space closer to a given lattice point than to any other) has volume $V$. Let $c(s)$ be a state-dependent scale and $N(s)$ the set of candidate points around the anchor. Define the policy to pick a point $\lambda \in N(s)$ with probability $p(\lambda|s) \propto \exp(Q(s,\lambda)/\alpha)$ and then add noise uniform over the scaled cell around $\lambda$. Any action $a$ maps back to its point by nearest-lattice-point rounding, $\lambda = Q_\Lambda(a)$. This gives

$$\log \pi(a|s) = \log p(Q_\Lambda(a)\,|\,s) - D\log c(s) - \log V, \qquad \mathcal{H}(\pi(\cdot|s)) = \mathcal{H}(p(\cdot|s)) + D\log c(s) + \log V,$$

where $D$ is the action dimension. Both identities follow because the density is constant on each disjoint cell.

The claim is that this policy recovers the multimodal exploration of Soft Q-learning at roughly the cost of a discrete soft update over a few hundred actions. The claim also says which lattice matters, and why. The quantity that controls the error of the discretized soft value is the critic's Lipschitz constant times the lattice's covering radius, meaning the farthest any action can be from a candidate. It is not the mean-squared "shaping gain" that motivates lattices in compression. At a fixed candidate count, $E_8$ and the hexagonal lattice cover a ball far more evenly than a cubic grid, whose covering radius grows like $\sqrt{D}/2$ relative to point spacing. This is a falsifiable prediction: lattice geometry should matter most in high-dimensional, strongly coupled action spaces like Ant and least in Hopper.

Relation to Prior Work

Two lines of work motivate the proposal by documenting the gap. Soft Q-learning showed that an energy-based policy $\pi \propto \exp(Q/\alpha)$ explores multiple modes. It found all four goals in a 2D multi-goal task, succeeded 100% of the time in an Ant maze where DDPG succeeded 60% of the time, and always found forward swimming in Swimmer where DDPG chose backward swimming 20% of the time. It reached this by training an amortized sampler with Stein variational gradient descent, an iterative particle method that is costly and unstable, and it has no exact log-density. Soft Actor-Critic made maximum entropy RL practical with a squashed Gaussian and automatic temperature tuning against an entropy target. DSAC-T states plainly that this Gaussian is unimodal. Our policy keeps Soft Q-learning's Boltzmann form but evaluates it exactly on a structured local candidate set, so the entropy target used by SAC's temperature tuning becomes exact rather than a sample estimate.

A second line discretizes continuous actions, and it constrains our design. Action Branching bins each joint separately (for example, 33 bins per dimension on Humanoid) and predicts per-branch values. This is cheap, but its greedy action choice factorizes across joints, so it cannot express a correlated diagonal move as a single choice. AQuaDem learns 10 to 30 state-dependent candidate actions from demonstrations and runs discrete RL on them. It reports near-100% success on Adroit, but the agent can never leave the demonstrated support. Maxi-mix-mini-com learns a reward-free Gaussian-mixture codebook. It reports recovering 56% of SAC's Ant performance with 64 components and over 80% on Hopper, and notes that it must collapse component variances to modes. Our candidates are not learned or frozen. They move with the anchor and scale at every state, they are fixed by geometry, and the policy over them stays entropy-regularized throughout training. Revisiting Discrete SAC supplies stabilizers we reuse: an entropy penalty against sudden entropy crashes, and averaged, clipped double critics in place of the minimum of two critics, which it shows underestimates values. That work only studies small, unordered button sets.

The lattice machinery comes from compression. LLVQ shows that high-dimensional lattices can be decoded and indexed algebraically and organized into concentric shells of equal norm. We use the first shell as our candidate set, and the shell structure is a natural route to a two-level policy if we later go beyond the first shell. Xi Zhang et al. and Savkin supply fast nearest-point rules for low-dimensional lattices and $E_8$. Kobus et al. use dithered relaxations to train through quantizers. Our within-cell uniform jitter is a dither in the same sense, but we use it to make the density exact rather than to estimate gradients. Runsen Feng quantifies the roughly 1.53 dB mean-squared gain of hexagonal over square cells. We argue that this is the wrong figure of merit for control and test covering instead.

Research Program

The method matches the lattice to the action dimension. Swimmer and the 2D multi-goal task use the hexagonal lattice (6 neighbors plus center). Hopper uses the face-centered cubic lattice (12 neighbors). Ant uses $E_8$ (240 neighbors). Humanoid, as a stretch case, uses a product of $E_8$ blocks with a short remainder.

The anchor network learns by moving its output toward the Boltzmann-weighted mean of its candidates, a soft mean-shift step. The scale network is trained by dual gradient descent so that the exact entropy above meets the target. The critic's soft value is $\alpha \log \sum_{\lambda\in N(s)} \exp(Q(s,\lambda)/\alpha)$ plus the exact volume term, computed in one batched forward pass.

Bounded action boxes are handled by masking candidates whose cell leaves the box and renormalizing, which keeps the density exact. As an option, the lattice is randomly rotated each episode so that its 240 directions carry no fixed orientation bias. A rotated lattice is still a lattice, so exactness is preserved.

The analysis part of the program states and checks the error bound between the discrete soft value and the continuous soft value over the anchor's neighborhood, as a function of covering radius. We measure the critic's empirical Lipschitz behavior along rollouts to see whether the bound predicts where lattice geometry helps.

Baselines

The naive reference point is a uniform-random action agent together with Gaussian SAC run at a fixed temperature, which gives the floor for exploration metrics.

The strongest competitor on returns is Soft Actor-Critic with automatic temperature tuning, run with the same critic architecture, replay ratio and environment steps, and a matched hyperparameter search budget. For SAC's MuJoCo returns there is no reported result found on Lacuna, so we rerun it ourselves.

The strongest competitor on multimodal exploration is Soft Q-learning. Its reported Ant-maze success is 100% versus 60% for DDPG, and its Swimmer optimal-mode rate is 100% versus 80% for DDPG (Soft Q-learning). Beating it means matching those rates at lower wall-clock cost and with lower variance across seeds.

For discretized control, the comparisons are Action Branching run with SD-SAC's discrete soft updates, and the maxi-mix-mini-com reported fractions of SAC (56% on Ant with $K=64$, over 80% on Hopper). Beating the discrete methods means exceeding those fractions on the same tasks.

The ablation that isolates the key ingredient keeps everything identical except the candidate geometry, comparing two variants at the same candidate count. In the first, the 240 $E_8$ neighbors are replaced by a cubic-grid neighborhood. In the second, they are replaced by 240 Gaussian perturbations of the anchor, with importance weighting for the log-density. A second ablation removes the within-cell jitter to measure what exact continuous entropy contributes over a purely discrete policy.

Why not just…?

Why not just bin each joint and use Action Branching? Per-joint bins are cheap, but each bin is chosen independently given the state embedding. A correlated move such as "both front legs forward" therefore cannot be expressed or explored as one choice. This is also an imperfect head-to-head: Action Branching as published uses ε-greedy Double DQN rather than an entropy-regularized policy (Action Branching), so we pair it with SD-SAC's soft updates to make the comparison fair. If branching matches us on Ant, then joint geometry does not matter there, and the claim fails.

Why not just sample a few hundred random Gaussian candidates around the anchor and take a softmax over the critic? This is the real threat. It matches our cost and our multimodality, and it differs only in two respects. Random candidates cover the neighborhood unevenly. They also require an approximate, importance-weighted log-density rather than an exact one. Our candidate-geometry ablation measures this directly, at equal candidate count and equal critic calls. If random candidates match returns and entropy-target tracking, the lattice adds nothing and we will report that. Our prediction, which follows from the covering argument, is that the gap grows with action dimension: negligible on Swimmer, visible on Ant.

Why not just use Gaussian SAC, or a Gaussian-mixture policy? Gaussian SAC is unimodal, and the Soft Q-learning results show what this costs on deceptive tasks. Mixture policies such as maxi-mix-mini-com reach only part of SAC's performance, at 56% on Ant, and need variance collapse to stay stable (Maxi-mix-mini-com). The Ant-maze and multi-goal experiments test directly whether our policy keeps several modes alive where Gaussian SAC commits to one.

Proposed Experiments

The first decisive check is mode discovery. On the 2D multi-goal task, Swimmer and the Ant maze, we measure the fraction of seeds that find all goals, or the optimal mode, using Soft Q-learning's protocol and reported rates as the bar.

The second is sample efficiency and final return on Hopper, Ant and Humanoid in MuJoCo over 10 seeds, reported as a fraction of our rerun SAC so that the maxi-mix-mini-com figures are directly comparable.

The third is entropy-target tracking. We compare the exact lattice entropy with SAC's sample estimate and record whether entropy crashes of the kind SD-SAC documents appear.

The fourth is the geometry test. On Ant we compare the $E_8$, cubic and random candidate sets at 241 candidates each, and plot return against covering radius. The paper's central claim stands or falls on whether the $E_8$ advantage appears on Ant and shrinks on low-dimensional tasks.

We also report wall-clock cost per update against Soft Q-learning's sampler.