Reinforcement learning agents that learn from raw physical state often waste millions of environment steps partly because their internal state representation collapses. The learned features shrink onto a few directions, so the value function has little to work with. iQRL showed that forcing the latent state onto a small grid of values prevents this collapse and makes learning on DeepMind Control locomotion markedly more sample-efficient. It uses Finite Scalar Quantization, which bounds each latent dimension and rounds it to one of a few levels. To train through the rounding step, however, iQRL uses the straight-through estimator: on the backward pass it pretends rounding is the identity function. Work on binarized networks shows this shortcut distorts the geometry of the discrete space and often fails to move any code at all (BOLD-SAM). Analyses of quantized learning dynamics show it produces long stretches where error stalls before dropping (High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator). In reinforcement learning every stalled stretch is paid for in environment samples.
We propose to drop the pretend-gradient and optimize the discrete latent codes directly. On each training minibatch, the quantized code of every sampled state is treated as an explicit discrete variable. A few sweeps of coordinate descent then lower a self-predictive objective: one dimension changes at a time, and a change is kept only if it provably reduces the loss. The encoder network is then trained by ordinary regression toward these refined codes. The hypothesis is that direct discrete targets shorten the stall phases that the straight-through estimator induces and keep the representation from collapsing at high update-to-data ratios, meaning many gradient updates per environment step. Together these should lower the environment steps needed to reach a given return. The code-refinement step also has a precise local-optimality notion: no single-coordinate change improves the objective. This lets us measure how far training is from a discrete stationary point rather than assume it.
Core Claim
The claim is that iQRL-style quantized representations are held back by how they are trained, and that the fix comes from the hashing literature. Deep hashing methods use asymmetric alternating optimization: discrete codes are solved by exact per-bit coordinate updates, and continuous networks are fit to them by backpropagation. Column Sampling Based Discrete Supervised Hashing and Discrete Image Hashing Using Large Weakly Annotated Photo Collections show this avoids the information loss of relax-then-round. Learning Binary Codes for Maximum Inner Product Search shows closed-form bit updates can match continuous first-order methods in runtime. Our work moves that scheme from static retrieval into the self-predictive representation learning inside an off-policy actor-critic. It also extends it from binary codes to multi-level scalar codes. BOLD-SAM's Boolean-hypercube machinery explicitly does not cover multi-level codes, and the file lists this as open.
Concretely, let $c_t \in \mathcal{L}^d$ be the code for state $t$, where $\mathcal{L}$ is a small set of levels (for example five) and $d$ is the latent width. Let $e_\theta$ be the encoder and $h_\phi$ the latent dynamics model. For a sampled sub-trajectory of length $H$ we minimize
$$J(\theta,\phi,C)=\sum_{t}\big\|h_\phi(c_t,a_t)-c_{t+1}\big\|^2+\lambda\,\big\|e_\theta(o_t)-c_t\big\|^2 ,$$
where $o_t$ is the observation, $a_t$ the action, and $\lambda$ trades off predictability against fidelity to the encoder. With $\theta,\phi$ fixed, $J$ takes finitely many values over $C$. Any update that accepts only strict decreases therefore terminates at a code where no single-coordinate change helps. That point is a 1-neighborhood local optimum, which is the discrete counterpart of a stationary point.
Enumerating every level for every coordinate would need $|\mathcal{L}|\cdot d$ dynamics forward passes, which is too many. Instead, each sweep scores candidate changes with a cheap first-order screen and proposes the top $m$. It keeps each one only after exact evaluation of $J$, so descent stays monotone. The critic and actor consume $\mathrm{round}(e_\theta(o))$ exactly as in iQRL. Only the representation's training target changes.
We do not claim a sample-complexity theorem. Uniform Convergence of Gradients for Non-Convex Learning and Optimization proves dimension-free sample bounds for smooth models and a dimension-dependent lower bound for non-smooth ones. Quantized objectives fall on the non-smooth side, so any efficiency gain here is empirical. Zuo et al. show that exact scalar subproblem solutions still trap coordinate methods at joint stationary points. We accept that 1-neighborhood optimality is a weak guarantee and plan to measure how often the refinement actually stops there.
Research Program
The work has three parts.
Controlled supervised testbed. Before touching reinforcement learning, we reproduce the teacher-student regression setting of High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator. That paper documents a "plateau and drop" pattern and reports ODE predictions matching simulations within 1% relative error. We then replace the straight-through update of the quantized hidden layer with our alternating code refinement. The question is narrow: does the plateau shorten, measured in samples, at matched learning rate and compute? If it does not, the reinforcement-learning version has no mechanism behind it and we stop.
Reinforcement-learning integration. We build the method on the public iQRL/TD3 setup using dm_control. Replay minibatches of length-$H$ sub-trajectories are initialized at $\mathrm{round}(e_\theta(o_t))$. We refine them with $S$ sweeps, where $S$ is small (1 to 3), and then update the encoder, dynamics, and critic. We log three things throughout. The discrete stationarity gap is the fraction of coordinates in a minibatch whose best single change would still lower $J$ after refinement. Code usage is the effective rank and per-level occupancy of the latents. Code churn is the Hamming change between successive targets for the same stored transition.
Stress at high update-to-data ratio. R2R2 shows self-predictive features collapse at UTD 20 and fixes this with a non-centered decorrelation loss. We test whether direct discrete targets alone remove the need for that loss, and whether they hold up past UTD 20. R2R2 reports that regime as still degrading.
Baselines
The naive reference point is plain TD3 with no representation loss, run from the same codebase. It shows how much any self-predictive latent helps at all.
The primary competitor is iQRL itself: Finite Scalar Quantization latents trained by the straight-through estimator with the same TD3 backbone. It isolates exactly the variable we change. We run it with its released hyperparameters and give it the same tuning budget we give our method: a sweep over quantization levels and latent width. We also give it matched wall-clock by letting it take extra gradient steps equal to the cost of our refinement sweeps. The file reports that iQRL solves Dog Run where most baselines fail, but gives no numeric return. We therefore record "no reported result found on Lacuna for iQRL episodic returns on DMC" and will rerun it ourselves on all 20 tasks with shared seeds. Beating it means a higher interquartile-mean normalized return at 500k and 1M environment steps across the 20 DMC tasks, with non-overlapping stratified bootstrap intervals. Equivalently, fewer environment steps to reach iQRL's final return.
The strongest existing method for the high-UTD question is R2R2. The file reports that it raises DMC-Hard normalized score from 1.02 to 1.32. It also reports that SimbaV2-SPL with R2R2 reaches 1.38 against SimbaV2's 1.20 at high UTD, and that R2R2 improves TD7 aggregate score by about 22% at UTD 20. We compare on DMC-Hard at UTD 20. First we compare within the same TD7-style backbone. Then we run a variant that adds our code refinement on top of SimbaV2-SPL, to see whether it composes with or replaces the decorrelation loss. Beating it means exceeding 1.32 in the matched backbone, or 1.38 when added to SimbaV2-SPL, at no more than R2R2's reported step-time overhead of 0.210 to 0.241 seconds per step.
Three ablations isolate the key ingredient. Zero sweeps sets the targets to the rounded encoder outputs, which is effectively iQRL's target with a stop-gradient. Random-proposal sweeps keep only candidates that lower $J$, but choose candidates at random rather than by screening. Unverified sweeps accept screened proposals without the exact check. The first ablation tests whether refinement matters at all. The second tests whether the screening signal matters. The third tests whether monotone descent is doing the work.
Why not just…?
Why not just keep the straight-through estimator and spend the extra compute on more updates? This is the most likely reason the method could fail. iQRL already works well, and extra gradient steps are the cheapest competitor. The evidence against it being sufficient is indirect. BOLD-SAM shows that continuous proxy updates stop flipping discrete values as proxy magnitudes grow. The quantized-dynamics analysis shows stall phases that more steps prolong rather than skip. The matched-wall-clock iQRL baseline settles this directly. If extra straight-through steps close the gap, we report that direct discrete optimization offers no benefit here.
Why not just use R2R2's continuous decorrelation loss and skip quantization? R2R2 directly targets collapse at high UTD and has the strongest reported numbers. It prevents collapse through a penalty on feature covariance, though, and it does not address the stall phases tied to quantization. Its own authors report degradation beyond UTD 20. Our UTD sweep at 20, 40, and 80, plus the combined SimbaV2-SPL variant, will show whether discrete refinement is a substitute, a complement, or neither. Substitution is not assumed.
Why not just relax the codes to continuous values and round at the end, perhaps with a carefully designed penalty? Relax-then-round is exactly what hashing work found to shatter learned geometry at longer code lengths (Discrete Image Hashing). Local Minima in Quadratic-Penalty Relaxations of Binary Linear Programs shows naive penalty relaxations often end at fractional or infeasible points, with 0% feasible solutions in their tests. Its fix, the diagonal-free, integer-gradient, and repairability conditions, applies to binary linear programs. It does not apply to a learned nonlinear dynamics loss, and exact penalties can require multipliers too large for 32-bit arithmetic. We still include a soft-rounding relaxation baseline in the supervised testbed, at equal tuning budget, so this claim is checked rather than asserted.
Proposed Experiments
The decisive checks run in order, and each can stop the project.
The first is the supervised plateau test on the Diabetes and California Housing regressions from the quantized-dynamics paper. Our method must shorten the stall phase in samples, at matched learning rate, relative to both the straight-through estimator and soft rounding.
The second is the 20-task DMC comparison at UTD 1 against TD3, iQRL (including its matched-compute version), and the three ablations. We use at least five seeds per task and report interquartile-mean return, steps-to-threshold, effective latent rank, and stationarity gap. The main result stands only if our method beats matched-compute iQRL and the zero-sweep ablation.
The third is DMC-Hard at UTD 20, 40, and 80 against R2R2 at its reported 1.32 and 1.38 operating points. We also track code churn, to test whether stable discrete targets explain any robustness at extreme update ratios.
Across all runs we correlate the stationarity gap with learning progress. This tests whether reaching discrete local optima tracks sample efficiency, or whether, as Zuo et al. would predict, coordinate-wise optimality is reached early and says little about final performance.
References
- iQRL - Implicitly Quantized Representations for Sample-efficient Reinforcement Learning
- Sharpness-Aware Minimization Directly on the Boolean Hypercube
- High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator
- Column Sampling Based Discrete Supervised Hashing
- Discrete Image Hashing Using Large Weakly Annotated Photo Collections
- Learning Binary Codes for Maximum Inner Product Search
- Uniform Convergence of Gradients for Non-Convex Learning and Optimization
- A Generalized Iterated Shrinkage Algorithm for Non-convex Sparse Coding
- DeepMind Control Suite
- R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning
- Local Minima in Quadratic-Penalty Relaxations of Binary Linear Programs