Reading Memory Policies Through Frozen Reservoirs

Agents that act under partial observability, meaning they cannot see the full state and must remember what happened earlier, are the hardest agents to explain. The interpretable reinforcement learning methods that compile policies into decision trees or short programs assume the current observation is all the agent needs. Recurrent networks, which carry memory in a learned hidden state, are opaque: their hidden units mean nothing on their own, and the meaning shifts every time the weights are updated. This proposal starts from a property of reservoir computing that has not been used for interpretation. A reservoir is a large recurrent network whose internal weights are random and never trained; only a linear readout on top of it is learned. Because the recurrent dynamics are frozen, any linear readout of the reservoir state can be rewritten exactly, or for a nonlinear reservoir approximately, as a filter over the agent's past observations. In plain terms, it becomes a weighted sum of "what I saw one step ago, two steps ago, and so on."

The proposal is to distill a reservoir-based agent into a small decision tree whose tests are sparse linear readouts of the reservoir. Each test is then compiled into an explicit temporal rule over the raw observation history, such as "if cart position now minus cart position one step ago exceeds 0.02, push left" or "if the decaying average of reward from arm 3 exceeds that of arm 1, pull arm 3." The result is a memory-dependent policy that can be read and edited like a program. Its faithfulness to the original agent can be measured directly, and its memory is expressed in the agent's own observations rather than in hidden units.

Core Claim

For a linear reservoir with state $x_t = W x_{t-1} + W_{in} u_t$, where $u_t$ is the observation at time $t$, $W$ the fixed recurrent matrix and $W_{in}$ the fixed input matrix, any readout $w$ satisfies

$$w^\top x_t = \sum_{\tau \ge 0} k_\tau^\top u_{t-\tau}, \qquad k_\tau = W_{in}^\top (W^\top)^{\tau} w .$$

Here $k_\tau$ is the "temporal kernel": how much each observation feature at lag $\tau$ contributes to the test. Under the fading-memory condition (spectral radius $\rho(W)<1$), the kernel decays roughly like $\rho(W)^\tau$, so truncating it at lag $L$ has an error that shrinks geometrically in $L$. This rate follows directly from the matrix power in $k_\tau$. For the usual leaky-tanh reservoir, we replace the exact kernel with the average input-output Jacobian $k_\tau \approx \mathbb{E}[\partial (w^\top x_t)/\partial u_{t-\tau}]$ over states the agent visits, and measure how much that approximation costs.

The claim is threefold. First, oblique trees with 16 to 64 nodes and sparse reservoir-readout splits can match the returns of an opaque reservoir agent on memory tasks. Second, their compiled lag-kernel form preserves that return. Third, on tasks with long or variable delays, this beats the obvious interpretable alternatives, which are trees over stacked recent frames and trees distilled from trained gated recurrent networks. None of this is possible with a trained recurrent network in the same form: its dynamics are nonlinear gates whose Jacobians vary sharply with state, so no single global kernel exists.

Relation to Prior Work

Three papers motivate the gap. Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks uses a fixed echo state network (the standard discrete-time reservoir) as the memory of a reinforcement learning agent. On Recall Match it reaches peak performance in about 5,000 episodes against nearly 50,000 for an LSTM, but its decision layer is an opaque multilayer perceptron and it never analyzes what the reservoir encodes. We keep its training recipe and replace the opaque head with a compiled tree. The interpretable-policy methods Imitation-Projected Programmatic Reinforcement Learning, Programmatic Reinforcement Learning without Oracles and Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning (PROPEL, π-PRL, INTERPRETER) each produce readable policies, but only over the current state. We extend the INTERPRETER-style oblique tree to history by giving it reservoir features and a way to read them back.

Two papers constrain the design. Discovering Dynamical Parameters by Interpreting Echo State Networks shows that readout weights of a shared frozen reservoir are meaningful: principal components of the Lorenz readouts correlate with σ, ρ, β at R² of 0.993, 0.996 and 0.925. That work interprets readouts statistically across many trained models of passive physics. We interpret a single readout mechanistically, by back-projecting it through the frozen dynamics, and we do it for actions rather than forecasts. Extracting Finite Automata from RNNs Using State Merging extracts exact automata from trained recurrent networks, but only after overtraining compresses hidden states into clusters. A frozen reservoir never undergoes that compression, so we deliberately do not attempt state-merging extraction over raw reservoir states. We extract temporal filters instead, which need no clustering.

Finally, Learning Reward Machines for Partially Observable Reinforcement Learning represents memory as a discrete automaton learned by Tabu search, but requires hand-built event detectors. Compiled reservoir splits are candidate learned event detectors. That link appears here only as a secondary stage, because fading memory cannot by itself hold an event indefinitely.

Research Program

The pipeline has four steps. First, train a reservoir agent with the recipe of McKee: a fixed sparse reservoir of about 500 to 2,000 units, and a PPO-trained head. We use two reservoir variants, a standard leaky-tanh echo state network and a linear reservoir with complex-eigenvalue spectrum, so the exact kernel identity holds.

Second, distill the head by DAgger-style imitation into an oblique tree. The distillation loop repeatedly collects states under the student and labels them with the teacher's actions. Each split in the tree is a readout $w$, and the sparsity penalty is placed on the kernel rather than on $w$. Because $k_\tau$ is linear in $w$, a group-lasso over (feature, lag) blocks of the truncated kernel is a convex penalty. This makes each split depend on a few observation features over a short band of lags, or on a smooth decaying average.

Third, compile each split into its truncated kernel. We then fit each kernel to a small readable family: a single-lag tap, a first difference, or an exponential average with a stated half-life. The compiled program is re-evaluated in the environment, so we measure the fidelity of what a human would actually read, not of the intermediate tree.

Fourth, for tasks whose memory must persist indefinitely, add a stage in which the tree may set and test a small number of discrete flags. The flags are triggered by compiled splits, in the spirit of the switching controller of Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machines. This turns compiled splits into the event detectors that reward machines lack.

The analysis questions are: how large the gap is between the opaque agent, the reservoir-split tree and the compiled program; how that gap depends on reservoir nonlinearity and spectral radius; and whether compiled kernels recover the structure a human would expect. Examples of expected structure are a finite difference on Stateless CartPole, and taps at lags 2 and 4 on Recall Match.

Baselines

The naive reference points are a memoryless oblique tree over the current observation and the memoryless MLP used in POPGym. POPGym: Benchmarking Partially Observable Reinforcement Learning reports that memoryless MLPs solve navigation mazes through wall-following. We therefore treat navigation tasks only as sanity checks. A win on a task counts only if the memoryless tree clearly fails there.

The upper reference is the opaque reservoir agent itself, trained as in McKee, alongside a GRU agent. POPGym reports the GRU had the highest mean performance across its tasks. No per-environment numerical return was found on Lacuna for POPGym, so we rerun it ourselves. The only reported number for the strongest memory-based competitor is from McKee on Recall Match: roughly 5,000 episodes to peak for ESN agents versus nearly 50,000 for LSTMs.

The strongest interpretable competitor is a GRU agent distilled into an oblique tree whose splits act on GRU hidden states, using the same DAgger loop and node budget. Its splits are opaque, so we also give it the best available readback: gradient attribution of each split onto input history, averaged over visited states. This is exactly what the Jacobian kernel does for tanh reservoirs. If a trained GRU's averaged attributions are as faithful as the reservoir's kernels, the frozen-dynamics argument fails.

The second strong competitor is INTERPRETER-style oblique trees over the last $k$ stacked observations, with $k$ tuned per task. A third is a tree over a bank of leaky integrators at log-spaced timescales, which is a diagonal real-valued reservoir. It isolates whether random recurrent mixing, with its oscillatory, delay-like kernels, adds anything beyond exponential averages.

Key-ingredient ablations remove kernel-space sparsity (plain L1 on $w$) and remove compilation (report the tree over raw reservoir units). The latter shows how uninterpretable the uncompiled splits are.

All methods share environment steps for the teacher, the same number of DAgger iterations, identical node budgets of 16, 32 and 64, and an equal hyperparameter search budget, with at least five seeds. Beating a baseline means the compiled program reaches higher normalized return at the same node budget and at most the same number of (feature, lag) taps per split, with non-overlapping seed confidence intervals.

Why not just…?

Why not just stack the last k frames and fit an interpretable tree? On short fixed-lag tasks like Recall Match, with lags 2 and 4, this will probably tie or win, and we expect to report that. The threat is real only where the needed lag is long or variable. There, stacking multiplies the input width by $k$, and π-PRL already reports that program search becomes intractable beyond depth 6 as the space grows. The Repeat First and bandit experiments settle this by sweeping the delay and plotting return versus required window. If stacking matches the reservoir at every delay within the same node and tap budget, the reservoir contributes nothing to interpretability and we will say so.

Why not just distill a trained GRU into a tree? GRUs are the strongest memory models in POPGym, so their teacher will likely be at least as good. The question is what the tree's splits mean. Merrill and Tsilivis show discrete structure emerges only after overtraining, and GRU attributions are state-dependent rather than global. The GRU-attribution baseline tests directly whether the frozen-dynamics readback is more faithful. The measure is the return drop from tree to compiled program at equal budget.

Why not just use a bank of exponential averages? This is the most dangerous alternative on reward-tracking tasks, where the ideal memory really is an exponential average. The leaky-integrator ablation is included for exactly this reason. We predict it wins or ties on the bandit task and loses on tasks needing delays or differences, such as Recall Match and Stateless CartPole, because sums of real exponentials represent delayed taps poorly. If it ties everywhere, the correct conclusion is that interpretable memory needs only a fixed filter bank, which is still a useful negative result.

Why not just hand-write a reward machine? Toro Icarte et al. note that this needs domain expertise and predefined detectors. The flag stage tests whether compiled splits can supply those detectors on the Cookie and 2-Keys domains. Their reported result is that LRM with Q-learning achieves near-optimal rewards; no numerical value was found on Lacuna.

Proposed Experiments

The core suite is the diagnostic tasks from McKee (Recall Match, Multi-armed Bandit, Sequential Bandits) and POPGym's Repeat First, Autoencode and Stateless CartPole, with Labyrinth Explore as a memoryless sanity check. The latching suite is the Cookie and 2-Keys domains from LRM. The metrics are normalized return of the teacher, the tree and the compiled program; the distillation gap between them; node count and taps per split; and a fidelity score, defined as action agreement between program and teacher on held-out episodes. We also test generalization to delays and episode lengths longer than those seen in training.

There are four decisive checks. First, on Stateless CartPole, a compiled split must recover a first-difference kernel on position, and editing its threshold must change behavior predictably. Second, on Recall Match, the kernels must concentrate at lags 2 and 4. Third, on Repeat First with delays swept beyond any tuned stacking window, the compiled program must keep its return while the stacked-frame tree collapses. Fourth, the tanh-reservoir Jacobian kernels must lose less return under compilation than GRU attribution kernels at equal budget.

Failure on the fourth check, with success on the linear reservoir, would still show that linear frozen memory yields exact readable policies. Failure on the third would show that, at these horizons, reservoirs buy speed of training but not interpretability.