Doubly Reparameterized Gradients for Particle Filters, and Whether They Cure Forgetting Under Occlusion

Sequential latent variable models track a hidden state through time with a particle filter. A particle filter is a set of $K$ weighted guesses ("particles") that are propagated forward frame by frame and periodically resampled so that bad guesses die and good ones multiply. These models are trained by maximizing a variational bound, a lower bound on the data's log-likelihood that gets tighter as $K$ grows. The trouble is well documented. Adding particles tightens the bound but drowns the learning signal for the proposal network, the network that decides where to place new particles. Its gradient's signal-to-noise ratio (SNR) falls as $O(1/\sqrt{K})$ (Auto-Encoding Sequential Monte Carlo). For static, single-step models this was fixed by the doubly reparameterized gradient estimator (DReG), which reverses the scaling to $O(\sqrt{K})$ at no extra cost. Its authors explicitly left the extension to particle filters open (Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives).

This note proposes to derive that extension and to test a specific consequence. Particle-filter models of video are known to "forget" objects during occlusion. We argue that part of this temporal context neglect is an optimization failure rather than an architectural one. When the proposal's gradient is mostly noise, the proposal cannot learn to use its temporal inputs (the carried-over state and neighbouring frames), and it falls back toward something close to the prior. If that is right, a sequential DReG estimator should make the learned proposal measurably more reliant on temporal context as $K$ grows, and the gradient hacks in use today should not.

Core Claim

We make two claims. The first is technical. The proposal gradient of a resampled sequential Monte Carlo (SMC) bound splits into three parts: a pathwise term through each particle's sample, a "direct score" term from $\log q_\phi$ inside each step's importance weight, and a discrete score term from the resampling step. The direct score terms can be doubly reparameterized step by step, and this removes the within-step source of SNR collapse.

For reference, the single-step DReG estimator for a $K$-sample bound replaces the high-variance score term with a purely pathwise one: $$\nabla_\phi \approx \sum_{i=1}^{K} \tilde w_i^{2}\,\frac{\partial \log w_i}{\partial z_i}\,\frac{\partial z_i}{\partial \phi},$$ where $z_i = g_\phi(\epsilon_i)$ is the $i$-th reparameterized sample, $w_i$ its importance weight, and $\tilde w_i = w_i/\sum_j w_j$ its normalized weight. In the sequential case, a particle's sample at time $t$ also affects its descendants' future weights. The squared-weight coefficient therefore has to be generalized to the particle's normalized weight times the credit it receives from its descendants. Deriving that coefficient exactly is the first deliverable.

We will produce two estimators. The fast estimator keeps the standard practice of treating resampled ancestor indices as constants and applies double reparameterization only to the direct score terms. It is as cheap as current practice and inherits a bias of its own, which we will characterize. The exact estimator additionally keeps the resampling score, which enters through the derivative of the downstream objective with respect to each particle, and tames it with a leave-one-out control variate. It is unbiased for the true SMC bound gradient.

The second claim is empirical and falsifiable. Under standard or hacked gradients, the proposal's sensitivity to temporal context should fall as $K$ increases. Under sequential DReG it should rise. That difference should track occlusion-recovery performance.

Where This Sits

Three papers motivate the gap by establishing that particle-filter bounds suffer SNR collapse and that practitioners work around it with biased hacks. Auto-Encoding Sequential Monte Carlo (AESMC) drops the resampling score terms and, because the proposal still starves at large $K$, alternates a large particle count for the generative model with a small one for the proposal. The authors call that schedule unstable on complex tasks. Variational Sequential Monte Carlo (VSMC) uses the same hack and shows the retained reparameterization term tracks the full gradient with much lower variance. Smoothing Nonlinear Variational Objectives with Sequential Monte Carlo (SVO) conditions proposals on future observations and again relies on dropped ancestor terms. None of the three touches the direct score term that DReG targets. Our fast estimator composes with their hack rather than replacing it.

The method comes from Tucker et al., which derived DReG for single-step bounds on MNIST and Omniglot. It also showed that sticking-the-landing, which simply drops the score term, carries a persistent squared bias. Generalized Doubly Reparameterized Gradient Estimators (GDREGs) extends the idea to hierarchical models and trainable priors via a "re-expression" trick, with under 10% overhead. We reuse that trick for the transition prior's parameters.

GDREGs also constrains the design. When posterior and prior factorize differently, some indirect score functions cannot be removed. SVO-style smoothing proposals, which condition on the future while the transition prior does not, are exactly such a mismatch. We therefore restrict the main claims to filtering proposals and treat smoothing as a measured extension, reporting how much residual score variance remains.

Tighter Variational Bounds are Not Necessarily Better supplies the $O(1/\sqrt{K})$ analysis we aim to invert. On the difficulty of unbiased alpha divergence minimization shows noise compounding multiplicatively across dimensions. Plain importance sampling over a length-$T$ sequence is such a product. The time-horizon analogue, whether per-step DReG plus resampling keeps SNR from decaying in $T$, is directly measurable and is part of the program.

On the vision side, Predicting Video Slot Attention Queries from Random Slot-Feature Pairs found that object-tracking transition models ignore next-frame features and fixed this architecturally in a deterministic model. We ask whether a similar neglect in stochastic models has an optimization cause. OpenSTL notes that current spatio-temporal predictors are deterministic and calls for stochastic ones, which is the regime where multi-sample bounds and their gradient pathologies appear.

Research Program

The first stage is derivation. We will write the SMC bound gradient with respect to the proposal parameters as the sum of the three parts above, apply the reparameterization identity to each step's direct score term, and obtain the descendant-credit coefficient. We will implement both estimators as surrogate losses in the style of GDREGs, building on the AESMC codebase tuananhle7/aesmc.

The second stage is gradient-level measurement on a linear Gaussian state-space model, where the Kalman filter gives the exact marginal likelihood and the locally optimal proposal. There we will estimate the true expected gradient from a very large number of unbiased replicates. We will measure bias, variance and SNR for every estimator as functions of $K$ (1 to 1000) and sequence length $T$ (10 to 500). The time-horizon question is tested here: the prediction is that per-step DReG SNR grows with $K$ and stays flat in $T$ under resampling. Plain importance sampling over the whole sequence, even with DReG, should decay in $T$.

The third stage is the temporal-context diagnostic. We define the proposal's context reliance as the average KL divergence between its output given true inputs and its output when the carried-over state is replaced by a draw from the prior or neighbouring frames are masked. We track this quantity through training and relate it to state-tracking error during occlusion.

Baselines

The naive reference is the full unbiased gradient of the SMC bound, including resampling score terms, with no variance reduction. It is what "do nothing principled" means here and shows how bad unmodified variance is.

The strongest existing method is the VSMC/AESMC gradient hack, meaning reparameterization terms only with resampling scores dropped, run both with a single $K$ and with AESMC's alternating schedule. It is the field default and the only thing reported to train these models stably. On the stochastic volatility model, VSMC reports a bound of 6936.6, against 6913.3 for importance-weighted training and 6905.1 for structured variational inference. We will first reproduce 6936.6 in our code. Beating it means a higher test bound at matched $K$ and wall-clock time, with a consistent sign across at least five seeds. On the Moving Agents video data, no reported result was found on Lacuna for AESMC as a scalar; its figures are qualitative. We will therefore rerun AESMC ourselves under identical budgets.

We also include DReG applied to plain importance sampling over whole sequences, as in Tucker et al.. This separates the benefit of the estimator from the benefit of resampling. Two further baselines are sticking-the-landing applied per step inside the hack, and a score-function estimator with control variates in the spirit of Optimal Variance Control of the Score-Function Gradient Estimator for Importance-Weighted Bounds.

Two ablations isolate the key ingredient. One applies DReG to the proposal only, versus the proposal plus the transition prior via re-expression. The other compares the fast and exact estimators, which isolates what keeping the resampling score buys.

All methods share architectures, data, particle counts and a fixed tuning budget: the same learning-rate sweep, with seeds and early stopping selected on validation bound.

Why not just…?

Why not just keep the gradient hack? VSMC's evidence that the hack tracks the full gradient is real, and it is our strongest competitor. However, the hack leaves the direct score term untouched. AESMC reports that proposal SNR still collapses at $O(1/\sqrt{K})$ under it, which is why they needed alternating particle counts. Our fast estimator is the hack plus one substitution, so the comparison is clean. If the proposal SNR curves on the linear Gaussian model do not separate, the proposal fails.

Why not just drop the score term per step, as sticking-the-landing does? This is the most serious threat. Since the hack is already biased, the extra bias from sticking-the-landing might not matter in the sequential setting. Tucker et al. showed its bias plateaus optimization in the single-step case, but that result need not transfer. The linear Gaussian experiment measures both estimators' bias against the true gradient and their final proposal quality against the known optimal proposal. If sticking-the-landing matches DReG there and on Moving Agents, we will report that the cheaper fix suffices.

Why not just use a deterministic recurrent-free predictor? OpenSTL shows such models are fast and robust to dropped frames. They answer a different question, however: they cannot represent multiple plausible hidden states behind an occluder, and OpenSTL itself flags determinism as the limitation. We include a SimVP-style predictor on Moving Agents as a point reference for tracking error. We do not claim to beat it on throughput.

Why not just use small $K$ for the proposal, as AESMC's alternating schedule does? This is included as a baseline. We will report its sensitivity across the shared tuning sweep, since instability is the documented failure.

Proposed Experiments

The linear Gaussian state-space model is the decisive gradient-level test. It measures SNR against $K$ and $T$, bias against a ground-truth gradient, and KL divergence from the learned proposal to the Kalman-optimal proposal. A successful result shows sequential DReG SNR rising with $K$ while the hack's falls, at negligible bias for the exact estimator.

The stochastic volatility model tests whether better gradients translate into a tighter learned bound against the 6936.6 reference.

The Moving Agents video dataset (5,000 sequences with partial occlusion, from AESMC) tests the context-neglect hypothesis. We will measure test bound, agent-position error during and after occlusion, and the context-reliance diagnostic. The decisive check is the interaction between particle count and estimator. If context reliance and occlusion recovery improve with $K$ under sequential DReG but degrade or stay flat under the hack and sticking-the-landing, the optimization account of forgetting is supported. If all estimators show the same trend, forgetting is architectural and our second claim is refuted, whatever happens to the SNR curves.

As a stated secondary test, we will run SVO's Fitzhugh–Nagumo task with smoothing proposals and report how much residual score variance the GDREGs factorization mismatch leaves.