Is Model Collapse Partly a Sampling Failure?

When a generative model is retrained on its own outputs, generation after generation, it slowly forgets rare parts of the data. This failure is called model collapse. The theory behind it assumes that each generation's synthetic data is a perfect draw from the current model, so the forgetting comes only from fitting a model to a finite sample. Many real generators do not draw perfect samples. Energy-based models, Langevin samplers and particle methods produce samples by taking small local steps along the model's density. When modes are separated by low-density valleys, these samplers get stuck in whichever mode they start near. This is called local mode entrapment, a known problem in the sampling literature. Our hypothesis is that in such pipelines, entrapment rather than finite-sample error drives much of the observed collapse. If so, collapse can be slowed by changing only how synthetic data is drawn, with no fresh real data required.

The stakes are practical. Defenses against collapse currently depend on outside resources: fresh real data, an archive of all past data, or a trusted verifier model. Human-generated data is projected to run short (Will we run out of data?), so it matters how much collapse is self-inflicted at sampling time. This proposal measures that share and tests whether samplers that make long jumps between modes remove it. It also pins down which part of collapse no sampler can fix.

Core Claim

Each round of recursive training contains two separate losses. Write $p_{\theta_k}$ for the model at generation $k$, $q_k$ for the distribution of samples its sampler actually emits, and $d$ for a distance on distributions such as total variation over mode weights. The drift per generation then satisfies

$$ d(p_{\theta_{k+1}}, p_{\theta_k}) \le \underbrace{d(q_k, p_{\theta_k})}_{\text{sampler gap}} + \underbrace{d(p_{\theta_{k+1}}, q_k)}_{\text{estimation gap}} . $$

Existing collapse theory sets the sampler gap to zero and studies only the estimation gap. Rate of Model Collapse in Recursive Training shows that under maximum-likelihood refitting, a rare symbol survives for a number of generations roughly linear in its original count. It also shows that a fitted Gaussian's spread shrinks to zero after about $n$ generations, where $n$ is the per-generation sample size.

We make two predictions. First, the sampler gap grows with the height of the energy barrier between modes, while the estimation gap does not depend on barrier height. Second, with a local sampler and well-separated modes, the sampler gap dominates: modes vanish in far fewer generations than the linear-in-$n$ rate predicts.

The proposed fix is to generate each round's synthetic data with a tempered sampler that corrects its own errors. We use Accelerated Parallel Tempering via Neural Transports, which runs chains at several temperatures and uses learned neural maps to move states between them. A Metropolis–Hastings acceptance step (a rule that accepts or rejects each proposed move so the chain stays faithful to the target) keeps sampling exact in the limit. The fix should push collapse down to the estimation floor and no further. We treat that limit as part of the claim, not something to argue around.

Relation to Prior Work

Three lines of work motivate the gap by establishing collapse while idealizing the generator. Rate of Model Collapse and Strong Model Collapse analyze exact draws from linear or Gaussian models. The second paper shows that even 0.1% synthetic data puts a floor under scaling. Self-Consuming Generative Models with Curated Data proves that filtering synthetic data by reward collapses the model onto the highest-reward region present at initialization. None of these papers asks which modes the sampler can actually reach. That is where we start.

Two papers constrain the design. Analyzing and Mitigating Model Collapse in Reflow Methods found that adding stochastic noise to the reverse pass raises diversity but does not stop collapse, and that roughly 30% real data was still needed. Generic added noise therefore does not stand in for crossing barriers, and our local baselines must include noise-injected variants. Self-Verification Provably Prevents Model Collapse bounds error by filtering against a model trained on real data, but it relies on a calibrated verifier. We use no verifier.

The sampling literature supplies the method. Annealed SVGD recovers all 16 modes of a grid where ordinary Stein variational gradient descent finds 4. Stein variational gradient descent, or SVGD, moves a set of particles toward high density while a repulsive force keeps them apart. Continuously Tempered HMC and the transport-proposal methods (On Sampling with Approximate Transport Maps) attack the same entrapment problem in different ways. All of these evaluate a sampler on a fixed target. We put them inside a retraining loop, where any bias at one round becomes the target for the next.

The closest single-round analogue is Improved Autoregressive Modeling with Distribution Smoothing. It fills the valleys between modes by training on noise-blurred data and was never tested across generations. We include it as a competitor because it changes the model rather than the sampler.

Research Program

The core testbed is a set of ground-truth distributions whose modes are known exactly. These are Gaussian mixtures of increasing dimension, matching the GMM-d targets up to 100 dimensions used by the parallel-tempering paper; ManyWell-32, which has $2^{16}$ modes; and the 2D Rings and Two Moons shapes. On each, we train an energy-based model, meaning a network that assigns an unnormalized log-density to each point. We run recursive training in the "replace" setting, where each generation trains only on the previous generation's samples. The only thing that varies is how each generation's samples are drawn. The options are short-run and long-run Langevin dynamics, Langevin with extra noise, SVGD and Annealed SVGD, plain parallel tempering, and APT with Metropolis–Hastings correction.

Gaussian-mixture models fit by maximum likelihood can be sampled exactly. Running that exact sampler gives an empirical estimate of the estimation-only floor at each generation, which we check against the theoretical rates.

The key manipulation is mode separation at a fixed sample size $n$. Our first prediction is concrete. With the exact or APT sampler, the number of generations until a mode of initial weight $w$ disappears should not depend on separation and should scale roughly with $nw$. With local samplers, that number should fall sharply as separation grows. The second prediction is that accumulation and tempered sampling address different gaps and therefore combine. We test this by crossing sampler choice with replace versus accumulate, following Accumulating Data Avoids Model Collapse.

A secondary study addresses the tension between curation and verification in the file. We add top-$K$ reward curation to the loop and ask whether the reward mode the model collapses onto is the global one or merely the one the sampler can reach.

The stretch study moves to CIFAR-10 with a class-conditional energy-based model. We track the class histogram of generated samples with a fixed classifier, alongside FID (Fréchet Inception Distance, a standard image-quality score where lower is better) over ten generations.

Baselines

The naive reference point is the standard loop: refit on the previous generation's output, drawn with short-run Langevin initialized from noise. This is the regime in which we expect sampler-driven collapse to be strongest. No reported result found on Lacuna for mode survival of Langevin-sampled energy-based models under recursive retraining. We will establish this curve ourselves.

The estimation-floor reference is exact sampling from the fitted mixture. It is not a competitor. It marks the best any sampler can do, and the theory in Rate of Model Collapse predicts its shape.

The strongest local competitor is Annealed SVGD (Annealed SVGD), reported to capture 16 of 16 grid modes against 4 for ordinary SVGD on a fixed target. It is strong because it reaches all modes through annealing alone, without learned transports. We give it the same budget of energy and gradient evaluations per generation as APT, and the same tuning budget across its schedule options (linear, tanh and cyclical). If our approach beats it, the reason must be that it puts the correct weight on each mode, not merely that it reaches each mode.

The strongest non-sampling defense is data accumulation (Accumulating Data Avoids Model Collapse), for which no numerical mode-coverage result is reported on Lacuna. The strongest real-data defense is a 50/50 real/synthetic mix. On CIFAR-10, that mix keeps the class distribution roughly uniform where pure curated retraining collapsed to airplanes alone within 10 iterations (Self-Consuming Generative Models with Curated Data). These defenses use the same model and generation count but extra data, so they bound performance from above.

Two ablations isolate the key ingredient. The first is APT with the Metropolis–Hastings step removed, which takes the neural transport's samples as they are. The APT authors note that neural transports can themselves drop modes, so we expect this variant to collapse. The second is plain parallel tempering with the same temperature ladder but no transport.

Beating the competitors means two things at matched evaluation budgets. The total-variation error over mode weights at generation 10 must be closer to the exact-sampling floor than Annealed SVGD's is. And the dependence of collapse speed on separation must disappear.

Why not just…?

Why not just accumulate the data? Accumulation keeps the original real data in the pool, so no mode can vanish entirely. However, a trapped sampler keeps adding samples with the wrong weights, and the pool's mode weights drift away from the truth as those samples come to dominate. Accumulation also assumes the archive is kept, which Strong Model Collapse suggests is not enough once synthetic data dominates. This alternative is a real threat. If accumulation with Langevin matches accumulation with APT at generation 10, the practical case for our method narrows to the replace setting, and our sampler-crossed-with-data-management experiment will show this directly.

Why not just raise the temperature or run Langevin longer? Sampling at a higher temperature changes the target. It flattens the weights within each mode as well as between them, so the model learns a broadened distribution. Distribution Smoothing also notes that sampling poorly modeled low-density regions produces unrealistic samples. Longer chains help only when barriers are low, which is exactly what our separation sweep tests. Both variants run at the matched budget. If long-run Langevin closes the gap at high separation, our hypothesis is wrong.

Why not just initialize chains from the training data? Persistent chains started at data points stay in their starting modes. That preserves the empirical mode weights, but only by replaying the data, which makes it accumulation in disguise. We include it as a baseline. It will be informative if it matches APT in the replace setting, because that would mean entrapment matters only for noise-initialized generation.

Why not just smooth the training distribution? Smoothing changes the model so its valleys are shallower. Across generations, though, each blur compounds the previous one unless it is undone. We run single-round smoothing applied at every generation as a competitor, and measure both mode weights and how widely the model spreads mass across each mode.

Decisive Checks

The central experiment is a separation sweep on GMM-d with $d$ in {2, 10, 32, 100} and ManyWell-32. It plots generations-to-loss for the lightest mode against barrier height for each sampler. The hypothesis is confirmed if the local samplers' curves fall steeply while the APT and exact curves stay flat and agree to within seed variance. It is refuted if all curves are flat, which would mean collapse here is estimation-driven, or if APT collapses as fast as Langevin.

Secondary metrics are maximum mean discrepancy (a kernel distance between sample sets) to the ground truth, the number of surviving modes, and mixing round trips. Round trips connect our results to the up-to-100x gain over plain parallel tempering reported on GMM-10 in Accelerated Parallel Tempering.

On CIFAR-10, the check is whether the class histogram under APT generation stays as close to uniform as the 50/50 real-data mix with no real data added. Success would show that a sizable share of self-inflicted collapse comes from samplers. Even a negative result would cleanly confirm the idealized theory's assumption, which is still useful.