When a model is repeatedly retrained on data that includes its own synthetic outputs, keeping all past data prevents "model collapse," the slow loss of diversity until the model forgets what real data looks like. Kazdan et al. showed this directly: their "accumulate" workflow stays stable, while discarding history ("replace") collapses. Accumulation, however, needs unbounded storage and compute. Under a fixed training budget, the same paper subsamples the growing pool uniformly at random, and performance degrades to a stable but worse plateau (Collapse or Thrive). We propose closing that plateau by deciding which items to evict from a fixed-size training buffer as new synthetic data streams in. An incoming synthetic sample may displace a buffered item only if it improves how well the buffer covers a small, held-out set of real "anchor" examples by more than a rising threshold. This is the thresholded preemption rule from streaming submodular optimization, here applied with a coverage target defined by real data rather than by generic diversity.
The same mechanism also gives a way to evaluate synthetic data. The admission threshold that a synthetic sample must clear, compared with the marginal value of a real sample, is a running estimate of the exchange rate between synthetic and real data. That is the quantity Chang et al. measure once, offline, as roughly 30:1 for tracking frames, and which they flag as poorly understood (On the Equivalency, Substitutability, and Flexibility of Synthetic Data). We test whether this online estimate predicts the substitution ratios obtained by actually retraining.
Core Claim
Under a fixed buffer size in a recursive synthetic-data loop, the eviction policy is the main factor behind the gap between bounded training and unbounded accumulation. Two choices close most of that gap: an objective that rewards covering held-out real anchors, and threshold-gated swapping. Generic diversity, point-wise quality filters, and uniform subsampling do not close it. The anchor-coverage objective matters most, and we isolate it explicitly. A secondary claim is that the ratio of admission thresholds for synthetic and real items tracks the empirical exchange rate as it changes during training. In particular, it should fall as real data grows past the "pollution threshold" that Kazdan et al. observe, where synthetic data stops helping.
Prior Work and Positioning
Three lines of work supply the method. Thresholded preemption lets a streaming algorithm undo earlier choices without thrashing: Online Submodular Maximization with Preemption shows revocation breaks impossibility barriers; Data Summarization at Scale swaps summary elements when marginal gain clears a threshold, with a 1/7 guarantee; and Streaming Algorithm for Monotone k-Submodular Maximization maintains per-category rising thresholds with free disposal, evicting the earliest item for proof reasons even though evicting the lowest-gain item may work better in practice. We reuse their swap rule. Their objectives are fixed functions over real data. Our objective is anchored to real data while the stream itself is synthetic and produced by the model being trained.
Two papers motivate the anchor-coverage choice and also constrain it. DiSF selects pre-training files offline by maximizing feature diversity, avoiding the "dimensional collapse" (embeddings squeezed into a narrow subspace) that similarity-based filters cause. It reports a 3.4% gain over DSIR on TinyLlama 1.1B (Combatting Dimensional Collapse). Diversity alone, though, rewards synthetic samples that are spread out but lie off the real distribution, which is exactly the failure a recursive loop produces. Model Collapse Despite Ground-Truth Verification proves that statistics computed only on source data, including perfect verifier pass rates, cannot detect target modes with no support. Its risk bound charges for this unsupported target mass. This rules out point-wise filters such as the CLIP-based SDS pipeline (A Training-free Synthetic Data Selection Method) as a complete answer. It also tells us what our objective must measure: coverage of real target samples. It also limits our claim. If the real anchors themselves miss a mode, nothing in our method can recover it, and we say so.
The nearest operational neighbor is gradient-coreset replay for continual learning. It beats reservoir sampling by 2–4% offline and up to 5% in one-pass streaming (GCR). GCR curates real data under task shift. It has no notion of synthetic provenance, real anchors, or self-consumption, but it is a credible competitor and we run it.
Research Program
The buffer $B$ holds at most $N$ items, real or synthetic, and is what the model trains on each generation. A held-out anchor set $R$ of real examples is never trained on. It is used only to score coverage. The objective is
$$F(B) = \sum_{r \in R} \max_{b \in B} s(r, b) \;+\; \lambda \, D(B),$$
where $s(r,b)$ is a nonnegative similarity between anchor $r$ and buffer item $b$ in a feature space, $D$ is a diversity term in the style of DiSF, and $\lambda \ge 0$ trades the two. The first term is a facility-location function. It is monotone and submodular, and it acts as a computable stand-in for one minus the unsupported target mass of Liu's ledger. The buffer keeps separate rising thresholds for real and synthetic items, $\phi_{\text{real}}$ and $\phi_{\text{syn}}$, following Ene and Nguyen's per-category scheme. An arriving item of type $c$ is admitted if its marginal gain exceeds $\phi_c$. When the buffer is full, the admitted item evicts a current item: either the earliest-added one, which keeps the theory intact, or the one with the smallest loss in $F$, which is the empirical variant. We compare both and so test the open question about disposal order. The ratio $\phi_{\text{syn}}/\phi_{\text{real}}$, together with admission counts, gives the running exchange-rate estimate.
Features for $s$ come either from a frozen proxy encoder or from the current model's own representations. Both carry risk. Proxy encoders inherit their own blind spots. The model's own features drift as it collapses. We therefore run both and report which one fails.
Baselines
The naive reference is Kazdan et al.'s accumulate-subsample: a uniform random subsample of size $N$ from the growing pool, run with their exact workflow, model, and generation count. The file gives no numeric cross-entropy values for this plateau (no reported result found on Lacuna for accumulate-subsample loss numbers), so we reproduce it ourselves. Beating it means lower held-out real cross-entropy at generation 10 across several seeds. Unbounded accumulation serves as the target ceiling. We report the fraction of the plateau-to-ceiling gap that each method closes.
Real-pinned uniform subsampling keeps every real item and fills the remaining slots uniformly with synthetic data. This is the simplest fix and the most dangerous competitor. Reservoir sampling and FIFO buffers are the standard streaming defaults. GCR is the strongest existing learned buffer policy. Its reported margin over reservoir sampling is 2–4% offline and up to 5% online accuracy on sequential CIFAR and ImageNet variants (GCR). We run it with an equal candidate pool and an equal number of gradient evaluations per step.
The strongest offline competitor reruns DiSF-style greedy diversity selection over the full accumulated pool every generation (Combatting Dimensional Collapse). Its reported number, 3.4% over DSIR on TinyLlama 1.1B pre-training, comes from a different regime, so we compare on our own runs. This baseline gets more memory than we do, because it stores the full pool. We match its selection-compute budget. We also run offline greedy on our own objective $F$, which separates the cost of streaming from the value of the objective.
Two ablations isolate the key ingredient. Setting the anchor term to zero gives a diversity-only streaming buffer with identical swap logic. Replacing threshold gating with "swap on any positive gain" tests whether the thresholds prevent thrashing. Every method uses the same generator, the same synthetic stream order, the same training steps per generation, and the same feature encoder wherever features are needed. Each method gets an equal hyperparameter sweep of eight configurations, tuned on a validation anchor set that is disjoint from both $R$ and the test data.
Why not just…?
Why not just pin all real data and subsample synthetic data uniformly? This is a real threat. It may recover much of the plateau, because the collapse results in Collapse or Thrive hinge on keeping real data. It does not help, however, once real data alone exceeds the budget. It also cannot distinguish synthetic samples that fill gaps between sparse real points from those that pollute. The paper's gap-filler result below 1,024 real samples and its pollution result above that point suggest this distinction matters. We sweep real-data size from 256 to well above 1,024. If real-pinned subsampling matches our method at every size, we will report that and drop the selection claim, keeping only the exchange-rate measurement.
Why not just rerun offline diversity selection each generation? At generation scale this is feasible, and with unlimited storage it could match us. Our hypothesis is that diversity without anchors selects off-distribution synthetic outliers. Liu's Proposition 1 implies that no source-only score, diversity included, can detect missing target support. The diversity-only streaming ablation and offline DiSF together test this. If DiSF matches anchor coverage, the anchor term adds nothing.
Why not just filter each synthetic sample with a quality score or verifier? Point-wise filters cannot see coverage. This is proved in Model Collapse Despite Ground-Truth Verification and observed as a "diversity dilemma" for similarity filters in DiSF. We include a likelihood-threshold filter on top of uniform subsampling as an additional baseline, and we measure per-mode coverage directly to show where it fails.
Why not tune a fixed real-to-synthetic mix? A fixed ratio cannot follow the shift from gap-filling to pollution. We give the fixed-mix baseline an oracle choice of its best ratio for each real-data size. If it matches us, our dynamic method offers convenience but no gain.
Proposed Experiments
The first testbed reproduces Kazdan et al.'s multivariate Gaussian and kernel density settings over 100 generations at fixed $N$. Here ground truth is known, so we can directly measure covariance shrinkage, density error, and coverage of every true mode, including deliberately rare modes. We also plant a mode that appears in the anchor set but that the generator under-produces, to check whether coverage-gated admission keeps the few synthetic samples that land there.
The second testbed follows their language-model protocol: recursive supervised fine-tuning of Gemma 2 2B for 10 generations. The metric is cross-entropy on held-out real data, reported as the fraction of the gap closed between accumulate-subsample and full accumulation. We also report the effective rank of representations, the measure DiSF uses for dimensional collapse.
The exchange-rate check runs within the language-model testbed. At several points we take the online estimate $\phi_{\text{syn}}/\phi_{\text{real}}$ and compare it with a measured ratio: how many synthetic samples must replace one real sample to hold held-out loss constant, found by retraining from that checkpoint. A Spearman correlation above 0.7 across checkpoints and real-data sizes would support the evaluation claim. We also predict that the estimated rate rises toward "synthetic is nearly worthless" as real data passes the pollution regime. If budget allows, we repeat the substitution measurement on MOT17 with Chang et al.'s setup. There, the published reference points are a 30:1 frame equivalency and the finding that 20% real data plus synthetic pre-training nearly matches 100% real.
Two results decide the project. The first is whether anchor-coverage eviction beats both real-pinned subsampling and diversity-only eviction at matched budget across the real-data sweep. The second is whether the threshold ratio predicts measured substitution. If the first fails and the second holds, the contribution narrows to a cheap online meter for the value of synthetic data. If both fail, a fixed buffer with the real data pinned is all that bounded self-consuming training needs, and that is worth knowing too.
References
- Collapse or Thrive: Perils and Promises of Synthetic Data in a Self-Generating World
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Online Submodular Maximization with Preemption
- Data Summarization at Scale: A Two-Stage Submodular Approach
- Streaming Algorithm for Monotone k-Submodular Maximization with Cardinality Constraints
- Combatting Dimensional Collapse in LLM Pre-Training Data via Submodular File Selection
- Model Collapse Despite Ground-Truth Verification
- A Training-free Synthetic Data Selection Method for Semantic Segmentation
- GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning