Multimodal language models, which answer questions about images by turning each image into hundreds or thousands of "visual tokens" placed before the text, spend most of their compute on those tokens. The standard remedy is pruning: discard most visual tokens early and let the language model work with the rest. Current pruners are one-way. Once a patch is dropped, no later step can bring it back, even when the question turns out to hinge on exactly that patch. This proposal borrows a fix from error-correcting codes. There, cheap iterative decoders sometimes lock onto a wrong answer, and good decoders detect this and undo or re-seed the specific unreliable parts. They do not restart from scratch. We apply the same logic to pruning. We prune aggressively, keep the dropped tokens' features in a side store (the vision encoder has already computed them), watch the model's own uncertainty while it answers, and, only when it is unsure, restore a small targeted set of dropped tokens and re-answer.
In two sentences: pruning should be treated as a reversible, first-pass decision, like a fast decoder's first guess. A cheap uncertainty trigger plus targeted restoration of the tokens behind the model's most-consulted surviving tokens should recover most pruning-induced errors at a fraction of the cost of pruning less or re-running the full model. The contribution is not "rollback" as a slogan. It is a specific claim about where the accuracy-versus-compute trade-off of pruned multimodal models can be moved, and it comes with the experiments that could show the claim is false.
Where the idea comes from and what is already done
Three coding papers supply the method. Iterative Neural Rollback Chase-Pyndiah Decoding shows that in iterative product-code decoding, early wrong updates derail the decoder at low signal-to-noise ratio. A small Transformer classifier that flags these updates and zeroes them out gains about 0.145 dB, and a cheaper decoder with this aid beats a more expensive decoder without it. Learned Decimation for Neural Belief Propagation Decoders escapes false fixed points by forcing the least reliable positions to a fixed value across a small list of parallel branches, getting within 1 dB of maximum-likelihood decoding. It also shows the cost: the list grows exponentially with how many positions are forced. These papers motivate two design choices. The intervention should be local and targeted rather than a restart, and the number of parallel candidates must stay tiny. We take the "fix only the unreliable part" principle and the bounded candidate list. We do not take the physics of spatial coupling or threshold saturation, which we do not claim transfers.
On the language-model side, Latent Phase-Shift Rollback rewinds the key-value cache (the stored per-token attention state) by one token when middle-layer signals and high token entropy indicate a reasoning error. This lifts LLaMA-3-8B on MATH-500 from 28.8% to 44.0%. The same paper reports that prompted self-correction lowered accuracy by 9 points. It works along the generation dimension on text-only math and never touches visual tokens or pruning. It also constrains our design: it finds that the layer that best detects an error is not the layer where correction works. So we will not assume that the signal and the intervention can share a location.
The pruners we build on are one-shot. ATP-LLaVA learns instance- and layer-wise pruning thresholds and keeps 98.1% of LLaVA-1.5 performance at 144 of 576 tokens, while noting that shallow layers are fragile. DART shows that attention-based "importance" often loses to random pruning. It instead keeps a few pivot tokens and drops tokens highly similar to them. CDPruner selects a query-conditioned diverse subset. None of the three can revisit a decision. The research file found no paper that restores dropped visual tokens during inference.
One tension in this literature matters here. Several of these pruners sometimes beat the unpruned model: DART on POPE, ATP-LLaVA on MMBench, and CDPruner reaching 100.1% on LLaVA-NeXT. So pruning partly acts as denoising. Restoration therefore has two ways to fail. The pool of errors caused by pruning may be small, and restoring tokens could reintroduce the noise that pruning removed. The program measures both before claiming anything.
Core Claim
Consider a pruned model run at an aggressive keep ratio. Pair it with a trigger based on the answer's own uncertainty and a restoration step that re-inserts only the dropped tokens behind the most-attended surviving tokens. We claim this pair reaches a better accuracy-per-compute curve than either keeping more tokens up front or falling back to the unpruned model when uncertain. The gain should be largest on fine-grained benchmarks (text reading, charts, small objects), where a single dropped patch decides the answer.
Research Program
The system has four parts, all training-free except an optional small trigger.
First, a first-pass pruner. We use DART because its pivot structure gives each surviving token an explicit set of dropped tokens it "stands in for." Dropped tokens' vision-encoder outputs stay in a side buffer, at a memory cost of one feature vector per token and no language-model compute.
Second, a trigger. For short-answer tasks we use the margin between the top two answer-token probabilities. Following the detection/correction dissociation reported for Latent Phase-Shift Rollback, we also test a middle-layer signal: the drop in similarity of the answer position's hidden state across layers. The threshold is set on a held-out split to hit a target trigger rate.
Third, targeted restoration. At the answer position we read attention over the surviving visual tokens. We take the top few surviving tokens and "un-merge" their DART clusters, restoring the dropped tokens assigned to them, capped at a budget $k$.
Fourth, re-answering. This is where the key engineering question lies. The cheap version appends restored tokens to the existing key-value cache with their original position indices and regenerates only the answer. This is cheap, but the prompt tokens' states were computed without the restored tokens. The expensive version re-runs prefill on the kept-plus-restored set. We also test a Chase-style variant, named after the decoder that tries a small list of test patterns and keeps the most reliable result. Here, two or three alternative restoration sets are tried in parallel, and the answer with the highest margin is kept, with the list size fixed to avoid the exponential growth seen in list decimation.
Compute is accounted honestly. The expected visual-token cost per query is $C = n_0 + r\,(k + c)$, where $n_0$ is the first-pass kept count, $r$ the fraction of queries that trigger, $k$ the restored count, and $c$ the extra cost of re-answering expressed in token-equivalents. Every comparison is made at matched $C$ and matched measured latency.
Baselines
The naive reference is the unpruned LLaVA-1.5-7B, together with the same pruner at its default keep ratio with no restoration. The strongest existing methods are DART, CDPruner and ATP-LLaVA, each run with authors' settings on the same models and benchmarks.
The file gives only aggregate numbers for these methods. CDPruner reports 94% of original accuracy at 95% FLOP reduction on LLaVA-1.5-7B. ATP-LLaVA reports 98.1% retained at 144 tokens. For DART, the file reports 88.9% pruning, a 2.99x prefill speedup, and 98.8% accuracy retained "with 192 tokens." These figures are not mutually consistent, since 192 of 576 is 67% pruning, so we will re-measure DART rather than quote it. For per-benchmark scores on TextVQA, ChartQA, OCRBench and GQA, there is no reported result found on Lacuna, and all such numbers will be produced in our own runs.
The two baselines that most directly threaten the claim are simple ones. The first is "prune less": DART or CDPruner at whatever keep ratio matches our average cost $C$. The second is the uncertainty cascade: the same trigger, but on firing it re-runs the full unpruned model.
Two ablations isolate the key ingredient. Random restoration draws the same $k$ dropped tokens uniformly and tests whether targeting matters. Oracle restoration uses the unpruned model's answer to choose which clusters to restore and bounds the headroom.
Beating the method means strictly higher accuracy than prune-less at equal measured latency, on at least the fine-grained benchmarks. It also means matching the cascade's accuracy at clearly lower cost. All methods share the same model checkpoints, the same evaluation harness, and the same held-out split for threshold tuning, with an equal number of threshold settings swept per method.
Why not just…?
Why not just keep more tokens? This is the real threat. If pruning errors are spread uniformly across queries, raising the keep ratio buys the same accuracy without any machinery. Our bet is that errors concentrate in a small set of queries that can be flagged. The matched-cost Pareto curve against prune-less settles the question directly. If prune-less matches us, the claim fails.
Why not just re-run the full model when uncertain? This cascade is conceptually close to our proposal and may match it on accuracy. Our only advantage is cost. When it fires, the cascade pays for a full prefill over roughly 576 or more tokens, while restoration adds $k$ tokens. If append-only restoration loses too much accuracy and only re-prefilled restoration works, the cost gap narrows, and we will report that honestly. The append-versus-re-prefill ablation exists to measure exactly this.
Why not just prompt the model to double-check the image? In pruned models the missing detail is simply not present, so no amount of re-reading can recover it. In addition, Latent Phase-Shift Rollback found that prompted self-correction cost an 8B model 9 points. We include a prompted re-check baseline at equal token cost for completeness.
Why not just use random dropping or a better one-shot selector? DART shows random pruning is surprisingly strong. That makes random restoration a necessary ablation. It also makes it plausible that a better one-shot selector closes much of the gap. For this reason we apply restoration on top of both DART and CDPruner and report the gain each time.
Proposed Experiments
The models are LLaVA-1.5-7B and LLaVA-NeXT. The benchmarks are GQA, VQAv2, MMBench, POPE, TextVQA, ChartQA and OCRBench, with MVBench as a stretch goal for long-sequence inputs. We report accuracy, average visual tokens processed including restorations, FLOPs, and wall-clock prefill-plus-decode latency with FlashAttention enabled.
The first decisive check is a headroom analysis done before building anything. On each benchmark we count the pruning-induced errors: queries the unpruned model gets right and the pruned model gets wrong. We weigh them against pruning-fixed errors, the reverse case. If pruning-induced errors are only a few percent of queries, as the denoising results in the file hint for POPE, then restoration's ceiling is small there. The work would then focus on text-reading and chart benchmarks.
The second check is trigger quality. We measure how well the margin or middle-layer signal separates pruning-induced errors from other errors, as area under the ROC curve. A trigger that fires mainly on questions the full model also fails wastes compute.
The third check is the matched-cost Pareto comparison against prune-less and the cascade. The fourth is the targeting and append ablations.
We expect a positive result to look like the following. On TextVQA and ChartQA, restoration of a few dozen tokens on a minority of queries recovers most of the gap between the pruned and unpruned model at an average cost near the pruned model's. On POPE the gain is flat or negative, which would confirm that restoration also undoes pruning's denoising. The negative result, that prune-less matches us at equal latency, would itself be informative: it would mean pruning errors are not concentrated enough in identifiable queries to justify selective recovery.
References
- Iterative Neural Rollback Chase-Pyndiah Decoding
- Learned Decimation for Neural Belief Propagation Decoders : Invited Paper
- Latent Phase-Shift Rollback: Inference-Time Error Correction via Residual Stream Monitoring and KV-Cache Steering
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
- Stop Looking for "Important Tokens" in Multimodal Language Models: Duplication Matters More
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs