Token Budget Reinvestment for High Resolution Vision

19 concepts11 papers1 proposal

When designing deep learning systems for computer vision, practitioners face a fundamental trade-off between input resolution and model capacity. Processing high-resolution images captures fine-grained spatial details but incurs a steep computational penalty.

A growing body of research focuses on token budget reinvestment: a strategy that exploits the non-linear relationship between input resolution and computational cost. By strategically reducing or pruning visual tokens, models can free up computational resources and reinvest them into higher-capacity architectures or finer spatial details where they matter most.

The Quadratic and Quartic Bottleneck

To understand why token budget reinvestment is so effective, we must look at how modern vision models process spatial data. In a Vision Transformer (ViT), an input image is divided into a grid of patches, each treated as a visual token. If an image's height and width are doubled, the total number of tokens ($N$) quadruples.

Because the self-attention mechanism computes pairwise interactions between all tokens, its computational and memory complexity scales quadratically with the number of tokens ($O(N^2)$). Expressed in terms of the input image dimensions, the cost scales quartically ($O(H^4)$). As Morlier and Léonardon demonstrate, this creates a severe non-linear penalty for high-resolution inputs in ViTs compared to convolutional networks, where costs scale quadratically.

However, this mathematical bottleneck is a double-edged sword. If increasing resolution triggers a quartic cost explosion, then decreasing resolution or dropping redundant tokens yields disproportionately massive computational savings.

Reinvesting the Token Budget via Pruning

Instead of simply using smaller images to speed up inference at the cost of accuracy, token budget reinvestment treats saved computations as a currency to be spent elsewhere.

One direct way to reclaim this budget is through spatial pruning during training or inference. Because natural images contain significant spatial redundancy (e.g., large expanses of background sky or uniform textures), many tokens contribute little to the final prediction.

[High-Res Input Image] ──> [Tokenization] ──> [Intelligent Pruning/Dropout]
                                                      │
                                           (Freed Computational Budget)
                                                      │
                                                      ▼
                                      [Reinvest in Larger Model Capacity]
                                      [or Finer Local Spatial Details]

Matsoukas et al. introduce PatchDropout, demonstrating that randomly discarding a large portion of input patches during training drastically reduces memory and compute demands. The authors show that this reclaimed budget can be immediately reinvested to train larger model architectures or process higher-resolution images within fixed hardware constraints.

Rather than dropping tokens randomly, pruning can also be guided dynamically by context:

  • Inference-Driven Fusion: Liang et al. (EViT) identify and fuse inattentive tokens mid-network, keeping only the most critical visual features active.
  • Cross-Modal Guidance: In vision-language tasks, Jiang et al. (TRIPS) leverage the accompanying text query to prune image patches that are irrelevant to the textual context.

In both cases, the authors demonstrate that early-stage token reduction allows the model to accept much higher-resolution raw inputs initially, capturing fine-grained details without increasing the overall FLOP footprint. This trade-off is highly effective in dense prediction tasks like video action detection, where Chen et al. show that the accuracy gained from processing higher-resolution frames far outweighs the minor information loss from discarding redundant spatio-temporal tokens.

Asymmetric and Inverted Architectures

Another facet of this research direction challenges the assumption that all image resolutions should be processed by the same network capacity. Traditional multi-scale architectures, such as image pyramids, run the same heavy backbone across all resolution tiers, resulting in massive computational waste at higher scales.

Wang et al. propose Parameter-Inverted Image Pyramid Networks (PIIP) to resolve this inefficiency. They assign model capacity inversely to input resolution:

Input Scale Target Feature Extraction Assigned Model Capacity
Low-Resolution Deep Global Semantics High-Capacity (Heavy Model)
High-Resolution Fine Spatial Details Lightweight (Small Model)

By routing high-resolution images through lightweight branches, the authors mitigate the quadratic cost explosion while preserving deep semantic understanding via the low-resolution, high-capacity branch.

A similar principle applies to deep metric learning and image retrieval. Tolias and Šuma reveal a structural asymmetry in image matching: pairing a low-resolution query image with high-resolution database images allows practitioners to use a much larger neural network for a fixed computational budget. This configuration yields higher precision than processing full-resolution queries with a smaller, reduced-capacity model.

Strategic Scaling Across Training and Inference

Budget reallocation can also be targeted to specific phases of a model's lifecycle to optimize throughput.

Phase-Based Pre-training

In large-scale vision-language pre-training, Li et al. (RECLIP) exploit token scaling by training CLIP on tiny $64\times64$ images for the bulk of the pre-training process. Because the sequence length is minimized when training is most computationally expensive, they slash overall FLOPs by up to 9x, requiring only a brief high-resolution fine-tuning stage to match standard baseline performance.

Constant Token Budgets in Generation

For pixel-space generative modeling, Lei et al. maintain a constant token budget by scaling patch sizes proportionally with the target image resolution. This allows high-resolution generative models to operate with low computational overhead, bypassing the need for complex latent-space autoencoders.

Dynamic Inference Selection

At inference time, Kimhi et al. (CARES) introduce a lightweight selector that predicts the minimum resolution required to answer a specific user query. By dynamically downsizing simple inputs, they reduce Vision-Language Model (VLM) latency by up to 80% with negligible accuracy loss.

Boundary Conditions of Resolution Scaling

While reducing resolution to fund model capacity is a powerful paradigm, it is not a universal remedy. To map out the boundaries of this trade-off, Palacio et al. conducted an extensive benchmark of over 45 Vision Transformer variants.

The authors discovered that under certain resource constraints, scaling model capacity (increasing depth or width) is actually more computationally efficient for achieving accuracy gains than increasing input image resolution. This finding solidifies the core philosophy of budget reinvestment: rather than naively feeding large images into standard networks, practitioners should prioritize compact token representations and spend their precious computational budgets on more expressive, higher-capacity model architectures.

Key Foundational Work

PatchDropout: Economizing Vision Transformers Using Patch Dropout (2023) — contributed 3 concept elements to this direction.
EViT: Expediting Vision Transformers via Token Reorganizations (2022) — contributed 2 concept elements to this direction.
Efficient Video Action Detection with Token Dropout and Context Refinement (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_computational_budget_reinvestment_932e9b0c,
  title = {computational budget reinvestment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-budget-reinvestment/rdr_932e9b0cd55c44febe11514c1efb83eb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_computational_budget_reinvestment_932e9b0c_v1,
  title = {computational budget reinvestment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-budget-reinvestment/rdr_932e9b0cd55c44febe11514c1efb83eb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake