Allocation Before Ranking: Decoupled Token Compression for OmniLLMs

Zhenghui Guo, Yilin Yang, Yuanbin Man, Miao Yin, Weidong Shi, Rabimba Karanjai, Omprakash Gnawali, Chengming ZhangarXiv 2026 · arXiv

TL;DR

Standard token compression in omnimodal large language models (OmniLLMs) uses a global top-$K$ ranking that accidentally starves video tokens because the model's shared attention space naturally favors audio. The authors introduce MACER, a training-free compressor that fixes this by explicitly allocating separate token budgets for audio and video before ranking them locally. This simple shift slashes token costs by 75% while preserving up to 98.7% of full-token performance.

What Problem Does This Paper Address?

Omnimodal Large Language Models (OmniLLMs) can process interleaved audio and video streams through a single unified decoder. While powerful, this design means that inference costs scale drastically with the combined length of the multimodal sequence, making token compression essential for real-world deployment.

Currently, most token compressors treat this as a single global ranking problem: score every multimodal token based on attention, and keep the top-$K$. The authors identify a fundamental structural flaw here, termed "one score, two decisions."

In a shared decoder, raw attention magnitude simultaneously decides two things:

  1. Capacity Allocation: How many tokens each modality gets to keep.
  2. Token Selection: Which specific tokens within that modality are the most important.

Because shared attention exhibits a persistent, content-responsive bias toward audio tokens, a global top-$K$ cutoff spends the token budget on audio before visual evidence even has a fair chance to compete.

Figure 1: Shared attention favors audio across depth. ca and cv denote the mean audio and video keys. (a) The query q is closer to ca than to cv, so shared attention favors audio. (b) Under both video-essential (VE) and audio-essential (AE) queries, log(massa/massv) stays positive across layers. (c) At L3 under VE queries, pooled query positions satisfy cos(q, ca) > cos(q, cv). This geometry illustrates why raw shared attention is not a neutral token-saliency space for omnimodal compression.

Methods

To break this coupling, the authors propose MACER (Modality-Aware Capacity-Explicit Retention), a training-free, inference-time compressor that operates on an "allocation before ranking" principle.

The MACER pipeline: staged modality-specific pruning within a single decoder prefill pass.

MACER introduces three core components:

  1. Capacity-Coordinated Split (CCS): Instead of letting raw attention dictate the budget, MACER explicitly assigns fixed token budgets ($K_a$ and $K_v$) to audio and video based on a target retention ratio.
  2. Allocation-Normalized Modality Scoring (ANMS): To rank tokens fairly, MACER normalizes attention scores within each modality separately. This strips away the cross-modal mass imbalance, ensuring tokens only compete against their peers from the same modality.
  3. Modality-Specific Shallow Readout: The authors discovered that audio and video features mature at different depths in the network. Audio attention stabilizes very early, while video requires a bit more depth to reveal true saliency. MACER performs staged pruning: it reads and prunes audio tokens at an early layer ($\ell_a$), lets the decoder proceed, and then reads and prunes video tokens at a slightly later layer ($\ell_v$).

Additionally, MACER applies a lightweight temporal coverage bonus during the video selection step to prevent the remaining video tokens from clustering too tightly around a single timestamp.

Datasets

The authors evaluated MACER across four distinct benchmarks representing different modality demands:

  • AVUTBench: Focused on audio-grounded tasks where audio carries the primary evidence.
  • DailyOmni: Focused on audio-video joint tasks requiring event alignment and temporal ordering.
  • WorldSense: A visually driven, visual-dominant benchmark with audio-augmented cases.
  • Video-MME: A video-centric anchor benchmark evaluated in a no-subtitle setting.

Key Insights and Ideas

  • Shared Attention is Not Neutral: Even when a query explicitly demands video evidence, the model's query representation sits geometrically closer to the audio centroid than the video centroid. Raw attention is a biased proxy for cross-modal budget allocation.
  • Budgeting vs. Selection: Improving the selection algorithm within a modality (e.g., using advanced merging techniques like ToMe vs. random dropping) has negligible impact on the cross-modal balance. The real lever for performance is explicitly controlling the budget split between modalities.
  • Asymmetric Readout Depths: Modality features don't become "readable" at the same time. Audio can be safely pruned as early as Layer 3 or 4, whereas video token selection benefits from waiting until Layer 5 or 7.

Key Results

MACER achieves state-of-the-art Pareto efficiency across multiple model sizes and architectures without requiring any retraining or architectural modifications:

  • Qwen2.5-Omni-7B: At an aggressive 25% token retention rate, MACER preserves 98.7% of the full-token model's performance. It beats the strong OmniZip baseline by using fewer FLOPs at 25% retention than OmniZip does at 45% retention, while delivering higher accuracy.
  • Qwen2.5-Omni-3B: At 25% retention, MACER retains 97.3% of full-token accuracy.
  • Cross-Backbone Generalization: When tested on a completely different architecture, OmniVinci-9B, MACER outperformed traditional shared top-$K$ ranking by up to 12.9 points, proving that the allocation-before-ranking principle generalizes across model families.

Limitations and Future Work

MACER is explicitly designed for audio-video OmniLLMs that route both modalities through a single, shared-decoder attention stream. The authors note that extending this allocation-selection decomposition to models with fundamentally different multimodal fusion designs (like cross-attention or separate encoder-decoder bridges) remains an open challenge for future work. Additionally, exploring how this principle scales to broader modality combinations (e.g., text, image, audio, video, and thermal data all interleaved) is a natural next step.

Summary updated

Abstract

Token compression in OmniLLMs is typically posed as a single saliency-ranking problem: score each multimodal token, keep the top-K. We argue this abstraction is mis-specified. The same attention score simultaneously decides two things: how much retained capacity each modality receives, and which tokens within a modality are kept. A shared top-K rule therefore inherits this audio-favoring allocation prior, spending retained capacity on audio before video tokens have a chance to compete. We propose Macer, a training-free compressor that first assigns explicit audio and video budgets, then performs allocation-normalized ranking within each modality at modality-specific shallow layers. Macer significantly reduces token cost while preserving accuracy across audio-grounded, audio--video joint, visual-dominant, and video-centric benchmarks. At 25 % retention, Macer preserves 98.7 % of full-token performance on Qwen2.5-Omni-7B and 97.3 % on Qwen2.5-Omni-3B. On Qwen2.5-Omni-7B, this 25 % setting reaches OmniZip-level performance at 45 % retention while using lower FLOPs. On OmniVinci-9B, the same allocation-before-ranking principle improves over shared top-K ranking by up to 12.9 points.
BibTeX
@misc{guo2026allocation,
  author = {Zhenghui Guo and Yilin Yang and Yuanbin Man and Miao Yin and Weidong Shi and Rabimba Karanjai and Omprakash Gnawali and Chengming Zhang},
  title = {{Allocation Before Ranking: Decoupled Token Compression for OmniLLMs}},
  year = {2026},
  eprint = {2608.01665},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.01665},
}
Figure