TL;DR
Standard token compression in omnimodal large language models (OmniLLMs) uses a global top-$K$ ranking that accidentally starves video tokens because the model's shared attention space naturally favors audio. The authors introduce MACER, a training-free compressor that fixes this by explicitly allocating separate token budgets for audio and video before ranking them locally. This simple shift slashes token costs by 75% while preserving up to 98.7% of full-token performance.
What Problem Does This Paper Address?
Omnimodal Large Language Models (OmniLLMs) can process interleaved audio and video streams through a single unified decoder. While powerful, this design means that inference costs scale drastically with the combined length of the multimodal sequence, making token compression essential for real-world deployment.
Currently, most token compressors treat this as a single global ranking problem: score every multimodal token based on attention, and keep the top-$K$. The authors identify a fundamental structural flaw here, termed "one score, two decisions."
In a shared decoder, raw attention magnitude simultaneously decides two things:
- Capacity Allocation: How many tokens each modality gets to keep.
- Token Selection: Which specific tokens within that modality are the most important.
Because shared attention exhibits a persistent, content-responsive bias toward audio tokens, a global top-$K$ cutoff spends the token budget on audio before visual evidence even has a fair chance to compete.
Methods
To break this coupling, the authors propose MACER (Modality-Aware Capacity-Explicit Retention), a training-free, inference-time compressor that operates on an "allocation before ranking" principle.
MACER introduces three core components:
- Capacity-Coordinated Split (CCS): Instead of letting raw attention dictate the budget, MACER explicitly assigns fixed token budgets ($K_a$ and $K_v$) to audio and video based on a target retention ratio.
- Allocation-Normalized Modality Scoring (ANMS): To rank tokens fairly, MACER normalizes attention scores within each modality separately. This strips away the cross-modal mass imbalance, ensuring tokens only compete against their peers from the same modality.
- Modality-Specific Shallow Readout: The authors discovered that audio and video features mature at different depths in the network. Audio attention stabilizes very early, while video requires a bit more depth to reveal true saliency. MACER performs staged pruning: it reads and prunes audio tokens at an early layer ($\ell_a$), lets the decoder proceed, and then reads and prunes video tokens at a slightly later layer ($\ell_v$).
Additionally, MACER applies a lightweight temporal coverage bonus during the video selection step to prevent the remaining video tokens from clustering too tightly around a single timestamp.
Datasets
The authors evaluated MACER across four distinct benchmarks representing different modality demands:
- AVUTBench: Focused on audio-grounded tasks where audio carries the primary evidence.
- DailyOmni: Focused on audio-video joint tasks requiring event alignment and temporal ordering.
- WorldSense: A visually driven, visual-dominant benchmark with audio-augmented cases.
- Video-MME: A video-centric anchor benchmark evaluated in a no-subtitle setting.
Key Insights and Ideas
- Shared Attention is Not Neutral: Even when a query explicitly demands video evidence, the model's query representation sits geometrically closer to the audio centroid than the video centroid. Raw attention is a biased proxy for cross-modal budget allocation.
- Budgeting vs. Selection: Improving the selection algorithm within a modality (e.g., using advanced merging techniques like ToMe vs. random dropping) has negligible impact on the cross-modal balance. The real lever for performance is explicitly controlling the budget split between modalities.
- Asymmetric Readout Depths: Modality features don't become "readable" at the same time. Audio can be safely pruned as early as Layer 3 or 4, whereas video token selection benefits from waiting until Layer 5 or 7.
Key Results
MACER achieves state-of-the-art Pareto efficiency across multiple model sizes and architectures without requiring any retraining or architectural modifications:
- Qwen2.5-Omni-7B: At an aggressive 25% token retention rate, MACER preserves 98.7% of the full-token model's performance. It beats the strong OmniZip baseline by using fewer FLOPs at 25% retention than OmniZip does at 45% retention, while delivering higher accuracy.
- Qwen2.5-Omni-3B: At 25% retention, MACER retains 97.3% of full-token accuracy.
- Cross-Backbone Generalization: When tested on a completely different architecture, OmniVinci-9B, MACER outperformed traditional shared top-$K$ ranking by up to 12.9 points, proving that the allocation-before-ranking principle generalizes across model families.
Limitations and Future Work
MACER is explicitly designed for audio-video OmniLLMs that route both modalities through a single, shared-decoder attention stream. The authors note that extending this allocation-selection decomposition to models with fundamentally different multimodal fusion designs (like cross-attention or separate encoder-decoder bridges) remains an open challenge for future work. Additionally, exploring how this principle scales to broader modality combinations (e.g., text, image, audio, video, and thermal data all interleaved) is a natural next step.
Summary updated