TL;DR
ReGATE accelerates Multimodal Large Language Model (MLLM) training by dynamically pruning text tokens that don't require visual grounding or are already easy for the model to learn. By filtering out these redundant tokens during both forward and backward passes, it cuts token usage by over 41% and speeds up training by up to 2$\times$ while matching or even exceeding baseline accuracy.
What Problem Does This Paper Address?
Training Multimodal Large Language Models (MLLMs)—especially for video-based tasks—is notoriously expensive. The self-attention mechanism scales quadratically with sequence length, and tokenizing multiple video frames creates massive inputs.
While many researchers have proposed token pruning or merging strategies to speed up inference, these methods don't help reduce training costs. During training, models typically still process every single token in every forward and backward pass. Current training-time efficiency methods often rely on crude heuristics like random token dropping, which can inadvertently delete crucial visual details or cross-modal dependencies. ReGATE addresses this gap by providing a principled, adaptive way to skip uninformative tokens during training without altering the model architecture.
Methods
ReGATE (Reference-Guided Adaptive Token Elision) introduces a parameter-free teacher-student framework to dynamically gate text tokens:
- The Text-Only Teacher: The authors take the student MLLM's language backbone, remove the visual encoder/projector, and freeze its weights. This text-only model calculates a reference loss for text tokens by replacing visual tokens with placeholders. If the teacher can easily predict a word from text alone, that word has a low reference loss and likely doesn't need visual grounding.
- The Student's Difficulty Tracking: As the student MLLM trains, it maintains a running Exponential Moving Average (EMA) of its historical cross-entropy loss for each token position. This tracks which tokens the model is still struggling to learn.
- Unified Scoring & Gating: ReGATE combines the teacher's reference loss and the student's EMA difficulty into a single score. Tokens with low scores are masked out, meaning the transformer decoder skips self-attention and MLP computations for them during both forward and backward passes. Crucially, all visual tokens are always retained to preserve multimodal context.
Datasets
The authors evaluated ReGATE across three distinct MLLM architectures: VideoLLaMA2-7B, VideoChat2-7B, and InternVL3.5-14B. Training datasets included the VideoChatGPT dataset (~300k instruction-response pairs) and a larger 2.6M instruction-pair corpus.
Evaluation spanned 5 image benchmarks (ScienceQA, MME, VizWiz, POPE, SEED) and 8 video benchmarks (including Video-MME, LongVideoBench, MLVU, EgoSchema, and MVBench) to test everything from short-term perception to hour-long temporal reasoning.
Key Insights and Ideas
- Textual Redundancy in Multimodal Tasks: A surprising amount of text in a multimodal response can be predicted purely from linguistic context (e.g., grammar tokens like "the" or "is"). ReGATE exploits this by focusing compute only where visual grounding is strictly necessary.
- The "Goldilocks" Teacher Requirement: The authors discovered a "capacity mismatch effect." The teacher model must be capacity-aligned with the student. If the teacher is too powerful (e.g., a 57B teacher for a 7B student), it predicts complex tokens using its own massive world knowledge and mistakenly tells the student to prune them, even though the student actually needs visual cues to learn them.
- Dual-Cycle Stability: To prevent training instability, ReGATE uses a deterministic schedule that alternates between full-token training steps (to let the model stabilize) and sparse-token steps.
Key Results
- 2$\times$ Faster Training: On the MVBench benchmark, ReGATE matched the peak accuracy of standard full-token training up to twice as fast, utilizing only 38% of the tokens on average.
- Better Final Performance: When allowed to train longer (but still using fewer total GPU hours than the baseline), ReGATE actually surpassed standard training baselines across multiple benchmarks while cutting total token usage by over 41%.
- Memory Savings: By processing fewer intermediate activations, ReGATE reduced average per-GPU memory consumption by 7 to 8 GB (roughly a 10-11% savings) across all tested backbones.
Limitations and Future Work
Currently, ReGATE relies on a fixed design for its sparsity schedule and token retention ratio (e.g., keeping a static 50% of sparse tokens). The authors note that dynamically adjusting this ratio based on training progress or input complexity could optimize the efficiency-performance trade-off further. Additionally, expanding the reference signal to incorporate stronger teachers capable of fine-grained spatial or temporal reasoning remains an open direction.
Summary updated