ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs

Chaoyu Li, Yogesh Kulkarni, Pooyan Fazli

TL;DR

ReGATE accelerates Multimodal Large Language Model (MLLM) training by dynamically pruning text tokens that don't require visual grounding or are already easy for the model to learn. By filtering out these redundant tokens during both forward and backward passes, it cuts token usage by over 41% and speeds up training by up to 2$\times$ while matching or even exceeding baseline accuracy.

What Problem Does This Paper Address?

Training Multimodal Large Language Models (MLLMs)—especially for video-based tasks—is notoriously expensive. The self-attention mechanism scales quadratically with sequence length, and tokenizing multiple video frames creates massive inputs.

While many researchers have proposed token pruning or merging strategies to speed up inference, these methods don't help reduce training costs. During training, models typically still process every single token in every forward and backward pass. Current training-time efficiency methods often rely on crude heuristics like random token dropping, which can inadvertently delete crucial visual details or cross-modal dependencies. ReGATE addresses this gap by providing a principled, adaptive way to skip uninformative tokens during training without altering the model architecture.

Methods

ReGATE (Reference-Guided Adaptive Token Elision) introduces a parameter-free teacher-student framework to dynamically gate text tokens:

  1. The Text-Only Teacher: The authors take the student MLLM's language backbone, remove the visual encoder/projector, and freeze its weights. This text-only model calculates a reference loss for text tokens by replacing visual tokens with placeholders. If the teacher can easily predict a word from text alone, that word has a low reference loss and likely doesn't need visual grounding.
  2. The Student's Difficulty Tracking: As the student MLLM trains, it maintains a running Exponential Moving Average (EMA) of its historical cross-entropy loss for each token position. This tracks which tokens the model is still struggling to learn.
  3. Unified Scoring & Gating: ReGATE combines the teacher's reference loss and the student's EMA difficulty into a single score. Tokens with low scores are masked out, meaning the transformer decoder skips self-attention and MLP computations for them during both forward and backward passes. Crucially, all visual tokens are always retained to preserve multimodal context.
Figure 2: Overview of REGATE. The framework operates in two interconnected stages. (1) Reference Loss Generation (Left): A frozen, text-only teacher LLM processes the input text (with padding tokens) and computes a per-token reference loss (ref_loss), which measures how difficult each token is to predict from text alone. Higher loss values suggest the token likely requires visual grounding (e.g., “white”, “red stripe”). (2) Student Training (Right): The ref_loss is combined with the student model’s historical learning difficulty to produce a unified importance score. This score is used to create a binary mask that selects the most informative tokens. During training, the student LLM receives the full multimodal input but only performs computation (e.g., self-attention and feed-forward operations) on the selected tokens, while skipping the rest.

Datasets

The authors evaluated ReGATE across three distinct MLLM architectures: VideoLLaMA2-7B, VideoChat2-7B, and InternVL3.5-14B. Training datasets included the VideoChatGPT dataset (~300k instruction-response pairs) and a larger 2.6M instruction-pair corpus.

Evaluation spanned 5 image benchmarks (ScienceQA, MME, VizWiz, POPE, SEED) and 8 video benchmarks (including Video-MME, LongVideoBench, MLVU, EgoSchema, and MVBench) to test everything from short-term perception to hour-long temporal reasoning.

Key Insights and Ideas

  • Textual Redundancy in Multimodal Tasks: A surprising amount of text in a multimodal response can be predicted purely from linguistic context (e.g., grammar tokens like "the" or "is"). ReGATE exploits this by focusing compute only where visual grounding is strictly necessary.
  • The "Goldilocks" Teacher Requirement: The authors discovered a "capacity mismatch effect." The teacher model must be capacity-aligned with the student. If the teacher is too powerful (e.g., a 57B teacher for a 7B student), it predicts complex tokens using its own massive world knowledge and mistakenly tells the student to prune them, even though the student actually needs visual cues to learn them.
  • Dual-Cycle Stability: To prevent training instability, ReGATE uses a deterministic schedule that alternates between full-token training steps (to let the model stabilize) and sparse-token steps.

Key Results

  • 2$\times$ Faster Training: On the MVBench benchmark, ReGATE matched the peak accuracy of standard full-token training up to twice as fast, utilizing only 38% of the tokens on average.
  • Better Final Performance: When allowed to train longer (but still using fewer total GPU hours than the baseline), ReGATE actually surpassed standard training baselines across multiple benchmarks while cutting total token usage by over 41%.
  • Memory Savings: By processing fewer intermediate activations, ReGATE reduced average per-GPU memory consumption by 7 to 8 GB (roughly a 10-11% savings) across all tested backbones.
Table 1: Zero-shot evaluation results on image understanding benchmarks. Previous best results are highlighted in bold, while REGATE’s best results are underlined. I: SEED benchmark results are reported only for the image subset. For baseline models, scores are taken from their official publications where available.

Limitations and Future Work

Currently, ReGATE relies on a fixed design for its sparsity schedule and token retention ratio (e.g., keeping a static 50% of sparse tokens). The authors note that dynamically adjusting this ratio based on training progress or input complexity could optimize the efficiency-performance trade-off further. Additionally, expanding the reference signal to incorporate stronger teachers capable of fine-grained spatial or temporal reasoning remains an open direction.

Summary updated

Abstract

The computational cost of training multimodal large language models (MLLMs) grows rapidly with the number of processed tokens. Existing efficiency methods mainly target inference via token reduction or merging, offering limited benefits during training. We introduce ReGATE (Reference-Guided Adaptive Token Elision), an adaptive token pruning method for accelerating MLLM training. ReGATE adopts a teacher-student framework, in which a frozen teacher LLM provides per-token guidance losses that are fused with an exponential moving average of the student's difficulty estimates. This adaptive scoring mechanism dynamically selects informative tokens while skipping redundant ones in the forward pass, substantially reducing computation without altering the model architecture. Across three representative MLLMs, ReGATE matches the peak accuracy of standard training on MVBench up to 2$\times$ faster, using only 38% of the tokens. With extended training, it even surpasses the baseline across multiple multimodal benchmarks, cutting total token usage by over 41%.
BibTeX
@inproceedings{li2026regate,
  author = {Chaoyu Li and Yogesh Kulkarni and Pooyan Fazli},
  title = {{ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2154},
}
Figure