TL;DR
Balanced Twin Weight Factorization (BTWF) is a new compression technique that significantly speeds up Multimodal Large Language Models (MLLMs) by exploiting "twin layers"—groups of layers like Query, Key, and Value projections that share the same input. By sharing a single basis computation across these twins and balancing their individual importance, BTWF achieves 2-6% better accuracy than existing state-of-the-art methods at the same computational cost, often without needing expensive fine-tuning.
What Problem Does This Paper Address?
Large-scale models like MLLMs are computationally expensive to run. Weight Factorization (WF) is a common solution: it breaks down a large weight matrix into two smaller ones (e.g., $W \approx AB$), reducing the number of floating-point operations (FLOPs).
However, standard WF faces three major hurdles in practice:
- Latency Overhead: Replacing one matrix multiplication with two often introduces overhead. If you don't reduce the number of launches of these operations, the model might actually run slower despite having fewer FLOPs.
- Large-Member Bias: In architectures like Grouped-Query Attention (GQA), different layers (twins) share the same input but have different sizes. Standard factorization methods tend to favor the larger layers, leaving smaller, critical layers poorly reconstructed.
- Local vs. Global Fidelity: Most methods try to minimize the error in reconstructing a specific weight matrix. But a "perfect" local reconstruction might not be what the overall model needs to maintain high performance on downstream tasks.
Methods
The authors introduce Balanced Twin Weight Factorization (BTWF) to solve these issues through three core components:
-
Shared Basis for Twin Layers: BTWF identifies "twin" layers—such as $Q, K, V$ in attention blocks or the gate and up projections in SwiGLU MLPs—that process the same input tensor. It decomposes them into a shared basis matrix $B$ and layer-specific mixing matrices $A_k$. This allows the model to compute the projection $z = Bx$ just once and reuse it for all twins, directly slashing latency.
Illustrative example of shared factorization and the improvements of BTWF -
Factor Component Balancing (FCB): To fix the bias toward larger layers, FCB normalizes each twin member's contribution before performing the decomposition. This ensures that smaller layers (like the Key/Value projections in GQA) are well-represented in the shared basis.
- Block Contribution Balancing (BCB): Instead of just looking at the weight matrix, BCB considers how errors propagate to the output of the entire transformer block. It weights the importance of different channels based on their sensitivity to the final block output, retaining the information that matters most for the model's "thinking."
- Overparameterized Groups (OG): For layers that are particularly sensitive to compression, the authors use a larger rank during training to boost capacity. Crucially, these factors are "baked" back into the original dense layer size for inference, providing a performance boost with zero extra cost at test time.
Datasets
The researchers evaluated BTWF across six different model architectures (including InternVL3 and Qwen2-VL) on nine diverse benchmarks:
- Image-Language: GQA, VQAv2, VizWiz, and TextVQA.
- Video-Language: CLEVRER, NExT-QA, Perception Test, Video-MME, and LongVideoBench.
They used a small set of 256 samples from the CLEVRER dataset for calibration, demonstrating that the method works well even with very little data.
Key Insights and Ideas
- Efficiency through Structure: The "aha moment" is realizing that MLLM architectures are full of redundant input projections. By mathematically forcing these layers to share a basis, you don't just save math; you save time by reducing the number of sequential operations the GPU has to manage.
- Fairness Matters: Standard SVD is "unfair" to small matrices. By simply balancing the norms (FCB), the authors show you can significantly improve the fidelity of the compressed model.
- Downstream Sensitivity: Figure 3 illustrates that while BCB might slightly increase the local reconstruction error of a weight matrix, it consistently decreases the error at the block output. This proves that not all weight errors are created equal; some are much more damaging to the model's final answer than others.
Key Results
- Superior Accuracy: BTWF consistently outperformed state-of-the-art methods like SVD-LLM and Basis Sharing by 2-6% in "criteria-only" settings (no fine-tuning).
- Effective Fine-tuning: When combined with short fine-tuning (3 epochs), it maintained a 1-3.5% lead over competitors.
- Real-world Speedups: At a 50% FLOP budget, BTWF achieved speedups of 1.08x to 1.35x compared to dense models, making it the fastest among the factorized methods tested.
- Drop-in Compatibility: The method works seamlessly as a replacement for other compression techniques, showing synergistic gains when combined with weight pruning, token pruning, and 4-bit quantization.
Limitations and Future Work
While BTWF provides significant speedups, the authors note that the latency reduction still lags behind the theoretical FLOP reduction (e.g., a 50% FLOP cut might only yield a 30% speedup). This is due to the inherent overhead of launching multiple smaller kernels on modern GPUs.
Future work could involve:
- Developing specialized GPU kernels to better support shared-basis operations.
- Exploring more sophisticated ways to utilize the extra capacity provided by Overparameterized Groups (OG) during training.
- Applying the "twin" logic to even more complex multi-modal architectures beyond standard transformers.
Summary updated