TL;DR
ARMADA is a novel framework that allows language-only models to "learn from images" by distilling knowledge from large vision-language teachers (like Stable Diffusion). It achieves significant performance gains across NLU and reasoning tasks—even using black-box teachers—without the massive computational cost of multimodal pre-training.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful, but their "worldview" is often limited to the text they were trained on. While humans learn about the world through multiple senses, most LLMs are strictly unimodal.
Existing Knowledge Distillation (KD) techniques—which compress large "teacher" models into smaller "student" models—usually require both models to share the same modality (e.g., text-to-text). Cross-modal KD does exist, but it typically requires "white-box" access to the teacher's internal weights or expensive, task-specific pre-training of the teacher. The authors of this paper identify a major gap: how can we efficiently transfer the abstract, grounded knowledge found in multimodal models (like those that understand the relationship between "a falling glass" and "shattering") into text-only models without breaking the bank or needing internal access to proprietary models?
Methods
The authors introduce ARMADA (Alignment-induced cross-modal knowledge distillation). The framework is designed to be "architecture-agnostic," meaning it works regardless of the specific structures of the teacher or student.
The core components include:
- TS Aligner: A lightweight module (adding only ~0.8% learnable parameters) that maps the teacher’s multimodal representations into a space the student can understand.
- Manifold Alignment: Instead of just matching final outputs, ARMADA projects the hidden states of both models onto a shared "manifold" (a common mathematical subspace). They use three types of distance measures ($L_{cosine}$, $L_{euclid}$, and $L_{elementwise}$) to minimize the gap between these representations.
- Auxiliary Output Alignment: To further regularize the student, they add auxiliary task heads to the manifold projections. This ensures the student isn't just mimicking the teacher's "math" but is actually learning features useful for downstream tasks.
- Black-box Compatibility: ARMADA only requires the teacher's output embeddings (e.g., the latent representations from Stable Diffusion), making it compatible with API-based or proprietary models where weights are inaccessible.
Datasets
The researchers evaluated ARMADA across a massive battery of benchmarks:
- Natural Language Understanding (NLU): 12 tasks from the GLUE (e.g., CoLA, MNLI) and SuperGLUE (e.g., BoolQ, COPA) benchmarks.
- Reasoning: 5 commonsense reasoning tasks (HellaSwag, WinoGrande, ARC-easy/challenge, OpenBookQA) and 3 mathematical reasoning tasks (MultiArith, SingleEq, AQuA).
- Instruction-Tuning: Evaluated on Dolly, SelfInst, Vicuna, SNI, and UNI datasets.
- Multimodal Tasks: MM-IMDb (genre classification) and Hateful Memes to test if the text-only student could internalize multimodal priors.
Key Insights and Ideas
- Visual Grounding for Text-Only Models: The most striking insight is that a model that never "sees" an image can still perform better on text tasks by aligning its internal representations with a model that does see images. This suggests that vision-language models capture structural semantic information that text-only training misses.
- Syntactic and Social Gains: Error analysis showed that ARMADA-distilled models improved most on syntactic tasks (handling complex sentence structures) and socially-grounded reasoning (understanding human actions and goals).
- The "Teacher Doesn't Have to be Perfect" Effect: Interestingly, the TS Aligner itself often performs poorly on the tasks, yet it still successfully "teaches" the student model to perform better. The process of alignment acts as a powerful regularizer.
- Theoretical Equivalence: The authors provide a mathematical proof (using the concept of homeomorphism) to justify why aligning these abstract manifold spaces ensures that the models' outputs will also align.
Key Results
- Performance Boosts: ARMADA achieved up to a 3.4% improvement on NLU tasks for BERT-6L and a 2.6% boost in generative reasoning for LLaMA-7B.
- Efficiency: It outperformed competitive unimodal KD methods (like MetaDistil) while using 100x fewer trainable parameters for the distillation process.
- Scalability: The method showed consistent gains across a wide range of model sizes, from 66M parameter BERT models to LLaMA-3.1-8B.
- Multimodal Transfer: On the Hateful Memes task, a text-only BERT model distilled via ARMADA saw an absolute accuracy gain of ~5%, effectively "importing" visual context to better understand memes.
Limitations and Future Work
- Input Sensitivity: The framework is sensitive to the quality of the teacher's inputs. Adding Gaussian noise to the teacher's representations significantly degraded the student's performance, though it rarely fell below the undistilled baseline.
- Bias Transfer: The authors acknowledge a risk that is common in cross-modal work: unintended biases present in vision models (e.g., gender or racial stereotypes in image generation) could be distilled into the language model.
- Future Directions: The authors suggest exploring how this cross-modal distillation could be applied to even broader domains, such as robotics or sensory data, to create more "world-aware" language models.
Summary updated