From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers

Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar, Tanmoy Chakraborty

TL;DR

ARMADA is a novel framework that allows language-only models to "learn from images" by distilling knowledge from large vision-language teachers (like Stable Diffusion). It achieves significant performance gains across NLU and reasoning tasks—even using black-box teachers—without the massive computational cost of multimodal pre-training.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful, but their "worldview" is often limited to the text they were trained on. While humans learn about the world through multiple senses, most LLMs are strictly unimodal.

Existing Knowledge Distillation (KD) techniques—which compress large "teacher" models into smaller "student" models—usually require both models to share the same modality (e.g., text-to-text). Cross-modal KD does exist, but it typically requires "white-box" access to the teacher's internal weights or expensive, task-specific pre-training of the teacher. The authors of this paper identify a major gap: how can we efficiently transfer the abstract, grounded knowledge found in multimodal models (like those that understand the relationship between "a falling glass" and "shattering") into text-only models without breaking the bank or needing internal access to proprietary models?

Methods

The authors introduce ARMADA (Alignment-induced cross-modal knowledge distillation). The framework is designed to be "architecture-agnostic," meaning it works regardless of the specific structures of the teacher or student.

The core components include:

  • TS Aligner: A lightweight module (adding only ~0.8% learnable parameters) that maps the teacher’s multimodal representations into a space the student can understand.
  • Manifold Alignment: Instead of just matching final outputs, ARMADA projects the hidden states of both models onto a shared "manifold" (a common mathematical subspace). They use three types of distance measures ($L_{cosine}$, $L_{euclid}$, and $L_{elementwise}$) to minimize the gap between these representations.
  • Auxiliary Output Alignment: To further regularize the student, they add auxiliary task heads to the manifold projections. This ensures the student isn't just mimicking the teacher's "math" but is actually learning features useful for downstream tasks.
  • Black-box Compatibility: ARMADA only requires the teacher's output embeddings (e.g., the latent representations from Stable Diffusion), making it compatible with API-based or proprietary models where weights are inaccessible.

Datasets

The researchers evaluated ARMADA across a massive battery of benchmarks:

  • Natural Language Understanding (NLU): 12 tasks from the GLUE (e.g., CoLA, MNLI) and SuperGLUE (e.g., BoolQ, COPA) benchmarks.
  • Reasoning: 5 commonsense reasoning tasks (HellaSwag, WinoGrande, ARC-easy/challenge, OpenBookQA) and 3 mathematical reasoning tasks (MultiArith, SingleEq, AQuA).
  • Instruction-Tuning: Evaluated on Dolly, SelfInst, Vicuna, SNI, and UNI datasets.
  • Multimodal Tasks: MM-IMDb (genre classification) and Hateful Memes to test if the text-only student could internalize multimodal priors.

Key Insights and Ideas

  • Visual Grounding for Text-Only Models: The most striking insight is that a model that never "sees" an image can still perform better on text tasks by aligning its internal representations with a model that does see images. This suggests that vision-language models capture structural semantic information that text-only training misses.
  • Syntactic and Social Gains: Error analysis showed that ARMADA-distilled models improved most on syntactic tasks (handling complex sentence structures) and socially-grounded reasoning (understanding human actions and goals).
  • The "Teacher Doesn't Have to be Perfect" Effect: Interestingly, the TS Aligner itself often performs poorly on the tasks, yet it still successfully "teaches" the student model to perform better. The process of alignment acts as a powerful regularizer.
  • Theoretical Equivalence: The authors provide a mathematical proof (using the concept of homeomorphism) to justify why aligning these abstract manifold spaces ensures that the models' outputs will also align.

Key Results

  • Performance Boosts: ARMADA achieved up to a 3.4% improvement on NLU tasks for BERT-6L and a 2.6% boost in generative reasoning for LLaMA-7B.
  • Efficiency: It outperformed competitive unimodal KD methods (like MetaDistil) while using 100x fewer trainable parameters for the distillation process.
  • Scalability: The method showed consistent gains across a wide range of model sizes, from 66M parameter BERT models to LLaMA-3.1-8B.
  • Multimodal Transfer: On the Hateful Memes task, a text-only BERT model distilled via ARMADA saw an absolute accuracy gain of ~5%, effectively "importing" visual context to better understand memes.

Limitations and Future Work

  • Input Sensitivity: The framework is sensitive to the quality of the teacher's inputs. Adding Gaussian noise to the teacher's representations significantly degraded the student's performance, though it rarely fell below the undistilled baseline.
  • Bias Transfer: The authors acknowledge a risk that is common in cross-modal work: unintended biases present in vision models (e.g., gender or racial stereotypes in image generation) could be distilled into the language model.
  • Future Directions: The authors suggest exploring how this cross-modal distillation could be applied to even broader domains, such as robotics or sensory data, to create more "world-aware" language models.

Summary updated

Abstract

Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly dropping performance. Traditional KD techniques assume homogeneity in modalities between the teacher (source) and the student (target) models. On the other hand, existing multimodal knowledge distillation methods require modality-specific pre-training of the teacher model, which is computationally infeasible in most cases. In this paper, we introduce ARMADA, an efficient cross-modal knowledge distillation framework designed to transfer knowledge from large vision-language models, including black-box models, to language-only models. Unlike existing KD techniques that rely on the internal structures of multimodal teachers or require computationally expensive pre-training, ARMADA leverages novel alignment techniques to distil knowledge without altering the teacher model, ensuring efficiency and scalability. We empirically validate ARMADA on twelve natural language understanding, eight complex generative reasoning and five instruction-tuning tasks, demonstrating consistent performance improvements in large models such as DeBERTa-v2-1.4B, OPT-1.3B, LLaMA-{3B, 7B, 8B}. ARMADA achieves up to 3.4% improvement on language understanding tasks and 2.6% boost in generative reasoning, all without requiring expensive multimodal pre-training or fine-tuning of the teacher model. Our findings challenge conventional knowledge distillation paradigms by demonstrating that even vision-language models, despite lacking direct textual understanding, can significantly enhance language models when distilled appropriately.
BibTeX
@misc{sengupta2026from,
  author = {Ayan Sengupta and Shantanu Dixit and Md Shad Akhtar and Tanmoy Chakraborty},
  title = {{From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers}},
  year = {2026},
  eprint = {2603.10877},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.10877},
}
Figure