When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs

Abhirama Subramanyam Penamakuri, Navlika Singh, Piyush Arora, Anand MishraEMNLP 2025

TL;DR

This paper introduces the Model Parity Aligner (MPA), a novel framework that enables smaller vision-language models (VLMs) to achieve performance parity with much larger, computationally expensive models. MPA achieves this by leveraging unlabeled data and a unique "parity identification" strategy to pinpoint and address specific knowledge gaps in the smaller models, all without requiring any human-labeled data.

What Problem Does This Paper Address?

Large Vision-Language Models (L-VLMs) have achieved impressive results in tasks like Visual Question Answering (VQA). However, their sheer size makes them impractical for many real-world applications due to high computational costs and latency. Smaller VLMs (S-VLMs) offer efficiency but suffer from a significant performance gap. Traditional methods like knowledge distillation often require labeled data and struggle with the complex interplay of vision and language. This paper tackles the challenge of efficiently transferring knowledge from L-VLMs to S-VLMs in a label-free manner, aiming to bridge the performance gap without the computational burden.

Methods

The core of the proposed solution is the Model Parity Aligner (MPA), a three-module framework:

  1. Pseudo Annotator (PA): This module uses an L-VLM to generate question-answer pairs for unlabeled images. This is akin to "self-supervision" where the large model acts as a teacher to create training data.
  2. Parity Identifier (PI): This is the novel contribution. Instead of using all generated pseudo-annotations, PI identifies specific instances where the S-VLM fails to answer correctly, but the L-VLM answers correctly. This "parity" highlights the exact knowledge gaps that need to be addressed. It effectively filters out noisy or irrelevant annotations, focusing training on critical deficiencies.
  3. Parity Leveler (PL): The S-VLM is then fine-tuned only on the carefully selected "parity" samples identified by PI. This targeted fine-tuning is more efficient and effective than training on the entire dataset.

Crucially, MPA works with only unlabeled images and requires only input-output access to the L-VLM, making it compatible with closed-source models.

Datasets

The authors evaluated MPA on four diverse VQA benchmarks:

  • TextVQA: Requires understanding text in images.
  • ST-VQA: Focuses on scene text understanding.
  • ChartQA: Involves interpreting charts and graphs.
  • OKVQA: Demands external world knowledge.

They explicitly state that they exclude question-answer annotations from the training splits of these datasets, adhering to their label-free approach.

Key Insights and Ideas

  • Targeted Knowledge Transfer is Key: The "parity identification" strategy is a significant departure from standard knowledge distillation. By focusing only on where the S-VLM differs from the L-VLM (and the L-VLM is correct), MPA efficiently addresses specific weaknesses rather than broadly mimicking the teacher.
  • Label-Free is Truly Label-Free: The ability to achieve significant performance gains without any human-labeled data is a major advantage, especially for data-scarce domains or when human annotation is prohibitively expensive.
  • MPA Works with Closed-Source Models: The framework's reliance on input-output access to the L-VLM makes it applicable to powerful proprietary models like GPT-40, opening up new avenues for improving smaller, open-source VLMs.
  • Beyond VQA: The paper suggests that MPA can transfer fundamental capabilities like OCR and text-aware captioning, indicating its potential for broader applications than just VQA.

Key Results

  • MPA consistently improves S-VLM performance across all four VQA benchmarks, achieving an average gain of 3.4% and a maximum gain of 15.2% in absolute terms.
  • For example, on ChartQA, MPA-aligned TinyLLaVA-2B achieved a +15.2% absolute improvement over its zero-shot performance.
  • MPA-aligned Qwen2VL-2B, guided by GPT-40, even outperformed a larger model (Qwen2VL-7B) on TextVQA (75.4% vs. 74.7%).
  • The PI module was shown to be crucial, with MPA consistently outperforming MPA without PI filtering, even when using significantly fewer training samples.
  • Qualitative results show MPA correcting errors in visual text reasoning, plot interpretation, and knowledge-based question answering.

Limitations and Future Work

  • Dependency on L-VLMs: MPA requires access to a powerful L-VLM for generating and validating pseudo-annotations. This could limit its applicability in extremely resource-constrained settings.
  • Closed-Source API Issues: Using commercial APIs for closed-source L-VLMs raises concerns about reproducibility and transparency due to limited insight into model behavior and potential API changes.
  • Generalization: The current work focuses on English VQA tasks. Generalization to multilingual settings or more complex reasoning tasks remains an open direction.
  • Future Work: The authors plan to explore more robust knowledge alignment strategies, including iterative refinement of pseudo-labels, leveraging diverse data sources, and integrating multi-step reasoning from L-VLMs into S-VLMs. They also aim to extend MPA to tasks beyond VQA.

Summary updated

BibTeX
@inproceedings{penamakuri2025when,
  author = {Abhirama Subramanyam Penamakuri and Navlika Singh and Piyush Arora and Anand Mishra},
  title = {{When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure