Distilling Cross-Modal Knowledge into Domain-Specific Retrievers for Enhanced Industrial Document Understanding

Jinhyeong Lim, Jeongwan Shin, Seeun Lee, Seongdeok Kim, Joungsu Choi, Jongbae Kim, Chun Hwan Jung, Youjin KangEMNLP 2025

TL;DR

This paper introduces a novel knowledge distillation framework that trains efficient, domain-specific retrievers for industrial documents by transferring knowledge from both Large Language Models (LLMs) and Vision-Language Models (VLMs). This approach significantly improves retrieval accuracy and structural understanding, outperforming larger models while being computationally efficient for real-world applications.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) systems are powerful, but their effectiveness in specialized industrial domains is hampered by the complexity of documents. These documents often contain not just specialized terminology but also crucial structural elements like tables, figures, and formulas (Document Structural Elements - DSE). Traditional retrievers struggle to understand both the domain-specific semantics and these visual/structural cues. While some methods use LLMs for domain adaptation or VLMs for visual understanding, they often overlook structural complexity or incur high computational costs, making them impractical for industrial settings. The authors aim to bridge this gap by creating efficient, domain-specific retrievers that can effectively handle both semantic and structural aspects of industrial documents.

Methods

The core of the proposed method is a cross-modal knowledge distillation framework. This framework trains a compact "student" retriever by distilling knowledge from larger "teacher" models:

  1. LLM-based Synthetic Query Generation: An LLM is used to generate synthetic queries. This process involves:
  2. Parsing documents to identify and crop visual elements.
  3. Using a VLM to generate textual descriptions of these visual elements, aligning them with potential queries.
  4. Sampling a positive document and identifying "hard negative" documents (semantically similar but contextually irrelevant).
  5. Instructing the LLM to generate a query that is highly relevant to the positive document but distinct from the negatives. This creates high-quality, domain-specific training data.
  6. VLM-based Synthetic Ranking Scores: A VLM is employed to generate fine-grained ranking scores. This involves:
  7. Pairing the LLM-generated query with positive and negative documents.
  8. The VLM outputs scores that reflect the visual-semantic alignment, considering layout, structure, figures, and other format-specific elements. These scores act as a supervision signal.
  9. Training the Student Retriever: The student retriever is trained using a combination of contrastive learning (to rank positive documents higher than hard negatives) and knowledge distillation (using the KL divergence loss to match the VLM's ranking score distribution). This joint training allows the retriever to learn both semantic relevance and structural awareness.

Datasets

The authors conducted experiments on two industrial domains:

  • Shipbuilding Domain: A training set of 6,625 samples from 10 PDF documents and a test set of 2,932 samples from 9 PDF documents.
  • Electrical Equipment Domain: A training set of 4,109 samples from 32 PDF documents and a test set of 3,312 samples from 36 PDF documents.

They also constructed a specialized dataset in the shipbuilding domain with 100 instances (20 per type) to evaluate RAG performance across different multimodal elements (figures, tables, formulas, charts, and plain text).

Key Insights and Ideas

  • Bridging the Semantic-Structural Gap: The most significant insight is that effectively retrieving industrial documents requires understanding both their textual content and their visual/structural layout. The proposed cross-modal distillation method successfully addresses this by combining LLM-driven semantic understanding with VLM-driven structural awareness.
  • Efficiency Through Distillation: Distilling knowledge from powerful but computationally expensive teacher models (LLMs and VLMs) into a compact student retriever is a highly effective strategy for achieving high performance with significantly reduced computational cost and latency. This makes advanced retrieval practical for real-world industrial applications.
  • Synthetic Data is Key: The framework's reliance on LLM-generated synthetic queries and VLM-generated ranking scores demonstrates the power of carefully crafted synthetic data for domain adaptation and training robust retrievers, especially when real-world annotated data is scarce or expensive.
  • Teacher Model Choice Matters: The visual retriever ColQwen2.5-3B, despite its complexity, proved to be an excellent teacher model, highlighting the value of advanced multimodal models in guiding the training of more efficient student retrievers.

Key Results

  • Superior Retrieval Performance: The proposed retriever consistently outperformed all baseline models (including OCR-based text retrievers, visual retrievers, and commercial embedding models) in both the shipbuilding and electrical domains across metrics like MRR and HIT@10.
  • Efficiency Gains: The distilled retriever achieved state-of-the-art performance with significantly lower latency and FLOPs compared to larger baselines, making it suitable for latency-sensitive industrial deployments.
  • RAG Performance: When integrated into a RAG system, the proposed retriever achieved performance comparable to that of the teacher model, demonstrating its effectiveness in improving downstream generation tasks.
  • Robustness Across Domains: The framework showed consistent improvements in both the shipbuilding and electrical domains, suggesting its generalizability to various industrial contexts.
  • Ablation Study: Both LLM-based query generation and VLM-based ranking score distillation were shown to be crucial components, contributing significantly to the overall performance improvement.

Limitations and Future Work

The authors acknowledge two primary limitations:

  • Table-based Query Retrieval: Performance on table-based queries could be further improved.
  • Conditional Query Retrieval: Handling conditional queries, especially those involving complex dependencies or multi-hop reasoning, remains a challenge.

For future work, they plan to explore:

  • Targeted supervision strategies for knowledge distillation to enhance reasoning over markdown-formatted tables.
  • Developing methods to improve the retrieval of conditional contexts involving multi-hop queries.

Summary updated

BibTeX
@inproceedings{lim2025distilling,
  author = {Jinhyeong Lim and Jeongwan Shin and Seeun Lee and Seongdeok Kim and Joungsu Choi and Jongbae Kim and Chun Hwan Jung and Youjin Kang},
  title = {{Distilling Cross-Modal Knowledge into Domain-Specific Retrievers for Enhanced Industrial Document Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure