Word Matters: What Influences Domain Adaptation in Summarization?

Yinghao Li, Siyu Miao, Heyan Huang, Yang Gao

Abstract

Domain adaptation is a critical challenge for Large Language Models (LLMs) in specialized tasks like summarization. While model size and data volume are often cited as primary drivers of performance, they fail to capture the nuances of cross-domain shifts. This paper introduces the Learning Difficulty Coefficient ($\lambda$) and demonstrates that performance gains in domain adaptation follow a predictable linear relationship with cross-domain word overlap. Our findings suggest that adaptation success can be estimated prior to training by analyzing vocabulary similarity and task difficulty, providing a more interpretable and efficient framework for LLM fine-tuning.

Introduction

Large Language Models (LLMs) trained on general corpora often struggle when applied to niche domains, such as medical literature or dialogue transcripts. This "domain shift" is typically addressed by increasing model parameters or training data, yet these methods do not explain why some datasets are inherently harder to adapt to than others. This research shifts the focus to word-level semantics to identify the underlying factors governing successful domain adaptation in summarization tasks.

Methodology

The authors propose a framework centered on two primary pillars:

  1. Learning Difficulty Coefficient ($\lambda$): This measures task complexity as the product of the Compression Ratio ($\alpha$, document length vs. summary length) and the Abstraction Level ($\beta$, reciprocal of ROUGE score between source and reference). High $\lambda$ values indicate tasks requiring significant information extraction and rephrasing.
  2. Cross-domain Overlap ($\gamma$): A statistical measure of vocabulary similarity between the source and target domains at the word level, rather than sub-word tokens.

The core hypothesis is that LD-Gain—the product of learning difficulty and the actual ROUGE improvement—is linearly proportional to cross-domain overlap.

Experimental Setup

The study evaluated three model architectures—Bloom-1.1B, Bloom-3B, and Llama-2 7B—across four diverse benchmarks: CNNDM (News), PubMed (Medical), SAMSum (Dialogue), and WikiHow (Instructional). To ensure a fair comparison, training sets were standardized to 35,000 instances, allowing the researchers to isolate the effects of vocabulary and difficulty from raw data scale.

Key Results

The experiments confirmed a strong linear correlation between LD-Gain and cross-domain overlap across all tested architectures. Notably, the total word count in the training set was not a significant predictor of adaptation success, debunking the myth that data volume alone guarantees performance gains.

Figure 2: Linear relationship between LD-Gain and cross-domain overlap across Bloom and Llama-2 models, demonstrating consistency across different model sizes.

This linear relationship enables Zero-Training Prediction. By calculating the overlap and difficulty metrics of a new dataset, researchers can estimate a model's potential ROUGE improvement before committing significant computational resources to fine-tuning.

Figure 6: Validation of the linear relationship in Multi-Domain and Mixed-Domain scenarios, showing the robustness of the LD-Gain metric.

Conclusion and Future Work

By focusing on word-level overlap and task difficulty, this work provides a predictive tool for domain adaptation that is both interpretable and computationally efficient. While the current study focuses on fine-tuning, future research will extend this framework to pre-training and alignment phases, as well as exploring the impact of instruction diversity on the Learning Difficulty Coefficient.

Summary updated

BibTeX
@inproceedings{li2024word,
  author = {Yinghao Li and Siyu Miao and Heyan Huang and Yang Gao},
  title = {{Word Matters: What Influences Domain Adaptation in Summarization?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.715},
}
Figure