Word Matters: What Influences Domain Adaptation in Summarization?

Yinghao Li, Siyu Miao, Heyan Huang, Yang Gao

Abstract

Adapting Large Language Models (LLMs) to specialized domains for summarization tasks is often a resource-intensive process with unpredictable outcomes. This paper investigates the underlying factors that drive successful domain adaptation. We introduce the Learning Difficulty Coefficient ($\lambda$), which accounts for task-specific compression and abstraction, and LD-Gain, a normalized performance metric. Our analysis reveals a robust linear relationship between cross-domain vocabulary overlap and adaptation performance across various model architectures, including Bloom and Llama-2. Conversely, we find that dataset size (word count) has a negligible impact on performance gains. These insights provide a framework for predicting model performance on new domains without the need for expensive fine-tuning.

1. Introduction

While LLMs demonstrate impressive zero-shot capabilities, their performance often degrades when applied to domains significantly different from their pre-training data, such as medical or legal texts. Traditional explanations for adaptation success often rely on macro-level metrics like model parameters or total training tokens. However, these metrics fail to capture the semantic nuances of domain shift. This paper proposes that "words," as the fundamental units of meaning, are the primary drivers of knowledge transfer. We seek to determine if adaptation performance can be predicted based on vocabulary-level statistics.

2. Methodology

To quantify domain adaptation, we propose a framework based on two primary pillars:

2.1 Learning Difficulty Coefficient ($\lambda$)

Summarization tasks vary in inherent difficulty. We define the Learning Difficulty Coefficient ($\lambda$) as the product of two factors:

  • Compression Ratio ($\alpha$): The ratio of source document length to summary length.
  • Abstraction Level ($\beta$): Measured as the reciprocal of the ROUGE score between the document and its reference summary, representing the degree of paraphrasing required.

The product $\lambda = \alpha \times \beta$ represents the inherent difficulty of the dataset.

2.2 LD-Gain and Cross-domain Overlap ($\gamma$)

To compare performance across domains fairly, we use LD-Gain, defined as $\lambda \times \text{Summarization Gain}$ (the improvement in ROUGE score after fine-tuning). This normalizes the gain by the task's difficulty. The independent variable is Cross-domain Overlap ($\gamma$), a word-level statistical metric calculating the ratio of unique words in the target domain that also appear in the source training data.

3. Experimental Results

We evaluated our framework using four diverse datasets: CNNDM (News), PubMed (Science), SamSum (Dialogue), and WikiHow (Instructional), using a consistent sample size of 35,000 training samples.

3.1 Linear Correlation across Architectures

Experiments across Bloom-1.1B, Bloom-3B, and Llama-2 7B models consistently demonstrated a strong linear correlation between cross-domain overlap ($\gamma$) and LD-Gain. This suggests that the model's ability to adapt is primarily constrained by its familiarity with the target domain's vocabulary.

Linear relationship between Cross-domain Overlap and LD-Gain across Bloom-1.1B, Bloom-3B, and Llama2-7B models.

3.2 Predictability and Validation

The linear relationship allows for the prediction of ROUGE scores on unseen domains by analyzing vocabulary overlap and task difficulty. Validation tests on multi-domain datasets confirmed the generalizability of this linear trend, matching actual experimental results closely.

Validation of the linear relationship in multi-domain scenarios.

4. Discussion

A key finding of this study is the lack of correlation between dataset size (word count) and adaptation performance. When training sets were incrementally increased, performance gains fluctuated without a clear trend. This indicates that the quality and relevance of the data—specifically its vocabulary overlap with the source—are far more critical than the sheer volume of tokens. Choosing relevant data is more important than simply collecting more data.

5. Conclusion

This research highlights the importance of word-level semantics in LLM domain adaptation. By using the Learning Difficulty Coefficient and LD-Gain, researchers can identify predictable patterns in model performance. While the study utilized LoRA for larger models due to hardware constraints, the consistency of the results suggests a fundamental property of domain adaptation. Future work will explore additional quality factors and further investigate the non-linear behavior of dataset volume in fine-tuning.

Summary updated

BibTeX
@inproceedings{li2024word,
  author = {Yinghao Li and Siyu Miao and Heyan Huang and Yang Gao},
  title = {{Word Matters: What Influences Domain Adaptation in Summarization?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.715},
}
Figure