Composition of Conditional Random Fields for Transfer Learning

Charles Sutton, Andrew McCallum

Introduction

Transfer learning has become a cornerstone of modern Natural Language Processing (NLP), allowing models to leverage knowledge from general-purpose tasks to solve domain-specific problems. However, the traditional method of chaining models—known as "cascaded transfer"—often suffers from rigid information flow and error propagation. This paper presents a novel approach: the Composition of Conditional Random Fields (CRFs). By performing joint decoding of separately trained models, this method enables a "two-way street" for information, significantly improving performance and data efficiency.

The Problem with Cascaded Transfer

In a typical NLP pipeline, tasks are executed sequentially. For instance, extracting information from an email might involve a part-of-speech (POS) tagger, followed by a named-entity recognizer (NER), and finally a domain-specific extraction model. This "cascade" has two fundamental flaws:

  1. Error Propagation: If an early model (e.g., NER) makes an incorrect classification, the downstream model is forced to accept it as a fixed feature, with no mechanism to correct the error.
  2. One-Way Information Flow: While general models provide useful context to specific tasks, the reverse is rarely true. Domain-specific clues (e.g., the word "Speaker:" preceding a name) could help a general NER model resolve ambiguity, but in a cascade, this feedback loop is absent.

Proposed Method: Composition of CRFs

The authors propose a framework that maintains the efficiency of independent training while gaining the benefits of joint inference.

Independent Training

Models for different tasks (e.g., a general NER model and a specific seminar-extraction model) are trained separately on their respective datasets. This allows researchers to utilize large, pre-existing datasets for subtasks without the computational burden of joint training.

Joint Decoding

At test time, the learned weights of these independent models are combined into a single, grid-shaped Factorial CRF (FCRF).

Structure of a composed CRF showing joint decoding across multiple tasks.

Instead of passing a single "best guess" from one model to the next, the system considers the entire joint distribution of possibilities across all tasks. This "uncertainty preservation" allows a confident main task to effectively "overrule" a subtask's initial hesitation. Inference in these complex structures is handled via Viterbi decoding or loopy belief propagation.

Experimental Results

The approach was evaluated on the CMU Seminar Announcements dataset and the ACE (Automatic Content Extraction) dataset.

Performance Gains

On the seminar data, the joint CRF achieved an overall F1 score of 89.2, outperforming both the cascaded version (88.4) and a baseline with no transfer (87.8). The improvements were most pronounced in difficult fields like "speaker" and "location." In the ACE dataset, joint decoding significantly improved the identification of "nominal" mentions—common nouns referring to entities—which are traditionally harder to detect than proper names. For example, "Organization nominal" detection jumped from an F1 of 35.3 (cascaded) to 40.8 (joint).

Data Efficiency

One of the most compelling findings is the impact on data requirements. Joint decoding acts as a force multiplier for small datasets.

F1 performance vs. training instances, demonstrating the data efficiency of joint decoding.

As shown in the learning curve, the joint model achieves the same level of accuracy as the cascaded model while requiring approximately 25% less training data. This makes the approach particularly valuable for niche domains where annotated data is scarce.

Conclusion

Composition of CRFs offers a powerful alternative to traditional cascaded transfer learning. By allowing information to flow bidirectionally during decoding, the system mitigates error propagation and leverages domain-specific clues to improve general-purpose subtasks. While joint decoding increases computational complexity at inference time, the gains in accuracy and data efficiency provide a strong argument for its adoption in complex NLP pipelines.

Summary updated

Abstract

Many learning tasks have subtasks for which much training data exists. Therefore, we want to transfer learning from the old, general-purpose subtask to a more specific new task, for which there is often less data. While work in transfer learning often considers how the old task should affect learning on the new task, in this paper we show that it helps to take into account how the new task affects the old. Specifically, we perform joint decoding of separately-trained sequence models, preserving uncertainty between the tasks and allowing information from the new task to affect predictions on the old task. On two standard text data sets, we show that joint decoding outperforms cascaded decoding.
BibTeX
@inproceedings{sutton2005composition,
  author = {Charles Sutton and Andrew McCallum},
  title = {{Composition of Conditional Random Fields for Transfer Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure