Abstract
Bootstrapping is the process of improving the performance of a trained classifier by iteratively adding data that is labeled by the classifier itself to the training set, and retraining the classifier. It is often used in situations where labeled training data is scarce but unlabeled data is abundant. In this paper, we consider the problem of domain adaptation: the situation where training data may not be scarce, but belongs to a different domain from the target application domain. As the distribution of unlabeled data is different from the training data, standard bootstrapping often has difficulty selecting informative data to add to the training set. We propose an effective domain adaptive bootstrapping algorithm that selects unlabeled target domain data that are informative about the target domain and easy to automatically label correctly. We call these instances bridges, as they are used to bridge the source domain to the target domain. We show that the method outperforms supervised, transductive and bootstrapping algorithms on the named entity recognition task.
Introduction
Machine learning models for Natural Language Processing (NLP) tasks, such as Named Entity Recognition (NER), often suffer severe performance drops when applied to out-of-domain data—for instance, training on Newswire articles and testing on conversational speech. This paper addresses the [S+T-] domain adaptation setting, where abundant labeled data exists in a source domain, but only unlabeled data is available in the target domain.
A common semi-supervised approach for this is bootstrapping (or self-training), where a model labels target data, selects its most confident predictions, and retrains itself. However, standard bootstrapping often fails in domain adaptation. The instances a source-trained model is most confident about are typically those that most closely resemble the source data, providing little new information about the target domain. Furthermore, in NER, high-confidence predictions are often dominated by "not-a-name" tokens, which do not help the model learn novel entity patterns.
Methodology: The "Bridge" Concept
To overcome the limitations of standard bootstrapping, the authors propose Domain Adaptive Bootstrapping (DAB). This method categorizes instances into three types:
- Generalists: Domain-independent instances that are easy to label but uninformative.
- Specialists: Highly domain-specific instances that are informative but too difficult for the source model to label correctly.
- Bridges: Instances containing both domain-independent and domain-specific features. These serve as the ideal candidates to bridge the gap between domains.
DAB identifies these "bridges" using a scoring function that compares two classifiers: a source classifier (trained on labeled source data) and a target classifier (trained on unlabeled target data using the source classifier's predictions as pseudo-labels).
Selection Criteria
The scoring function selects instances based on two primary metrics:
- Informativeness: Measured by the difference in entropy between the source and target classifiers' predictions. If the source classifier is uncertain (high entropy) but the target classifier is certain (low entropy), the instance is likely a bridge—novel to the source but consistent within the target domain.
- Reliability: The algorithm estimates the precision of the predicted label using a hold-out split on the source data to ensure the pseudo-labels are likely correct.
DAB employs a dynamic termination criterion, stopping the loop when the top candidates begin to resemble generalists rather than bridges (i.e., when the source classifier becomes more confident than the target classifier).
Experimental Setup
The method was evaluated using the ACE 2005 dataset, focusing on the Entity Detection task. The authors tested three types of adaptation:
- Cross-source transfer: Adapting between Newswire (NW), Broadcast Conversation (BC), Weblog (WL), and Conversational Telephone Speech (CTS).
- Cross-topic transfer: Adapting between topics like Military Operations (MO), Politics (POL), and Terrorism (TER).
- Capitalization recovery: Adapting from lowercase-only Broadcast News to standard mixed-case domains.
Key Results and Insights
DAB consistently outperformed supervised, transductive, and standard bootstrapping baselines across various settings.
Performance Gains
In cross-source transfer using a MaxEnt classifier, DAB achieved an average F-measure of 60.95%, compared to 58.15% for the supervised baseline. In contrast, standard and balanced bootstrapping either stagnated or slightly degraded performance.
The Failure of Transductive Learning
A significant finding was the collapse of Transductive SVMs (TSVMs) under domain shift. While TSVMs leverage unlabeled data, they assume the unlabeled data follows the same distribution as the labeled data. In cross-source experiments, TSVMs collapsed to an average F-measure of 39.00%, while SVM-DAB reached 60.97%.
Conclusion
Standard bootstrapping is often too conservative for domain adaptation, reinforcing existing knowledge rather than exploring the target domain. By leveraging entropy differences to identify "bridge" instances, DAB effectively navigates the domain shift. While the method relies on source-domain precision estimates that may not always hold in radically different target domains, it provides a robust framework for improving NER performance without target-domain labels.
Summary updated