Unsupervised Domain Adaptation for Event Detection using Domain-specific Adapters

Nghia Ngo Trung, Duy Phung, Thien Huu NguyenACL 2021

TL;DR

This paper introduces a novel framework called DAA (Domain-specific Adapter-based Adaptation) to improve event detection in scenarios where training and testing data come from different domains, especially when target domain labels are unavailable. By leveraging domain-specific adapters within a pre-trained BERT model and employing Wasserstein distance for data selection, DAA significantly boosts performance on target domains, outperforming existing methods.

What Problem Does This Paper Address?

Event detection (ED) is a crucial task in natural language understanding, aiming to identify and classify event triggers within text. A major challenge arises from "domain shift," where models trained on data from one domain (e.g., news articles) perform poorly on data from a different domain (e.g., literary texts) due to variations in language, style, and event expression. Traditional ED methods often rely on large amounts of labeled data, which is expensive and time-consuming to obtain for every new domain. This paper specifically tackles the unsupervised domain adaptation (UDA) problem for ED, meaning the model must learn to perform well on a target domain using only unlabeled data from that domain, in addition to labeled data from a source domain. Existing UDA methods, particularly those based on adversarial learning (like DANN), have shown limited success when applied to large transformer models like BERT in the NLP domain, sometimes even degrading performance.

Methods

The core of the proposed DAA framework lies in its innovative use of domain-specific adapters integrated into a fixed, pre-trained BERT model. Instead of fine-tuning the entire BERT model, which can lead to overfitting on the source domain, DAA injects small adapter modules into each layer of BERT. These adapters are designed to capture domain-specific nuances:

  • Shared-Private Architecture: Three types of adapters are used: source adapters, target adapters, and a joint adapter. Source and target adapters learn private representations for their respective domains, while the joint adapter aims to learn a domain-invariant representation.
  • Layer-wise Domain-Adversarial Training (LDA): To ensure the joint adapter's representations are domain-invariant, an adversarial training approach is applied layer by layer. This involves training a domain classifier to distinguish between source and target representations, and then training the joint adapter to "fool" this classifier.
  • Wasserstein Distance for Data Selection: To further mitigate negative transfer and improve adaptation, the paper uses Wasserstein distance to identify and exclude "hard" or out-of-distribution source instances that might introduce noise. This is achieved by estimating the transport cost between source and target representations and selecting a subset of source samples with the lowest costs.
  • Adapter-wise Domain Disentanglement (ADD): An orthogonality constraint and a self-supervised Masked Language Modeling (MLM) task are used to ensure that the private adapters capture distinct domain knowledge and that the joint adapter focuses on shared information.

Datasets

The paper evaluates its approach on several established datasets for event detection:

  • ACE-05: A widely used dataset for event detection, annotated with 33 event types across 6 domains (Newswire, Broadcast News, Broadcast Conversation, Weblog, Usenet Newsgroups, Conversational Telephone Speech). The experiments use combinations of these domains for source and target settings.
  • TimeBank: A corpus of temporally annotated events from news media.
  • LitBank: A corpus of literary events.

The experiments involve transfer learning scenarios, such as TimeBank-to-LitBank and LitBank-to-TimeBank, and also explore adaptation across different domains within ACE-05.

Key Insights and Ideas

  • Adapters are Key for BERT Adaptation: Fine-tuning the entire BERT model for UDA is problematic. Using lightweight, domain-specific adapters within a fixed BERT backbone is a more effective strategy for adapting large language models to new domains.
  • Wasserstein Distance for Smarter Data Selection: Simply using all source data can be detrimental. Identifying and excluding problematic source instances using Wasserstein distance significantly improves adaptation, especially when domain shifts are large.
  • Layer-wise Adversarial Training is Crucial: Applying domain alignment at each layer of BERT, rather than just the output, leads to more robust domain-invariant representations.
  • The "Bottleneck" Effect: The paper's adapter architecture, which involves down-sampling and then up-sampling, not only helps in aligning domain representations but also increases the model's capacity for the main downstream task.

Key Results

The proposed DAA framework achieves state-of-the-art results on unsupervised domain adaptation for event detection.

  • ACE-05: DAA significantly outperforms baselines (BERT, BERT+DANN, BERT+Adapter) by 3-5 F1 points when transferring to dissimilar domains (wl and un).
  • TimeBank-to-LitBank: DAA achieves an F1 score of 53.6, a substantial improvement over BERT+DANN (44.1) and BERT (42.2).
  • LitBank-to-TimeBank: DAA achieves an F1 score of 61.1, outperforming BERT+DANN (49.6) and BERT (42.7).
  • Ablation Studies: Removing any of the core components (domain-adversarial training, data selection, self-supervised task, or orthogonality constraint) leads to a performance drop, highlighting the importance of each element.

Limitations and Future Work

The authors acknowledge several directions for future work:

  • Extending to More General Settings: Adapting the framework to multi-source domain adaptation and domain generalization scenarios.
  • Novel Domains: Applying the framework to new domains for event detection.
  • Improving Target Domain Adaptation: Developing methods to further leverage the target domain's private adapter to enhance out-of-domain performance.

Summary updated

BibTeX
@inproceedings{trung2021unsupervised,
  author = {Nghia Ngo Trung and Duy Phung and Thien Huu Nguyen},
  title = {{Unsupervised Domain Adaptation for Event Detection using Domain-specific Adapters}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure