Abstract
Existing models for cross-domain named entity recognition (NER) rely on numerous unlabeled corpus or labeled NER training data in target domains. However, collecting data for low-resource target domains is not only expensive but also time-consuming. Hence, we propose a cross-domain NER model that does not use any external resources. We first introduce a Multi-Task Learning (MTL) by adding a new objective function to detect whether tokens are named entities or not. We then introduce a framework called Mixture of Entity Experts (MoEE) to improve the robustness for zero-resource domain adaptation. Finally, experimental results show that our model outperforms strong unsupervised cross-domain sequence labeling models, and the performance of our model is close to that of the state-of-the-art model which leverages extensive resources.
Introduction
Named Entity Recognition (NER) is a fundamental task in natural language processing, involving the identification and classification of entities such as individuals, organizations, and locations. While modern NER models achieve high performance when training and testing data share the same distribution (e.g., news articles), their efficacy drops significantly when applied to new "target" domains, such as scientific literature or social media.
Traditional solutions involve unsupervised domain adaptation, which requires a large volume of unlabeled text from the target domain to bridge the gap. However, for many niche or emerging fields, even unlabeled data is scarce. This research addresses the zero-resource cross-domain challenge: developing a model capable of adapting to a new domain using only the training data from a single source domain, without any additional target-domain resources.
Methodology
The proposed framework builds upon a standard BiLSTM-CRF architecture, enhanced by two primary components designed to improve generalization and robustness.
Multi-Task Learning (MTL)
To prevent the model from over-specializing on the specific vocabulary of the source domain, the authors introduce a secondary binary classification task. Alongside the standard NER task (identifying specific entity categories), the model must determine whether a token is a named entity or not. This auxiliary objective forces the model to learn the structural and contextual "shape" of entities in general, rather than just memorizing a list of names. This helps the model distinguish potential entities from regular vocabulary even in unfamiliar contexts.
Mixture of Entity Experts (MoEE)
Cross-domain NER often suffers from "context confusion," where the surrounding sentence structure might support multiple entity types. For instance, in the phrase "Arafat met with officials," the context could support either a person or an organization. To mitigate this, the authors developed the Mixture of Entity Experts (MoEE) module. This component consists of individual linear layers (experts) for each entity type and an "expert gate." The gate analyzes the context and dynamically weights the contributions of each expert, creating a robust, meta-expert representation for the final prediction.
Experimental Setup
Datasets
- Source Domain: The model was trained on the CoNLL-2003 English NER dataset, comprising approximately 15,000 samples of general news text.
- Target Domain: Evaluation was conducted on SciTech News, a dataset of 2,000 sentences focused on scientific and technical reporting. This represents a significant domain shift in terms of both vocabulary and stylistic conventions.
Implementation Details
The researchers found that freezing word embeddings (using FastText) during training yielded better results than fine-tuning. Fine-tuning tended to cause the model to overfit to the specific word usage of the source domain, whereas frozen embeddings preserved a more generalizable semantic space.
Results and Discussion
The proposed model demonstrates strong performance in a zero-resource setting, achieving an F1-score of 69.53 on the SciTech dataset. This result outperforms established baselines, such as the Robust Sequence Tagger (68.12).
Notably, the model's performance closely approaches that of state-of-the-art (SOTA) models (73.59) that utilize massive amounts of unlabeled target-domain data. This suggests that the combination of MTL and MoEE effectively captures domain-invariant features of named entities.
One surprising finding was that word-level embeddings (FastText) outperformed subword-based models like BERT in this specific zero-resource setup. The authors hypothesize that because NER is fundamentally a word-level task, the subword tokenization used by BERT may lose crucial information necessary for cross-domain adaptation when no target data is available to recalibrate the subword representations.
Conclusion
This work presents a robust framework for zero-resource cross-domain NER. By leveraging Multi-Task Learning to identify general entity characteristics and a Mixture of Entity Experts to resolve contextual ambiguity, the model achieves competitive results without the need for expensive target-domain data collection. Future research will focus on refining the expert combination mechanisms and testing the framework across more diverse domains, such as legal and medical texts, as well as morphologically rich languages.
Summary updated