Proceedings of the 5th Workshop on Representation Learning for NLP

Lin Chen, Danlu Chen, Yue Chen, Yu Cheng, Manuel R. Ciosici, William W. Cohen, Christopher Davis, Eliezer De Souza Da, Luciano Silva, Del Corro, Zhihong Deng, Leon Derczynski, Shehzaad Dhuliawala, Giuseppe Antonio, Di Luna, Kalpit Dixit, Aleksandr Drozd, Kevin Duh, Necati Bora, E. Guy, Emerson Fernandes, Orhan Fırat, Rainer Gemulla, Kevin Gimpel, Hongyu Gong, Ana Suárez González, Batool A Haider, He He, Jiaji Huang, Sung Hee Hwang, Robin Jia, Mark Johnson, Arzoo Katiyar, Santosh Kesiraju, Douwe Kiela, Ekaterina Kochmar, Julia Kreutzer, Shankar Kumar, John Lalor, Carolin Lawrence, Kenton Lee, Xiang Li, Shaohua Li, Tao Li, Bill Yuchen Lin, Chu‐Cheng Lin, Peng Liu, Feifan Liu, Suresh Manandhar, Luca Massarelli, Sneha Mehta, Todor Mihaylov, Tsvetomila Mihaylova, Swaroop Mishra, Ashutosh Modi, Lili Mou, Maximilian Mozes, Khalil Mrini, Phoebe Mulcaire, Nikita Nangia, Shashi Narayan, Thien Huu Nguyen, Tsuyoshi Okita, Ankur Padia, Ashwin Paranjape, Tom Pelsmaeker, Aleksandra Piktus, Vassilis Plachouras, Edoardo Maria Ponti, Ratish Puduppully, Leonardo Querzoni, Chris Quirk, Vipul Raheja, Muhammad Mahbubur Rahman, Natraj Raman, Surangika Ranathunga, Siva Reddy, Sravana Reddy, Roi Reichart, Devendra Singh Sachan, Marzieh Saeidi, Avneesh Saluja, Hinrich Schütze, Tianze Shi, Vered Shwartz, Kyungwoo Song, Daniil Sorokin, Lucia Specia, Mark Steedman, Karl Stratos, Ming Sun, Jörg Tiedemann, Ivan Titov, Nadi Tomeh, Shubham Toshniwal, Kristina Toutanova, Lifu Tu, Lyle Ungar, Menno van ZaanenOpenalex Unknown 2020 · DOI · OpenAlex

Abstract

Existing models for cross-domain named entity recognition (NER) rely on numerous unlabeled corpus or labeled NER training data in target domains. However, collecting data for low-resource target domains is not only expensive but also time-consuming. Hence, we propose a cross-domain NER model that does not use any external resources. We first introduce a Multi-Task Learning (MTL) by adding a new objective function to detect whether tokens are named entities or not. We then introduce a framework called Mixture of Entity Experts (MoEE) to improve the robustness for zero-resource domain adaptation. Finally, experimental results show that our model outperforms strong unsupervised cross-domain sequence labeling models, and the performance of our model is close to that of the state-of-the-art model which leverages extensive resources.

Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing, involving the identification and classification of entities such as individuals, organizations, and locations. While modern NER models achieve high performance when training and testing data share the same distribution (e.g., news articles), their efficacy drops significantly when applied to new "target" domains, such as scientific literature or social media.

Traditional solutions involve unsupervised domain adaptation, which requires a large volume of unlabeled text from the target domain to bridge the gap. However, for many niche or emerging fields, even unlabeled data is scarce. This research addresses the zero-resource cross-domain challenge: developing a model capable of adapting to a new domain using only the training data from a single source domain, without any additional target-domain resources.

Methodology

The proposed framework builds upon a standard BiLSTM-CRF architecture, enhanced by two primary components designed to improve generalization and robustness.

Multi-Task Learning (MTL)

To prevent the model from over-specializing on the specific vocabulary of the source domain, the authors introduce a secondary binary classification task. Alongside the standard NER task (identifying specific entity categories), the model must determine whether a token is a named entity or not. This auxiliary objective forces the model to learn the structural and contextual "shape" of entities in general, rather than just memorizing a list of names. This helps the model distinguish potential entities from regular vocabulary even in unfamiliar contexts.

Mixture of Entity Experts (MoEE)

Cross-domain NER often suffers from "context confusion," where the surrounding sentence structure might support multiple entity types. For instance, in the phrase "Arafat met with officials," the context could support either a person or an organization. To mitigate this, the authors developed the Mixture of Entity Experts (MoEE) module. This component consists of individual linear layers (experts) for each entity type and an "expert gate." The gate analyzes the context and dynamically weights the contributions of each expert, creating a robust, meta-expert representation for the final prediction.

Experimental Setup

Datasets

  • Source Domain: The model was trained on the CoNLL-2003 English NER dataset, comprising approximately 15,000 samples of general news text.
  • Target Domain: Evaluation was conducted on SciTech News, a dataset of 2,000 sentences focused on scientific and technical reporting. This represents a significant domain shift in terms of both vocabulary and stylistic conventions.

Implementation Details

The researchers found that freezing word embeddings (using FastText) during training yielded better results than fine-tuning. Fine-tuning tended to cause the model to overfit to the specific word usage of the source domain, whereas frozen embeddings preserved a more generalizable semantic space.

Results and Discussion

The proposed model demonstrates strong performance in a zero-resource setting, achieving an F1-score of 69.53 on the SciTech dataset. This result outperforms established baselines, such as the Robust Sequence Tagger (68.12).

Notably, the model's performance closely approaches that of state-of-the-art (SOTA) models (73.59) that utilize massive amounts of unlabeled target-domain data. This suggests that the combination of MTL and MoEE effectively captures domain-invariant features of named entities.

One surprising finding was that word-level embeddings (FastText) outperformed subword-based models like BERT in this specific zero-resource setup. The authors hypothesize that because NER is fundamentally a word-level task, the subword tokenization used by BERT may lose crucial information necessary for cross-domain adaptation when no target data is available to recalibrate the subword representations.

Conclusion

This work presents a robust framework for zero-resource cross-domain NER. By leveraging Multi-Task Learning to identify general entity characteristics and a Mixture of Entity Experts to resolve contextual ambiguity, the model achieves competitive results without the need for expensive target-domain data collection. Future research will focus on refining the expert combination mechanisms and testing the framework across more diverse domains, such as legal and medical texts, as well as morphologically rich languages.

Summary updated

Abstract

Existing models for cross-domain named entity recognition (NER) rely on numerous unlabeled corpus or labeled NER training data in target domains.However, collecting data for low-resource target domains is not only expensive but also time-consuming.Hence, we propose a cross-domain NER model that does not use any external resources.We first introduce a Multi-Task Learning (MTL) by adding a new objective function to detect whether tokens are named entities or not.We then introduce a framework called Mixture of Entity Experts (MoEE) to improve the robustness for zero-resource domain adaptation.Finally, experimental results show that our model outperforms strong unsupervised cross-domain sequence labeling models, and the performance of our model is close to that of the state-of-theart model which leverages extensive resources.
BibTeX
@inproceedings{chen2020proceedings,
  author = {Lin Chen and Danlu Chen and Yue Chen and Yu Cheng and Manuel R. Ciosici and William W. Cohen and Christopher Davis and Eliezer De Souza Da and Luciano Silva and Del Corro and Zhihong Deng and Leon Derczynski and Shehzaad Dhuliawala and Giuseppe Antonio and Di Luna and Kalpit Dixit and Aleksandr Drozd and Kevin Duh and Necati Bora and E. Guy and Emerson Fernandes and Orhan Fırat and Rainer Gemulla and Kevin Gimpel and Hongyu Gong and Ana Suárez González and Batool A Haider and He He and Ji He and Jiaji Huang and Sung Hee Hwang and Robin Jia and Mark Johnson and Arzoo Katiyar and Santosh Kesiraju and Douwe Kiela and Ekaterina Kochmar and Julia Kreutzer and Shankar Kumar and John Lalor and Carolin Lawrence and Kenton Lee and Xiang Li and Shaohua Li and Tao Li and Bill Yuchen Lin and Chu‐Cheng Lin and Peng Liu and Feifan Liu and Suresh Manandhar and Luca Massarelli and Sneha Mehta and Todor Mihaylov and Tsvetomila Mihaylova and Swaroop Mishra and Ashutosh Modi and Lili Mou and Maximilian Mozes and Khalil Mrini and Phoebe Mulcaire and Nikita Nangia and Shashi Narayan and Thien Huu Nguyen and Tsuyoshi Okita and Ankur Padia and Ashwin Paranjape and Tom Pelsmaeker and Aleksandra Piktus and Vassilis Plachouras and Edoardo Maria Ponti and Ratish Puduppully and Leonardo Querzoni and Chris Quirk and Vipul Raheja and Muhammad Mahbubur Rahman and Natraj Raman and Surangika Ranathunga and Siva Reddy and Sravana Reddy and Roi Reichart and Devendra Singh Sachan and Marzieh Saeidi and Avneesh Saluja and Hinrich Schütze and Tianze Shi and Vered Shwartz and Kyungwoo Song and Daniil Sorokin and Lucia Specia and Mark Steedman and Karl Stratos and Ming Sun and Jörg Tiedemann and Ivan Titov and Nadi Tomeh and Shubham Toshniwal and Kristina Toutanova and Lifu Tu and Lyle Ungar and Menno van Zaanen and Edoardo Ponti},
  title = {{Proceedings of the 5th Workshop on Representation Learning for NLP}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.18653/v1/2020.repl4nlp-1},
}
Figure