Cross-Lingual Cross-Domain Nested Named Entity Evaluation on English Web Texts

Barbara Plank

Abstract

This paper introduces EWT-NNER, a novel dataset for Nested Named Entity Recognition (NNER) spanning five diverse English web domains. We address the challenges of domain shift and data scarcity in informal web text, such as emails and forums, where standard Named Entity Recognition (NER) often fails. By employing a multi-task learning approach with a two-headed decoder, we evaluate the impact of cross-lingual transfer from German and Danish. Our findings demonstrate that while nested structures in informal text are inherently difficult to parse, joint training across languages and domains significantly enhances performance, yielding a top average F1 score of 78.01.

Introduction

Most Named Entity Recognition (NER) systems are optimized for "flat" entities, assuming a single category per text span. However, natural language is frequently recursive. For instance, in the phrase "the Chicago-based company," "Chicago" is a location nested within a larger organizational entity. Standard NER models often overlook these internal structures or struggle with derivations (e.g., "Italian cuisine").

Furthermore, NNER research has historically focused on high-quality, formal domains like news or biomedical data. This study tackles the domain gap and data scarcity of nested entities in "wild" web text—including emails, blogs, and forums—where unconventional spelling and informal grammar present significant hurdles for automated systems.

Methods

The researchers utilized a multi-task learning framework via the MaChAmp toolkit. Rather than treating nested layers as a single complex label, they implemented a two-headed decoder architecture:

  1. Encoder: Fine-tuned contextualized embeddings (mBERT and XLM-R for cross-lingual tasks; BERT and RoBERTa for monolingual baselines).
  2. Decoders: Two independent CRF-like heads—one dedicated to the outer entity layer and another for the inner nested layer.

This design enables the model to capture dependencies within each layer while sharing underlying linguistic representations. Performance was measured using the strict span-based F1 metric.

The EWT-NNER Dataset

The primary contribution is EWT-NNER, built upon the English Web Treebank (EWT). It encompasses five distinct domains: Answers (Yahoo!), Newsgroups, Emails (Enron), Weblogs, and Reviews (Google). The dataset includes 12 classes based on four major types (LOC, ORG, PER, MISC), including suffixes for derivations and partial entities. Notably, EWT-NNER features a higher density of nested entities (11.6%) compared to traditional datasets like GermEval (7.7%).

Results and Analysis

Cross-Domain Variation

Evaluation revealed a substantial performance gap between domains. Models performed best on the Weblogs domain, likely due to its stylistic similarity to the formal news data used in pre-training. Conversely, the Email and Answers domains proved most challenging. A primary source of error in these domains was the "lowercase trap," where models failed to recognize entities like "ipod" due to a lack of standard capitalization cues.

Cross-domain performance (Strict F1) across the five English web domains.

Cross-Lingual Synergy

A key finding is that adding German and Danish training data (EN+DA+DE) improved English performance. This suggests that the structural patterns of nested entities are somewhat universal across these Germanic languages. While zero-shot transfer from German/Danish news to English web domains resulted in relatively low F1 scores (52–64), the multi-lingual approach successfully mitigated some of the domain-specific noise.

Performance comparison of zero-shot, in-language, and cross-lingual training configurations.

Conclusion

While in-language training on all five English domains achieved a high F1 of 86.80, the study highlights the persistent difficulty of "noisy" web text. Future work will explore alternative decoding methods, such as graph-based parsing, and unsupervised pre-training on informal corpora to better handle the linguistic irregularities of web-based communication.

Summary updated

BibTeX
@inproceedings{plank2021cross,
  author = {Barbara Plank},
  title = {{Cross-Lingual Cross-Domain Nested Named Entity Evaluation on English Web Texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.158},
}
Figure