Modeling Named Entity Embedding Distribution into Hypersphere

Zhuosheng Zhang, Bingjie Tang, Zuchao Li, Hai ZhaoarXiv 2019 · arXiv

Abstract

Named Entity Recognition (NER) typically relies on static dictionaries, which struggle with the dynamic nature of language. This paper proposes a geometric approach by modeling named entity distributions as hyperspheres in high-dimensional embedding space. By defining entities through their topological structure rather than rigid lists, the authors demonstrate improved performance in both monolingual and cross-lingual NER tasks.

Introduction

Traditional NER models often treat named entities as discrete items in a closed dictionary. However, language evolves rapidly, rendering static lists obsolete. This research shifts the focus toward the "gathering feature" of named entities—the observation that entities like people, places, and organizations occupy specific, dense regions in vector space. By modeling these regions as hyperspheres, the authors provide an "open" definition of entities that can be mapped across languages.

Methodology

The proposed framework consists of three primary components:

  1. Hypersphere Modeling: Word embeddings are analyzed to identify clusters of entity types. Each type is modeled as a hypersphere defined by a center vector ($C$) and a radius ($R$). Any word falling within this sphere is considered a potential named entity.
  2. Cross-lingual Mapping: Using a Wasserstein Generative Adversarial Network (WGAN), the model learns to align embedding spaces between a source language (e.g., English) and a target language (e.g., Chinese). This allows the projection of "entity spheres" across languages without requiring target-language dictionaries.
  3. Feature Enhancement: The Euclidean distance of a word from the hypersphere center is normalized into a $z$-score. This geometric "cue" is then integrated as a feature into state-of-the-art models like BiLSTM-CRF and BERT.
Figure 1: 3D visualization of named entity distributions showing the "gathering feature" where entities cluster in specific regions of the embedding space.

Experimental Setup

The study utilized Word2Vec embeddings trained on Wikipedia dumps (14GB for English, 1.4GB for Chinese). Evaluation was conducted on standard benchmarks: CoNLL 2003 for English, and MSRA/CityU for Chinese. Initial hypersphere parameters were derived from self-collected dictionaries across five languages (English, Chinese, Dutch, German, and Spanish).

Results and Discussion

The integration of hypersphere features consistently improved NER performance across all tested models. In English, the F1-score for a BERT-based model rose from 91.62 to 91.81. In Chinese, the MSRA dataset saw a boost from 95.33 to 95.53 F1 using BERT, while a standard BiLSTM-CRF model improved by over 1.0 F1 point.

Figure 3: Distribution of different entity types (PER, LOC, ORG) in 3D space, illustrating how they share similar centers while maintaining distinct geometric neighborhoods.

A key insight is the "Universal Center" phenomenon, where different entity types often overlap in lower-dimensional projections but separate in higher dimensions. Furthermore, the WGAN-based mapping proved effective for cross-lingual discovery, allowing the identification of Chinese entities using English hypersphere coordinates even in the absence of a Chinese dictionary.

Conclusion

Modeling named entities as hyperspheres offers a robust, geometrically-grounded alternative to static dictionaries. While the method currently depends on initial dictionaries for parameter estimation and is sensitive to embedding quality, it paves the way for more flexible, cross-lingual NER systems. Future work will explore more complex composition functions for multi-word entities to improve their representation in the hypersphere space.

Summary updated

Abstract

This work models named entity distribution from a way of visualizing topological structure of embedding space, so that we make an assumption that most, if not all, named entities (NEs) for a language tend to aggregate together to be accommodated by a specific hypersphere in embedding space. Thus we present a novel open definition for NE which alleviates the obvious drawback in previous closed NE definition with a limited NE dictionary. Then, we show two applications with introducing the proposed named entity hypersphere model. First, using a generative adversarial neural network to learn a transformation matrix of two embedding spaces, which results in a convenient determination of named entity distribution in the target language, indicating the potential of fast named entity discovery only using isomorphic relation between embedding spaces. Second, the named entity hypersphere model is directly integrated with various named entity recognition models over sentences to achieve state-of-the-art results. Only assuming that embeddings are available, we show a prior knowledge free approach on effective named entity distribution depiction.
BibTeX
@misc{zhang2019modeling,
  author = {Zhuosheng Zhang and Bingjie Tang and Zuchao Li and Hai Zhao},
  title = {{Modeling Named Entity Embedding Distribution into Hypersphere}},
  year = {2019},
  eprint = {1909.01065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.01065},
}
Figure