Field Embedding: A Unified Grain-Based Framework for Word Representation

Junjie Luo, Xi Chen, Jichao Sun, Yuejia Xiang, Ningyu Zhang, Xiang Wan

Abstract

Field Embedding is a unified framework designed to jointly learn word and sub-word ("grain") embeddings by integrating diverse linguistic information—including morphology, phonetics, and syntax—into a single model. By capturing both the unique nuances of specific linguistic fields and the shared semantic information across them, this approach significantly outperforms traditional word embeddings and rivals heavy-weight dynamic models like BERT on specific downstream tasks.

Introduction

Traditional word embedding models, such as Word2Vec or GloVe, often treat words as atomic units, thereby ignoring the rich internal structure that provides vital semantic clues. While sub-word models like FastText attempt to address this by utilizing character n-grams, they typically focus on a single linguistic aspect, usually morphology.

This paper identifies two major gaps in existing methodologies: first, a limited scope that ignores "broad linguistic fields" like phonetics (how a word sounds) and syntax (its grammatical role); and second, the production of coarse grain sequences where many components carry little to no semantic weight. The Field Embedding framework is proposed to ingest any number of linguistic fields and extract high-quality, semantically meaningful "grains" to build superior representations.

Methodology

The Field and Grain Abstraction

The core of the framework is the "Field and Grain" abstraction. A "Field" represents a specific linguistic aspect, such as Chinese strokes, English phonemes, or Part-of-Speech (POS) tags. A "Grain" is an individual element within that field.

Figure 1: Examples of linguistic fields and their corresponding grain sequences for Chinese and English words.

Fine-Grained Pipeline

To improve grain quality, the model employs a fine-grained pipeline consisting of two primary techniques:

  1. N-Gram Expansion: This combines adjacent grains to capture higher-level meaning, such as combining two Chinese radicals that together form a specific concept.
  2. Grain Dropping: This filters out extremely low-frequency grains (noise) and randomly drops high-frequency grains during training to prevent the model from over-relying on common, less-informative units.

Model Architecture and Loss Function

The architecture extends the Continuous Bag-of-Words (CBOW) model, projecting multiple fields into a shared embedding space to predict target words.

Figure 2: The Field Embedding architecture, illustrating the flow from input grain sequences through linguistic fields to the prediction layer.

A critical innovation is the Dual-Gradient Loss Function ($L_{fe}$), which decomposes learning into two parts: shared information ($grad_0$) common across all linguistic fields, and unique information ($grad_f$) specific to a single field (e.g., a specific phonetic pattern).

Experiments and Results

The model was trained on Chinese and English Wikipedia dumps and evaluated across lexical and downstream tasks.

Key Results

  • Lexical Tasks: In Chinese word similarity tasks (Sim240), the Field Embedding model achieved a score of 58.08, significantly outperforming the baseline JWE (53.44) and Word2Vec (48.16).
  • Text Classification: On the NewsGroup English dataset, the model's grain embeddings achieved 80.12% accuracy, a nearly 10% improvement over standard Word2Vec.
  • Named Entity Recognition (NER): The framework proved highly effective for NER, achieving an F1 score of 92.34, which is remarkably close to BERT's 92.80, despite being a much shallower model.

Discussion and Conclusion

The research demonstrates that phonetic and syntactical information provide significant boosts to word representation. Furthermore, "grain embeddings" ($E_G$) often outperformed "word embeddings" ($E_W$) in downstream tasks, suggesting the model successfully captures universal semantic building blocks. While the model remains static and does not account for polysemy as effectively as Transformers, its efficiency and performance make it a powerful alternative for resource-constrained environments. Future work may involve integrating this framework into the pre-training phase of dynamic models.

Summary updated

BibTeX
@inproceedings{luo2021field,
  author = {Junjie Luo and Xi Chen and Jichao Sun and Yuejia Xiang and Ningyu Zhang and Xiang Wan},
  title = {{Field Embedding: A Unified Grain-Based Framework for Word Representation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.140},
}
Figure