Abstract
Field Embedding is a unified framework designed to jointly learn word and sub-word ("grain") embeddings by integrating diverse linguistic information—including morphology, phonetics, and syntax—into a single model. By capturing both the unique nuances of specific linguistic fields and the shared semantic information across them, this approach significantly outperforms traditional word embeddings and rivals heavy-weight dynamic models like BERT on specific downstream tasks.
Introduction
Traditional word embedding models, such as Word2Vec or GloVe, often treat words as atomic units, thereby ignoring the rich internal structure that provides vital semantic clues. While sub-word models like FastText attempt to address this by utilizing character n-grams, they typically focus on a single linguistic aspect, usually morphology.
This paper identifies two major gaps in existing methodologies: first, a limited scope that ignores "broad linguistic fields" like phonetics (how a word sounds) and syntax (its grammatical role); and second, the production of coarse grain sequences where many components carry little to no semantic weight. The Field Embedding framework is proposed to ingest any number of linguistic fields and extract high-quality, semantically meaningful "grains" to build superior representations.
Methodology
The Field and Grain Abstraction
The core of the framework is the "Field and Grain" abstraction. A "Field" represents a specific linguistic aspect, such as Chinese strokes, English phonemes, or Part-of-Speech (POS) tags. A "Grain" is an individual element within that field.
Fine-Grained Pipeline
To improve grain quality, the model employs a fine-grained pipeline consisting of two primary techniques:
- N-Gram Expansion: This combines adjacent grains to capture higher-level meaning, such as combining two Chinese radicals that together form a specific concept.
- Grain Dropping: This filters out extremely low-frequency grains (noise) and randomly drops high-frequency grains during training to prevent the model from over-relying on common, less-informative units.
Model Architecture and Loss Function
The architecture extends the Continuous Bag-of-Words (CBOW) model, projecting multiple fields into a shared embedding space to predict target words.
A critical innovation is the Dual-Gradient Loss Function ($L_{fe}$), which decomposes learning into two parts: shared information ($grad_0$) common across all linguistic fields, and unique information ($grad_f$) specific to a single field (e.g., a specific phonetic pattern).
Experiments and Results
The model was trained on Chinese and English Wikipedia dumps and evaluated across lexical and downstream tasks.
Key Results
- Lexical Tasks: In Chinese word similarity tasks (Sim240), the Field Embedding model achieved a score of 58.08, significantly outperforming the baseline JWE (53.44) and Word2Vec (48.16).
- Text Classification: On the NewsGroup English dataset, the model's grain embeddings achieved 80.12% accuracy, a nearly 10% improvement over standard Word2Vec.
- Named Entity Recognition (NER): The framework proved highly effective for NER, achieving an F1 score of 92.34, which is remarkably close to BERT's 92.80, despite being a much shallower model.
Discussion and Conclusion
The research demonstrates that phonetic and syntactical information provide significant boosts to word representation. Furthermore, "grain embeddings" ($E_G$) often outperformed "word embeddings" ($E_W$) in downstream tasks, suggesting the model successfully captures universal semantic building blocks. While the model remains static and does not account for polysemy as effectively as Transformers, its efficiency and performance make it a powerful alternative for resource-constrained environments. Future work may involve integrating this framework into the pre-training phase of dynamic models.
Summary updated