FIRE: Semantic Field of Words Represented as Non-Linear Functions

Xin Du, Kumiko Tanaka-IshiiNeurIPS 2022 · OpenReview

TL;DR

FIRE (FIeld REpresentation) is a novel word embedding framework that represents words as nonlinear functions within a continuous semantic field, rather than as static point vectors. This approach allows models to naturally capture polysemy (multiple word meanings) while maintaining the "compositionality"—the ability to combine word meanings through addition—that makes traditional embeddings so powerful.

The Polysemy Bottleneck

Standard word embeddings like Word2Vec or GloVe map words to single points in a linear vector space. While highly efficient, this creates a "polysemy bottleneck." If a word like "bank" is represented by a single vector, it must sit somewhere between its financial meaning and its geographic (river bank) meaning. This often results in a representation that is a muddy average of both, failing to accurately capture either.

Contextual embeddings like BERT solve this by dynamically changing a word's vector based on the surrounding sentence. However, BERT is massive, computationally expensive, and requires context to function. It doesn't inherently represent the fact that a word has multiple distinct meanings in isolation. The authors of FIRE sought a non-contextual representation that is both interpretable and mathematically capable of handling multiple "modes" of meaning without losing the ability to add words together (e.g., "red" + "apple").

Methods: Words as Semantic Fields

The authors propose representing each word $w$ as a pair: a set of locations ($\mu$) in a $D$-dimensional space and a nonlinear function ($f$) covering that space.

  • The Semantic Field: Think of the space as a physical field, much like an electric or magnetic field. Every point in this field has a "semantic strength."
  • The Function: The function $f(s)$ is implemented as a small neural network—specifically a "pseudo-planar transformation" (a simplified version of a normalizing flow). This function creates "peaks" of semantic strength in the field.
  • Similarity: Instead of a simple dot product, the similarity between two words is calculated by looking at how much "strength" one word's function has at the other word's locations.
  • Compositionality: To represent a phrase or sentence, you simply add the functions and locations of the individual words. Because function addition is linear, the model retains the additive properties of traditional vectors.
Qualitative comparison of FIRE against traditional vector and random-variable representations.

Key Insights and Ideas

  • Interpretability through Low Dimensions: One of the most striking features of FIRE is that it performs remarkably well even in just two dimensions ($D=2$). This allows for direct, human-readable visualization of semantic fields.
  • Multimodality is the Key to Polysemy: Because $f(s)$ is nonlinear, it can have multiple "peaks." In the case of "bank," the model naturally develops one peak near financial terms and another near geographic terms.
  • The "Peak" vs. "Valley" Constraint: By using the negative trace of the Jacobian in their function definition, the authors force the model to create "peaks" (high values) rather than "valleys." This makes it easier for the model to "discover" distinct word senses during training.
Visualization of the 2D semantic field for the word "bank", showing distinct peaks for its financial and geographic meanings.

Key Results

The models were trained on two major corpora: Wacky (3 billion tokens) and text8 (17 million tokens). Evaluation was performed across 12 standard word similarity benchmarks, the Word-in-Context (WiC) task, and Semantic Textual Similarity (STS) tasks.

  • Word Similarity: FIRE achieved results comparable to or better than Word2Vec and GloVe when restricted to the same number of parameters. For example, on the Wacky dataset with 50 parameters per word, FIRE achieved an average Spearman correlation of 49.8%, nearly matching Word2Vec's 50.4%.
  • Polysemy Detection: This is where FIRE truly shines. In predicting the number of word senses using Core WordNet, FIRE significantly outperformed both Word2Vec and BERT. While BERT is excellent at disambiguation with context, FIRE is much better at identifying the inherent "multi-meaning" nature of words in a non-contextual setting.
  • Sentence Similarity: Despite being a non-contextual model, FIRE performed competitively on STS tasks, proving that its additive compositionality is robust.

Limitations and Future Work

While promising, FIRE does come with trade-offs. Training FIRE is more computationally intensive than training standard vectors; training on the 3-billion-token Wacky dataset took about 10 hours on a single GPU. Furthermore, the current implementation relies on Skip-gram with Negative Sampling (SGNS). The authors suggest that more advanced training methods, such as those used for GloVe or incorporating self-attention mechanisms, could further improve performance. Finally, while $D=2$ is excellent for visualization, exploring how the model behaves in much higher dimensions (similar to BERT's 768 or 1024) remains an exciting open area for research.

Summary updated

Abstract

State-of-the-art word embeddings presume a linear vector space, but this approach does not easily incorporate the nonlinearity that is necessary to represent polysemy. We thus propose a novel semantic FIeld REepresentation, called FIRE, which is a $D$-dimensional field in which every word is represented as a set of its locations and a nonlinear function covering the field. The strength of a word's relation to another word at a certain location is measured as the function value at that location. With FIRE, compositionality is represented via functional additivity, whereas polysemy is represented via the set of points and the function's multimodality. By implementing FIRE for English and comparing it with previous representation methods via word and sentence similarity tasks, we show that FIRE produces comparable or even better results. In an evaluation of polysemy to predict the number of word senses, FIRE greatly outperformed BERT and Word2vec, providing evidence of how FIRE represents polysemy. The code is available at https://github.com/kduxin/firelang.
BibTeX
@inproceedings{du2022fire,
  author = {Xin Du and Kumiko Tanaka-Ishii},
  title = {{FIRE: Semantic Field of Words Represented as Non-Linear Functions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=hb6uTCWyrS},
}
Figure