TL;DR
FIRE (FIeld REpresentation) is a novel word embedding framework that represents words as nonlinear functions within a continuous semantic field, rather than as static point vectors. This approach allows models to naturally capture polysemy (multiple word meanings) while maintaining the "compositionality"—the ability to combine word meanings through addition—that makes traditional embeddings so powerful.
The Polysemy Bottleneck
Standard word embeddings like Word2Vec or GloVe map words to single points in a linear vector space. While highly efficient, this creates a "polysemy bottleneck." If a word like "bank" is represented by a single vector, it must sit somewhere between its financial meaning and its geographic (river bank) meaning. This often results in a representation that is a muddy average of both, failing to accurately capture either.
Contextual embeddings like BERT solve this by dynamically changing a word's vector based on the surrounding sentence. However, BERT is massive, computationally expensive, and requires context to function. It doesn't inherently represent the fact that a word has multiple distinct meanings in isolation. The authors of FIRE sought a non-contextual representation that is both interpretable and mathematically capable of handling multiple "modes" of meaning without losing the ability to add words together (e.g., "red" + "apple").
Methods: Words as Semantic Fields
The authors propose representing each word $w$ as a pair: a set of locations ($\mu$) in a $D$-dimensional space and a nonlinear function ($f$) covering that space.
- The Semantic Field: Think of the space as a physical field, much like an electric or magnetic field. Every point in this field has a "semantic strength."
- The Function: The function $f(s)$ is implemented as a small neural network—specifically a "pseudo-planar transformation" (a simplified version of a normalizing flow). This function creates "peaks" of semantic strength in the field.
- Similarity: Instead of a simple dot product, the similarity between two words is calculated by looking at how much "strength" one word's function has at the other word's locations.
- Compositionality: To represent a phrase or sentence, you simply add the functions and locations of the individual words. Because function addition is linear, the model retains the additive properties of traditional vectors.
Key Insights and Ideas
- Interpretability through Low Dimensions: One of the most striking features of FIRE is that it performs remarkably well even in just two dimensions ($D=2$). This allows for direct, human-readable visualization of semantic fields.
- Multimodality is the Key to Polysemy: Because $f(s)$ is nonlinear, it can have multiple "peaks." In the case of "bank," the model naturally develops one peak near financial terms and another near geographic terms.
- The "Peak" vs. "Valley" Constraint: By using the negative trace of the Jacobian in their function definition, the authors force the model to create "peaks" (high values) rather than "valleys." This makes it easier for the model to "discover" distinct word senses during training.
Key Results
The models were trained on two major corpora: Wacky (3 billion tokens) and text8 (17 million tokens). Evaluation was performed across 12 standard word similarity benchmarks, the Word-in-Context (WiC) task, and Semantic Textual Similarity (STS) tasks.
- Word Similarity: FIRE achieved results comparable to or better than Word2Vec and GloVe when restricted to the same number of parameters. For example, on the Wacky dataset with 50 parameters per word, FIRE achieved an average Spearman correlation of 49.8%, nearly matching Word2Vec's 50.4%.
- Polysemy Detection: This is where FIRE truly shines. In predicting the number of word senses using Core WordNet, FIRE significantly outperformed both Word2Vec and BERT. While BERT is excellent at disambiguation with context, FIRE is much better at identifying the inherent "multi-meaning" nature of words in a non-contextual setting.
- Sentence Similarity: Despite being a non-contextual model, FIRE performed competitively on STS tasks, proving that its additive compositionality is robust.
Limitations and Future Work
While promising, FIRE does come with trade-offs. Training FIRE is more computationally intensive than training standard vectors; training on the 3-billion-token Wacky dataset took about 10 hours on a single GPU. Furthermore, the current implementation relies on Skip-gram with Negative Sampling (SGNS). The authors suggest that more advanced training methods, such as those used for GloVe or incorporating self-attention mechanisms, could further improve performance. Finally, while $D=2$ is excellent for visualization, exploring how the model behaves in much higher dimensions (similar to BERT's 768 or 1024) remains an exciting open area for research.
Summary updated