Abstract
Embeddings are generic representations that are useful for many NLP tasks. In this paper, we introduce DENSIFIER, a method that learns an orthogonal transformation of the embedding space that focuses the information relevant for a task in an ultradense subspace of a dimensionality that is smaller by a factor of 100 than the original space. We show that ultradense embeddings generated by DENSIFIER reach state of the art on a lexicon creation task in which words are annotated with three types of lexical information - sentiment, concreteness and frequency. On the SemEval2015 10B sentiment analysis task we show that no information is lost when the ultradense subspace is used, but training is an order of magnitude more efficient due to the compactness of the ultradense space.
Introduction
Standard word embeddings, such as word2vec or GloVe, are designed as generic representations, capturing a wide array of syntactic and semantic information across hundreds of dimensions. However, for specific downstream tasks like sentiment analysis, much of this high-dimensional information acts as noise. The challenge lies in isolating task-relevant signals without compromising the inherent structure of the embedding space. We propose DENSIFIER, a method that utilizes orthogonal transformations to compress these generic embeddings into "ultradense" subspaces tailored for specific lexical properties.
Methodology: The DENSIFIER Approach
The core of DENSIFIER is the learning of an orthogonal transformation matrix $Q$. Unlike standard linear layers, an orthogonal matrix preserves the bilinear form, meaning that Euclidean distances and cosine similarities between words remain invariant. This ensures that the transformation only rotates or reflects the space, reordering information rather than adding or removing it.
Subspace Targeting and Training
The objective is to "push" all information relevant to a specific property (e.g., sentiment) into a low-dimensional subspace $E_u$. For tasks like sentiment lexicon creation, this subspace can be as small as a single dimension.
The model is trained using Stochastic Gradient Descent (SGD) to minimize distances between words sharing the same property (e.g., "peace" and "love") and maximize distances between those that differ (e.g., "war" and "love"). To maintain the orthogonality of $Q$ during training, the matrix is re-orthogonalized after each update using Singular Value Decomposition (SVD), where $Q$ is set to $UV^T$.
Experimental Evaluation
We evaluated DENSIFIER across five languages (Czech, English, French, German, and Spanish) using various benchmarks, including SemEval 2015 Task 10 (Sentiment Analysis) and several lexicon resources for concreteness and frequency.
Key Findings
- Efficiency of Low Dimensions: For lexicon creation, a single dimension often suffices to capture lexical strength.
- Harvesting vs. Training: Training high-dimensional embeddings (e.g., 400D) and then "harvesting" relevant information into a subspace (e.g., 60D) outperforms training low-dimensional embeddings from scratch. High-dimensional spaces are better at capturing nuances during initial unsupervised learning.
- Data Efficiency: DENSIFIER can learn effective transformations with as few as 300 training words, making it highly suitable for low-resource scenarios.
Results
DENSIFIER achieved state-of-the-art results on the SemEval 2015 Task 10E (lexicon creation), with a Kendall’s $\tau$ of 0.654. Furthermore, in text classification tasks, using 4-dimensional ultradense embeddings instead of the original 400-dimensional ones resulted in a 44x speedup in training time with negligible loss in accuracy. For instance, on English sentiment classification, accuracy only dropped from 0.666 to 0.662 when reducing dimensions from 400 to 40.
Conclusion
DENSIFIER demonstrates that task-specific information in word embeddings can be effectively concentrated into ultradense subspaces through orthogonal transformations. This not only improves interpretability by isolating specific lexical properties but also significantly enhances computational efficiency for downstream neural network training. Future work will explore decomposing the entire embedding space into multiple interpretable orthogonal subspaces.
Summary updated