Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings

Saniya Karwa, Navpreet SinghNAACL 2025

TL;DR

This paper introduces a novel framework and dataset (LDSP-10) to pinpoint which dimensions within neural network embeddings (like BERT) encode specific linguistic properties. They found that some linguistic features, like negation, are surprisingly localized to a few dimensions, while others, like synonymy, are more distributed, offering a step towards more interpretable and controllable language models.

What Problem Does This Paper Address?

Neural network embeddings, particularly those from large language models like BERT, are incredibly powerful but notoriously opaque. Understanding why they make certain predictions or how they represent language is a major challenge. While embeddings capture complex linguistic nuances, their high-dimensional nature makes it difficult to decipher which specific "neurons" or dimensions are responsible for encoding particular linguistic features (e.g., tense, negation, synonymy). This lack of interpretability hinders our ability to debug models, mitigate biases, and develop more transparent AI systems.

Methods

The authors propose a framework for analyzing embedding dimensions. Key components include:

  • Linguistically Distinct Sentence Pairs (LDSP-10) Dataset: A new dataset designed to isolate ten specific linguistic properties (LPs) by creating pairs of sentences that differ in only one LP while keeping the overall meaning similar. This allows for targeted analysis.
  • Wilcoxon Signed-Rank Test: A statistical test used to determine if there's a significant difference in embedding dimensions between paired sentences.
  • Mutual Information (MI): Used to quantify the dependence between embedding dimensions and the linguistic property.
  • Recursive Feature Elimination (RFE): A machine learning technique to identify the most important dimensions for predicting an LP.
  • Embedding Dimension Importance (EDI) Score: A novel metric that combines the results from the Wilcoxon test, MI, and RFE to provide a single score indicating a dimension's relevance to a specific LP.

Datasets

  • LDSP-10 Dataset: A newly curated dataset consisting of 1000 sentence pairs for each of the 10 linguistic properties investigated (definiteness, factuality, intensifier, negation, polarity, quantity, synonym, tense, control, and voice). This dataset was generated using Google's gemini-1.5-flash model API.
  • Standard Datasets: Not explicitly mentioned for training, but the analysis is performed on embeddings from pre-trained models like BERT, GPT-2, and MP-Net.

Key Insights and Ideas

  • Linguistic Properties are Not Equally Distributed: Some linguistic features are surprisingly "localized" to a small number of embedding dimensions, while others are more "distributed" across many dimensions.
  • Negation and Polarity are Highly Localized: These properties show very strong signals in specific dimensions, suggesting they are robustly encoded.
  • Synonymy is Elusive: This property proved difficult to disentangle, with its encoding appearing more complex and spread out, making it hard to isolate specific dimensions. This might be due to the nature of synonymy itself and the dataset's construction.
  • EDI Score as a Reliable Indicator: The proposed EDI score effectively identifies dimensions relevant to specific LPs, as demonstrated by evaluation experiments where high-EDI dimensions were sufficient to achieve near-baseline classification accuracy.
  • Cross-Property Generalization: Dimensions important for one LP are generally not as informative for others, suggesting specialization, though some conceptual similarities (like negation and polarity) show some cross-property transfer.

Key Results

  • Negation: 13 dimensions showed an EDI score of 0.8 or above, with a maximum EDI score of 0.9987.
  • Polarity: Similar to negation, with a maximum EDI score of 0.9977 and over 20 dimensions with scores above 0.8.
  • Definiteness: Achieved a perfect EDI score of 1.0 for dimension 180.
  • Control Group: Showed very low EDI scores (max 0.3683), as expected for unrelated sentences.
  • Synonym: Exhibited low maximum EDI scores and inconsistent results across methods, indicating a more complex encoding.
  • Classifier Performance: The LP classifier achieved a test accuracy of 0.863, demonstrating that the embedding difference vectors contain sufficient separable information to distinguish between LPs.
  • Efficiency: For many LPs, fewer than 12 high-EDI dimensions were needed to recover at least 95% of the baseline classifier's accuracy.

Limitations and Future Work

  • Dataset Generation Quality: Generating high-quality, diverse, and non-repetitive LDSPs with LLMs is challenging, and overly simplistic outputs can be difficult to avoid.
  • Model Scale: Experiments were limited to smaller, open-source models due to computational constraints. Analysis on larger, state-of-the-art models could solidify generalizability.
  • Layer-wise Analysis: Future work could analyze EDI scores across different layers of the network to understand how LP information propagates.
  • Bias and Control: The method could potentially isolate dimensions responsible for encoding sensitive attributes like gender, which might introduce biases. Further experiments using downstream task accuracy and counterfactual measures are needed to validate this.
  • EDI-Informed Modifications: Future work could explore making EDI-informed modifications to dimensions to observe information loss or changes in model behavior.

Summary updated

BibTeX
@inproceedings{karwa2025disentangling,
  author = {Saniya Karwa and Navpreet Singh},
  title = {{Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure