Discriminating between Lexico-Semantic Relations with the Specialization Tensor Model

Goran Glavas, Ivan Vulic

Introduction

Standard word embeddings, such as GloVe or Word2Vec, are highly effective at capturing general semantic relatedness—for instance, identifying that "coffee" and "cup" are associated. However, they often struggle to distinguish specific lexico-semantic relations, such as whether two words are synonyms, antonyms, or part-whole pairs. Distinguishing between "hot" and "cold" (antonyms) versus "hot" and "burning" (synonyms) requires more nuanced transformations of the underlying vector space.

Existing approaches typically rely on either specialized embeddings for a single relation or complex classifiers that process syntactic paths from massive text corpora. The Specialization Tensor Model (STM) addresses these limitations with a lightweight, portable neural architecture that distinguishes multiple relations simultaneously using only raw word vectors.

The Specialization Tensor Model (STM)

The core innovation of the STM is its use of a "tensor" of transformations to analyze word pairs through multiple specialized lenses. Rather than applying a single global transformation, the model learns $K$ distinct views of the data.

Architecture of the Specialization Tensor Model (STM), illustrating the multi-view specialization and bilinear interaction process.

As shown in Figure 1, the process follows three main steps:

  1. Multiple Specializations: Input vectors $(x_1, x_2)$ pass through $K$ non-linear feed-forward layers ($f_S^{(k)}$). each acting as a "slice" of the specialization tensor to emphasize different semantic properties (e.g., hierarchical features vs. oppositional features).
  2. Bilinear Interaction: For each specialized pair, the model computes a "bilinear product score" ($f_P^{(k)}$), measuring the compatibility of the words within that specific subspace.
  3. Final Classification: These $K$ scores are concatenated into a feature vector s and processed by a final softmax layer ($f_{class}$) to predict the specific relation (synonymy, antonymy, hypernymy, meronymy, or no relation).

Evaluation Framework

The authors evaluated the STM across several benchmarks:

  • CogALex-V: A challenging standard benchmark for semantic relations characterized by high class imbalance (71% "random" pairs) and lexical repetitiveness.
  • WN-LS (WordNet-Lexico-Semantic): A larger, more balanced dataset of 10,000 word pairs introduced by the authors.
  • Multilingual WN-LS: To test portability, the authors created versions for German (DE), Spanish (ES), and a "resource-lean" test set for Croatian (HR).

Results and Analysis

The STM demonstrates that projecting vectors into specific subspaces is crucial for identifying diverse relations. A key finding is that "one view is not enough"; the STM provided a significant performance boost over the BILIN-TENS baseline, which lacks the specialization tensor.

Performance comparison on the CogALex-V benchmark, showing STM outperforming LexNet and the BILIN-TENS baseline.

As detailed in Figure 3, the STM achieved a state-of-the-art F1 score of 45.3 on CogALex-V, surpassing the previous best of 42.3 and even outperforming LexNet (44.5), a significantly more complex model that utilizes recurrent neural networks to encode syntactic paths. This suggests that high-quality distributional vectors contain sufficient signal for relation identification when transformed correctly, eliminating the need for expensive corpus-based path mining.

Furthermore, the model excels in Zero-Shot Portability. By aligning vector spaces between languages, an STM trained on English can classify relations in Croatian. For example, training on English and testing on Spanish yielded an F1 of 58.6, remarkably close to the performance of models trained directly on native Spanish data.

Conclusion

The Specialization Tensor Model offers a simple, effective, and language-agnostic solution for lexico-semantic relation discrimination. While its performance is currently tied to the quality of bilingual vector alignment in cross-lingual tasks, its ability to outperform complex syntactic models using only raw embeddings marks a significant step forward. Future work may explore finer-grained relations and more advanced neural architectures to further refine the tensor-based specialization approach.

Summary updated

BibTeX
@inproceedings{glavas2018discriminating,
  author = {Goran Glavas and Ivan Vulic},
  title = {{Discriminating between Lexico-Semantic Relations with the Specialization Tensor Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-2029},
}
Figure