Few Shot Transfer Learning BetweenWord Relatedness and Similarity Tasks Using A Gated Recurrent Siamese Network

James O'Neill, Paul BuitelaarAAAI 2018 · DOI

Introduction

In Natural Language Processing (NLP), a fundamental challenge lies in distinguishing between word similarity (words sharing semantic features, like coffee and tea) and word relatedness (words associated by context, like coffee and cup). While humans navigate these nuances intuitively, computational models often struggle to generalize across these distinct semantic dimensions.

This research addresses three critical hurdles in the field:

  1. Data Heterogeneity: Benchmarks vary wildly in focus (e.g., verbs vs. rare words), complicating the creation of universal models.
  2. Data Scarcity: High-quality human-annotated datasets are typically small and expensive to produce.
  3. The Symbol Grounding Problem: Traditional distributional models (like Word2Vec) rely on co-occurrence, lacking "external" knowledge of what words actually mean.

Methods: The Gated Recurrent Siamese Network (GRU-SN)

To overcome these issues, the authors propose the Gated Recurrent Siamese Network (GRU-SN). Unlike standard models that rely on vector proximity, the GRU-SN leverages Wiktionary definitions to ground word meanings.

The GRU-SN architecture processes dictionary definitions through twin gated recurrent units to determine semantic distance.

Architecture and Transfer Strategy

The Siamese architecture consists of two identical GRU sub-networks that process the definitions of a word pair. These definitions are encoded into hidden representations, and the distance between them is used to predict a similarity score.

A key innovation is the model's transfer learning capability. The authors observed that lower hidden layers capture general semantic structures (universal semantics), while higher layers are more task-specific. By transferring weights from a "source" task (e.g., relatedness) and fine-tuning only the upper layers for a "target" task (e.g., similarity), the model achieves high performance even with minimal target data.

Experimental Results

The study evaluated the GRU-SN across eight diverse benchmarks, including SimLex-999, SimVerb-3500, and WordSim353. The model achieved state-of-the-art (SoTA) results on seven of these eight datasets. Notably, on the challenging SimLex-999 dataset, the model improved the Spearman correlation ($\rho_s$) from a baseline of 0.576 to 0.691.

In-Domain Transfer Performance

The effectiveness of the transfer learning approach is particularly evident in "few-shot" scenarios. The model demonstrated that knowledge from one domain (like rare words) could be effectively mapped to another (like general word similarity) with high correlation scores across various regression models, including Huber, Gradient Boosting (GB), Random Forest (RF), and Support Vector Regression (SVR).

Performance metrics for in-domain transfer tasks, showing Pearson (PC) and Spearman (SC) correlations across different regression models.

Key Insights and Conclusion

The research highlights that dictionary definitions provide a richer source of information than simple co-occurrence vectors, particularly for handling rare or morphologically complex words. Furthermore, the success of the layered transfer strategy suggests that Siamese networks naturally organize semantic information hierarchically.

While the model shows high variance on extremely small datasets like RG65, its ability to perform "zero-shot" transfer—such as moving from SimVerb to SimLex with a 0.512 correlation without seeing any SimLex training data—marks a significant step forward in robust semantic modeling. Future work will explore applying this architecture to other pairwise tasks like textual entailment and question-answering.

Summary updated

BibTeX
@inproceedings{oneill2018few,
  author = {James O'Neill and Paul Buitelaar},
  title = {{Few Shot Transfer Learning BetweenWord Relatedness and Similarity Tasks Using A Gated Recurrent Siamese Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11995},
}
Figure