Toward a Computational Multidimensional Lexical Similarity Measure for Modeling Word Association Tasks in Psycholinguistics

Bruno Gaume, Lydia Mai Ho-Dac, Ludovic Tanguy, Cécile Fabre, Bénédicte Pierrejean, Nabil Hathout, Jérôme Farinas, Julien Pinquier, Lola Danet, Patrice Péran, Xavier De Boissezon, Mélanie JuclaNAACL 2019

TL;DR

This paper introduces a novel, multidisciplinary approach to computationally measure the similarity between French words, aiming to better understand how words are related in the human mind. By combining psycholinguistic data with advanced Natural Language Processing (NLP) techniques, the authors propose a "multidimensional lexical similarity" that goes beyond simple co-occurrence to capture nuanced semantic relationships, offering a more robust tool for psycholinguistic research.

What Problem Does This Paper Address?

Understanding how humans access and relate words in their mental lexicon is a fundamental question in psycholinguistics. Traditional methods, like word association tasks, provide valuable insights but face challenges: a lack of comprehensive normative data for French words and the time-consuming, subjective nature of manual analysis. Current computational methods for measuring word similarity, while powerful for tasks like analogy resolution, often aggregate diverse semantic relations (synonymy, hypernymy, association) into a single score, failing to capture the fine-grained distinctions crucial for psycholinguistic interpretation. This paper aims to bridge this gap by developing a computational tool that can automatically characterize the relationships between words as they appear in psycholinguistic experiments, offering a more objective and scalable approach.

Methods

The authors propose a "multidimensional lexical similarity" measure, built upon six computational techniques that leverage three distinct linguistic resources:

  • Corpus-based similarity:
    • FrWaC.1st: Measures similarity based on the frequency of words co-occurring in the large FrWaC French web corpus. This captures a broad range of semantic relations.
  • FrWaC.2nd: Employs distributional semantics, specifically using Word2vec embeddings on the FrWaC corpus. It captures similarity based on words appearing in similar contexts, reflecting a deeper semantic relationship.
  • Dictionary-based similarity:
    • TLF.1st: Utilizes the Trésor de la Langue Française (TLF) dictionary. Similarity is determined if one word appears in the definition of another, creating a graph-based representation.
  • TLF.2nd: Applies a random walk technique on the TLF graph. This "second-order" similarity considers indirect relationships and provides a more robust measure than direct dictionary inclusion.
  • Crowdsourced lexical resource-based similarity:
    • JDM.1st: Based on the "Jeux De Mots" (JDM) crowdsourced lexical resource, where similarity is derived from the number of times words are associated in a word association task.
  • JDM.2nd: Also uses a random walk approach, but applied to the JDM graph, similar to TLF.2nd, capturing more nuanced associations from the crowdsourced data.

A key methodological innovation is the distinction between "first-order" (direct co-occurrence or definition) and "second-order" (contextual or indirect relationships) similarity, and the exploration of how these different approaches interact with various linguistic resources.

Datasets

  • Evolex Pilot Study Dataset: This is a newly collected dataset from a pilot study of the Evolex project. It consists of 559 distinct stimulus-response word pairs generated by 30 participants in a word association task. The stimuli were selected based on grammatical category, syllable count, and frequency. This dataset was manually annotated with 12 tags, including 7 classical semantic relations.
  • FrWaC Corpus: A large French web corpus (1.6 billion words) used for corpus-based similarity measures.
  • TLF (Trésor de la Langue Française): A comprehensive French dictionary used for dictionary-based similarity measures.
  • JDM (Jeux De Mots): A crowdsourced lexical resource from a game where players generate associated words, used for crowdsourced similarity measures.

Key Insights and Ideas

  • Multidimensionality is Key: No single similarity measure captures all aspects of lexical relations. Different resources and methods highlight different types of semantic connections.
  • Second-Order Similarity Matters: Shifting from first-order to second-order similarity (especially with random walks) generally improves the correlation with human-generated response frequencies, suggesting these methods better reflect how words are related in the mind.
  • Resource Specificity: Corpus-based methods (FrWaC) tend to capture co-hyponymy well, while dictionary-based (TLF) and crowdsourced (JDM) methods are better at capturing hypernymy.
  • Classical vs. Non-Classical Relations: All similarity measures show positive correlations with classical semantic relations (synonymy, hypernymy, etc.) but are negatively correlated with non-classical relations like "none found" and "associated" word pairs. This suggests that computational models are better at identifying structured semantic links than looser associations.
  • Surprising Gaps: Even first-order corpus similarity (FrWaC.1st) struggled to capture syntagmatic relations, highlighting the complexity of modeling all linguistic phenomena.

Key Results

  • Correlation with Response Frequency: All six similarity measures showed statistically significant positive correlations with the frequency of stimulus-response pairs in the Evolex dataset.
  • JDM.2nd Dominance: The JDM.2nd measure (crowdsourced, second-order similarity) achieved the highest Spearman correlation coefficient (0.51), indicating it best predicts human response patterns among the tested methods.
  • Improvement with Second-Order: The shift to second-order similarity consistently improved correlation: TLF-based methods saw a 65% increase (0.23 to 0.38), and JDM-based methods an 8% increase (0.47 to 0.51).
  • PCA Analysis: A Principal Component Analysis (PCA) on the similarity measures and semantic relations revealed distinct clustering patterns, visually demonstrating how different measures capture different semantic aspects and how second-order similarity preserves these distinctions.

Limitations and Future Work

  • Exceeding 0.51 Correlation: The authors acknowledge that achieving correlations significantly higher than 0.51 might be challenging and warrants further investigation.
  • Future Directions: The Evolex project plans to incorporate other factors, such as reaction times and results from other cognitive tasks, to further refine their models. The ultimate goal is to use these multidimensional similarity profiles to evaluate how various factors (age, education, language deficits) influence word association patterns, serving as a diagnostic tool in clinical studies. The paper also suggests that word embeddings, due to their ability to represent words in a vector space, can be used to identify clusters of responses, aiding in the identification of phenomena as hidden variables.

Summary updated

BibTeX
@inproceedings{gaume2019toward,
  author = {Bruno Gaume and Lydia Mai Ho-Dac and Ludovic Tanguy and Cécile Fabre and Bénédicte Pierrejean and Nabil Hathout and Jérôme Farinas and Julien Pinquier and Lola Danet and Patrice Péran and Xavier De Boissezon and Mélanie Jucla},
  title = {{Toward a Computational Multidimensional Lexical Similarity Measure for Modeling Word Association Tasks in Psycholinguistics}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure