Combining embedding methods for a word intrusion task

Finn Årup Nielsen, Lars Kai HansenKONVENS 2019

Abstract

This study evaluates the effectiveness of different word embedding techniques for the "word intrusion" task in Danish. We compare statistical models like fastText and Byte-Pair Encoding (BPE) with contextual models like BERT and knowledge-graph-based embeddings from Wikidata (Wembedder). Our results indicate that while fastText is the strongest individual model, an ensemble combining fastText, Wikidata, and BERT achieves the highest accuracy of 83%, significantly outperforming the 25% random baseline and individual models.

Introduction

The word intrusion task is a standard benchmark for evaluating the semantic coherence of word representations. In this task, a model is presented with a set of four words, three of which belong to a specific category, while the fourth is an "intruder" or outlier. For example, in the set {apple, pear, cherry, chair}, "chair" is the intruder. Successfully identifying the outlier requires a deep understanding of semantic categories, relationships, and world knowledge.

While much research has focused on English, this paper addresses the task for Danish. Danish poses specific challenges, including a smaller set of pre-trained resources and linguistic features like homographs and complex morphology. We explore whether "off-the-shelf" models can be combined to create a more robust semantic representation for Danish.

Methods

We evaluated four primary embedding architectures:

  1. fastText (FT): A model that represents words as bags of character n-grams, making it robust to rare words and morphological variations.
  2. Byte-Pair Encoding (BPE): A subword-based approach. We tuned this model across various vocabulary sizes and dimensions to optimize performance.
  3. BERT: A multilingual transformer-based model. We tested its performance on isolated words to see if its contextual power translates to static word sets.
  4. Wembedder (W): Embeddings derived from the Wikidata knowledge graph, capturing structured relationships between entities.

To identify the intruder, we calculated a correlation matrix for each word set based on cosine similarity. The word with the lowest average similarity to the others was selected as the outlier. We also tested ensemble methods by concatenating vectors from different models (e.g., FT+W+BERT).

Table 1: Examples of word sets and the outliers selected by different models. The ensemble model (FT+W+BERT) leverages multiple signals to identify semantic outliers.

Results

The performance of the models varied significantly across the 100-set Danish benchmark. As shown in Table 2, fastText was the most effective standalone model, achieving 78% accuracy. In contrast, BERT performed poorly at 32%, likely because it is optimized for words within a sentence context rather than isolated tokens.

Table 2: Accuracy scores for individual and ensemble models on the Danish word intrusion task. The ensemble of fastText, Wikidata, and BERT achieves the highest overall performance of 83%.

The integration of Wikidata (Wembedder) provided a notable boost when combined with fastText, particularly for sets involving named entities or specific world knowledge. The final ensemble (FT+W+BERT) reached a peak accuracy of 83%, setting a new performance baseline for this task in Danish.

Discussion

The "BERT Paradox"—where a state-of-the-art model underperforms—highlights the importance of task-specific evaluation. BERT's reliance on context makes it ill-suited for zero-context word sets unless specifically adapted.

The success of the ensemble approach suggests that semantic meaning is multifaceted. fastText captures statistical usage and subword structure, while Wembedder provides a "knowledge backbone" for entities. However, challenges remain. Homographs, such as the Danish word tog (which can mean "train" or "took"), continue to cause errors because static embeddings collapse multiple meanings into a single vector.

Conclusion

We have established a new baseline for the Danish word intrusion task. Our findings demonstrate that ensembling diverse embedding types—statistical, contextual, and knowledge-based—is a powerful strategy for improving NLP performance in languages with fewer resources. Future work will explore providing BERT with the entire word set as a "pseudo-context" and integrating more granular linguistic data from Wikidata lexemes.

Summary updated

BibTeX
@inproceedings{nielsen2019combining,
  author = {Finn Årup Nielsen and Lars Kai Hansen},
  title = {{Combining embedding methods for a word intrusion task}},
  booktitle = {KONVENS},
  year = {2019},
}
Figure