Understanding Feature Focus in Multitask Settings for Lexico-semantic Relation Identification

Houssam Akhmouch, Gaël Dias, Jose G. MorenoACL 2021

TL;DR

This paper proposes a novel approach to identifying lexico-semantic relations between words by combining specialized feature engineering with multitask learning architectures. They demonstrate that carefully selected combinations of symmetric, asymmetric, and pattern-based features, when integrated into shared-private multitask models, significantly improve performance over existing methods.

What Problem Does This Paper Address?

Automatically understanding the relationships between words (like synonymy, hypernymy, etc.) is crucial for many Natural Language Processing (NLP) tasks, from question answering to text summarization. While various methods exist, they often focus on either hand-crafting features or fine-tuning latent semantic spaces (like word embeddings). A key challenge is effectively combining these approaches, especially within multitask learning frameworks where multiple related tasks are learned simultaneously. Existing multitask methods often use generic features or struggle to integrate specialized ones, and there's a lack of comprehensive study on how different types of features (symmetric vs. asymmetric) interact within these complex architectures. This paper aims to bridge this gap by investigating how to best engineer and combine features for lexico-semantic relation identification, particularly within shared-private multitask models.

Methods

The authors introduce three families of features designed for continuous semantic spaces:

  • Distributional Representation: This is the standard approach of concatenating word embeddings (w₁ ⊕ w₂).
  • Symmetric Distributional Features (CosF): These extend the basic cosine similarity by incorporating element-wise multiplication (generalized cosine, cosG) and its variations (cosBr, cosG1D) to capture richer distributional information.
  • Asymmetric Distributional Features (KullF): Inspired by discrete methods, these features leverage the Kullback-Leibler (KL) divergence to capture directional relationships between words. They are adapted for continuous spaces using sigmoid functions and include multiplicative versions (kullG, kullBr, kullG1D).
  • Pattern-based Paradigmatic Features (PatF): These features encode specific word sequences (patterns) that link word pairs, using either BiLSTM or Universal Sentence Encoder (USE) for encoding.

These features are then integrated into two multitask architectures:

  • Fully-Shared Architecture: A single shared representation learns all tasks.
  • Shared-Private Architecture: This architecture has a shared layer for common information and private layers for task-specific information, allowing for a more nuanced focus on features.

A forward selection process is used to systematically find the optimal combinations of these features for both shared and private layers.

Datasets

The paper evaluates their approach on four established gold-standard datasets for lexico-semantic relation identification:

  • RUMEN: Introduced for synonymy.
  • ROOT9: Contains word pairs randomly extracted from various sources.
  • WEEDS: Originally proposed for lexical similarity measures.
  • BLESS: A well-known dataset for distributional semantics.

Additionally, they extracted patterns from Wikipedia dumps to augment the datasets, as patterns were not present in the original ones.

Key Insights and Ideas

  • Feature Combinations Matter: Simply using individual features is often suboptimal. Combining features within a family (e.g., multiple symmetric features) consistently improves performance.
  • Asymmetric Features are Crucial: Asymmetric distributional features (KullF) are particularly effective for identifying asymmetric relations (like hypernymy) and play a significant role in shared layers of multitask models, even when private models focus on symmetric features.
  • Shared-Private Architectures Excel: These architectures demonstrate a superior ability to balance feature focus between private (task-specific) and shared (common) layers, leading to better overall performance compared to fully-shared models.
  • Symmetry is Key for Synonymy: For symmetric relations like synonymy, symmetric distributional features (CosF) are often sufficient and lead to the best results, highlighting the importance of matching feature type to relation type.
  • Patterns Add Value, Especially with More Data: Pattern-based features (PatF) can be powerful, particularly when a large number of patterns are available (as seen with the ROOT9 dataset).

Key Results

The paper reports significant improvements over state-of-the-art models. For instance, their "Best Shared-private" (Best SP) model, using optimized feature combinations, statistically outperforms previous methods on most datasets and relation types. While specific F1 scores vary across datasets and relations, the general trend shows substantial gains. For example, in Table 3, the Best SP model achieves F1 scores in the high 0.8s and 0.9s for many tasks, often outperforming the "Best MLP" (which represents the best single-task model with feature combinations) and "Best Fully-shared" models. The paper also shows that their approach is competitive with, and sometimes surpasses, models using specialized spherical text embeddings (JoSE).

Limitations and Future Work

  • Computational Cost: The forward selection process for feature optimization can be computationally intensive.
  • Dataset Specificity: While the approach shows broad applicability, the performance of pattern-based features is noted to be dependent on the availability of a large number of patterns.
  • Future Work: The authors suggest further exploration into optimizing the feature selection process and investigating more complex multitask architectures. They also hint at the potential for combining their feature-based approach with advanced neural semantic spaces.

Summary updated

BibTeX
@inproceedings{akhmouch2021understanding,
  author = {Houssam Akhmouch and Gaël Dias and Jose G. Moreno},
  title = {{Understanding Feature Focus in Multitask Settings for Lexico-semantic Relation Identification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure