TLDR at SemEval-2022 Task 1: Using Transformers to Learn Dictionaries and Representations

Aditya Srivastava, Harsha Vardhan VemulapatiNAACL 2022

TL;DR

This paper introduces TLDR, a pair of deep learning models that leverage transformer architectures and contrastive learning to learn rich representations from dictionary definitions. Their "definition modeling" system achieved first place in SemEval-2022 Task 1, demonstrating the power of transformers for this task.

What Problem Does This Paper Address?

Dictionaries are incredibly valuable linguistic resources, offering clean and structured information that can be hard to find in noisy, naturally occurring text. However, effectively harnessing this information for NLP tasks, especially for low-resource languages, has been a challenge. Existing approaches have either used dictionaries to learn word representations or tried to generate definitions from representations. This paper tackles the problem of learning meaningful representations from dictionary definitions and, conversely, generating accurate definitions from these representations, specifically within the context of the SemEval-2022 Task 1.

Methods

The authors developed two main systems, collectively called TLDR (Transformers for Learning Dictionaries and Representations), for the two subtasks of SemEval-2022 Task 1:

  • Reverse Dictionary Subtask: This system aims to reconstruct dictionary glosses from pre-existing word embeddings (SGNS, character, and ELECTRA). It employs a DistilBERT-based transformer encoder. Key techniques include:
  • Pretraining: The model is pretrained on a masked language modeling (MLM) objective using only the dictionary glosses.
  • Contrastive Learning: Negative sampling is used to create "strong" and "weak" pairs, encouraging the model to push similar representations closer and dissimilar ones further apart.
  • Multi-objective Loss: The system optimizes a combination of Mean Squared Error (MSE) for reconstruction, Cosine Embedding (CE) loss for similarity, and a contrastive loss (CL) for ranking.
  • Hypothesis: The authors hypothesized that pretraining, individual models for each representation type, and a combined loss function would outperform simpler approaches.
  • Definition Modeling Subtask: This system generates dictionary glosses from word representations. It utilizes a transformer decoder architecture. Key techniques include:
  • Concatenation and Attention: Representations from SGNS, character, and ELECTRA embeddings are concatenated and then processed through a multi-head attention mechanism to capture semantic information.
  • Transformer Decoder: A standard transformer decoder architecture is employed for text generation, with masked self-attention and cross-attention layers.
  • Label Smoothing: Used during training to improve generalization.

Datasets

The primary dataset used is from SemEval-2022 Task 1: Comparing Dictionaries and Word Embeddings (CODWOE). This dataset consists of dictionary glosses paired with SGNS, character, and ELECTRA embeddings. The authors used the provided training (43,608 samples), development (6,375 samples), and test (6,221 samples) splits. No new benchmarks were introduced; the work focuses on improving performance on the existing task.

Key Insights and Ideas

  • Transformers are Powerful for Dictionary Tasks: The success in both subtasks, particularly the first place in definition modeling, highlights the effectiveness of transformer architectures for tasks involving dictionary data.
  • Pretraining is Beneficial: Pretraining the reverse dictionary model on dictionary glosses provided a good prior, leading to faster convergence and competitive results, even without using external pre-trained language models.
  • Multi-representation Fusion Works: Concatenating and using attention over multiple types of word representations (SGNS, character, ELECTRA) significantly boosted performance in the definition modeling task, suggesting that each representation type captures unique semantic information.
  • Contrastive Learning's Nuance: While contrastive learning was beneficial for improving cosine similarity, its impact on cosine ranking was less pronounced than initially expected.
  • MSE Loss Dominance: For the reverse dictionary task, MSE loss appeared to be the most critical component for performance, significantly outperforming CE and CL losses when used alone.

Key Results

  • Definition Modeling: The TLDR system achieved first place on the definition modeling subtask for English data in SemEval-2022 Task 1.
  • Reverse Dictionary: The system achieved competitive performance, ranking fifth, sixth, and seventh for the character, ELECTRA, and SGNS targets, respectively.
  • Ablation Studies:
    • Table 1 showed minimal difference between using pretraining versus random initialization for the reverse dictionary task, though pretraining led to faster convergence.
  • Table 3 indicated that individual models for each representation type slightly outperformed a single multitask model for the reverse dictionary task.
  • Table 6 demonstrated that ELECTRA representations outperformed character and SGNS, but concatenating all three and using attention significantly boosted performance, showing the value of diverse representations.

Limitations and Future Work

  • Contextual Disambiguation: The definition modeling system sometimes struggles with words that have multiple meanings, especially in the absence of sentence context. It can pick a correct definition but may not always infer the intended one.
  • Template Re-use and Repetition: The model tends to learn and reuse definition templates (e.g., "Of, pertaining to") and can occasionally exhibit degenerate repetition in generated text.
  • Future Work:
    • Explore multilingual settings for their models.
  • Compare their models against larger pre-trained language models like BART and BERT.
  • Investigate joint training mechanisms where the two subtask models can inform and improve each other.

Summary updated

BibTeX
@inproceedings{srivastava2022tldr,
  author = {Aditya Srivastava and Harsha Vardhan Vemulapati},
  title = {{TLDR at SemEval-2022 Task 1: Using Transformers to Learn Dictionaries and Representations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure