One-Shot Learning for Language Modelling

Talip Ucar, Adrian Gonzalez-Martin, Matthew Lee, Adrian Daniel SzwarcarXiv 2020 · OpenReview

Abstract

Humans can infer a great deal about the meaning of a word using the syntax and semantics of surrounding words, even upon first encounter. In machine learning, learning from one or few examples remains a challenge, particularly in Natural Language Processing (NLP). This work tackles one-shot learning for NLP by adapting the Matching Networks framework and exploring various similarity measures—Cosine, Euclidean, Poincaré, and Minkowski. Using the WikiText-2 dataset for a missing word prediction task, we demonstrate that the optimal distance metric depends on the number of training shots, challenging the notion of a universally superior metric. We also establish a benchmark for one, two, and three-shot learning on this publicly available dataset.

Introduction

Standard deep learning models are typically "data hungry," requiring thousands of examples to generalize concepts. In contrast, humans possess a remarkable ability for few-shot learning. While computer vision has seen significant progress in this area, NLP has historically lagged. Most neural networks are parametric, storing knowledge in fixed weights, which necessitates expensive retraining for new concepts. This paper investigates non-parametric approaches, such as k-Nearest Neighbors, which can incorporate new data instantly. The performance of these models is heavily dependent on the choice of distance metric used to determine similarity between linguistic contexts.

Methods: The Matching Networks Framework

The proposed architecture builds upon the Matching Networks framework, designed to map a small "Support Set" of labeled examples and a "Query" sentence to a predicted label.

The Matching Networks architecture for one-shot language modelling, showing the flow from tokenization to distance calculation and attention.

The system comprises several key components:

  • Two-Level Embeddings: Sentences are initially converted into vectors via max-pooling. Subsequently, a Full Context Embedding (FCE) layer utilizing bidirectional LSTMs processes the support set. This allows the representation of each sentence to be informed by other examples in the set, mimicking contextual comparison.
  • Attention Mechanism: A softmax-based attention scheme calculates the relevance of each support set example to the query.
  • Distance Metric Exploration: The core novelty lies in testing multiple metrics beyond standard Cosine similarity, including Euclidean (L2), Minkowski (p=1 and p=3), and the hyperbolic Poincaré distance.

Experimental Setup

The study utilizes the WikiText-2 dataset, framed as a "missing word" prediction task. The model must identify the correct word to fill a <blank_token> based on a few provided examples.

Example of a Support Set and Query sentence from the WikiText-2 dataset used for the missing word prediction task.

Benchmarks were established for 1, 2, and 3-shot learning scenarios. In a "5-way" task, the model chooses the correct word from five possible options.

Results and Discussion

The experiments revealed a "No Silver Bullet" phenomenon: the effectiveness of a distance metric is tied to the number of shots.

  • 1-Shot Learning: Euclidean distance performed best, achieving 30.1% accuracy.
  • 2-Shot Learning: Cosine similarity was most effective at 32.8%.
  • 3-Shot Learning: Minkowski (p=1, or Manhattan distance) reached the top performance of 37.7%.

These findings challenge the common assumption that a single metric is universally optimal. Interestingly, Poincaré distance did not outperform simpler metrics, likely because the max-pooling embedding scheme was not optimized for hyperbolic space. Furthermore, as the task complexity increased (e.g., to a "20-way" choice), accuracy dropped significantly, indicating non-linear scaling of difficulty in one-shot learning.

Conclusion and Future Work

This research establishes a baseline for few-shot learning in NLP and highlights the importance of metric selection. Future improvements could include using recurrent attentional mechanisms to mitigate LSTM ordering effects, integrating pre-trained models like BERT, and training embeddings directly in hyperbolic space to properly leverage Poincaré distance.

Summary updated

Abstract

Humans can infer a great deal about the meaning of a word, using the syntax and semantics of surrounding words even if it is their first time reading or hearing it. We can also generalise the learned concept of the word to new tasks. Despite great progress in achieving human-level performance in certain tasks (Silver et al., 2016), learning from one or few examples remains a key challenge in machine learning, and has not thoroughly been explored in Natural Language Processing (NLP). In this work we tackle the problem of oneshot learning for an NLP task by employing ideas from recent developments in machine learning: embeddings, attention mechanisms (softmax) and similarity measures (cosine, Euclidean, Poincare, and Minkowski). We adapt the framework suggested in matching networks (Vinyals et al., 2016), and explore the effectiveness of the aforementioned methods in one, two and three-shot learning problems on the task of predicting missing word explored in (Vinyals et al., 2016) by using the WikiText-2 dataset. Our work contributes in two ways: Our first contribution is that we explore the effectiveness of different distance metrics on k-shot learning, and show that there is no single best distance metric for k-shot learning, which challenges common belief. We found that the performance of a distance metric depends on the number of shots used during training. The second contribution of our work is that we establish a benchmark for one, two, and three-shot learning on a language task with a publicly available dataset that can be used to benchmark against in future research.
BibTeX
@misc{ucar2020one,
  author = {Talip Ucar and Adrian Gonzalez-Martin and Matthew Lee and Adrian Daniel Szwarc},
  title = {{One-Shot Learning for Language Modelling}},
  year = {2020},
  eprint = {2007.09679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.09679},
}
Figure