Analysis and Prediction of NLP Models Via Task Embeddings

Damien Sileo, Marie-Francine Moens

Abstract

Task embeddings are low-dimensional representations that are trained to capture task properties. In this paper, we propose MetaEval, a collection of $101$ NLP tasks. We fit a single transformer to all MetaEval tasks jointly while conditioning it on learned embeddings. The resulting task embeddings enable a novel analysis of the space of tasks. We then show that task aspects can be mapped to task embeddings for new tasks without using any annotated examples. Predicted embeddings can modulate the encoder for zero-shot inference and outperform a zero-shot baseline on GLUE tasks. The provided multitask setup can function as a benchmark for future transfer learning research.

Introduction

The prevailing paradigm in Natural Language Processing (NLP) involves fine-tuning large-scale pre-trained models, such as RoBERTa, on specific downstream tasks. While effective, this approach is computationally expensive and lacks interpretability regarding how the model's internal logic adapts to different linguistic challenges. Furthermore, traditional fine-tuning is inapplicable in zero-shot scenarios where no labeled data is available. This research addresses these challenges by representing tasks as compact, low-dimensional vectors—task embeddings—that act as control mechanisms for a shared model.

Methods: Conditional Adapters

To efficiently handle a vast array of tasks, the researchers utilize Conditional Adapters. Instead of training separate parameters for each task, a single shared adapter layer is integrated into the transformer architecture. This layer is modulated by a 32-dimensional task embedding ($z_i$).

High-level architecture of the MetaEval framework, showing how task embeddings modulate a shared model.

The framework involves three key stages:

  1. Joint Training: A single RoBERTa model is trained across 101 diverse tasks simultaneously, learning a unique embedding $z$ for each task.
  2. Probing: The learned embeddings are analyzed to determine if they capture meaningful linguistic and structural information.
  3. Task Embedding Regression: A secondary model is trained to predict these embeddings based on high-level task "aspects" (e.g., task type, domain, text length), enabling adaptation to new tasks without training data.

The MetaEval Dataset

The study introduces MetaEval, the most extensive collection of NLP classification tasks compiled for this type of analysis. It comprises 101 tasks sourced from:

  • Standard Benchmarks: GLUE and SuperGLUE.
  • Social Media Analysis: Sentiment, emotion, and stance detection from CrowdFlower.
  • Linguistic Probing: Tasks focusing on syntax, sentence length, and grammaticality (e.g., Blimp).
  • Discourse and Logic: PragmEval and NLI-style conversions (Recast).

Analysis: The Geometry of Tasks

Visualizing the learned task embeddings using UMAP reveals a structured "map" of the NLP task space. Rather than being randomly distributed, the embeddings form distinct clusters based on task characteristics.

UMAP visualization of task embeddings clustered by task type.

As shown in the visualization, categories such as grammar, sentiment, and emotion tasks occupy specific regions of the embedding space. This suggests that the model learns a universal representation of task properties. Furthermore, the researchers found that lower-level tasks (like syntax checking) result in more stable embeddings across training runs compared to higher-level reasoning tasks like Natural Language Inference (NLI).

Key Results

The conditional adapter approach demonstrates both efficiency and effectiveness:

  • Parameter Efficiency: The model achieves 79.7% accuracy on MetaEval while using only 32 task-specific parameters per task—a massive reduction compared to the 124 million parameters required for full fine-tuning.
  • Zero-Shot Transfer: By predicting embeddings from high-level metadata, the model achieved 70.4% average accuracy on the GLUE benchmark in a zero-shot setting, outperforming baseline methods.
  • Superior Representations: The learned embeddings $z$ were found to be more effective at capturing task-specific logic than simple text-based embeddings (TextEmb), indicating that they represent the nature of the problem rather than just the vocabulary of the dataset.

Conclusion and Future Work

While MetaEval provides a powerful framework for task analysis, it is currently limited to classification. Future research could extend this to generative tasks like translation or summarization. Additionally, while zero-shot performance is promising, a gap remains compared to fully supervised models. Improving task descriptors and using more complex regression models (e.g., hypernetworks) are potential avenues to bridge this performance gap and further refine our understanding of the NLP task landscape.

Summary updated

Abstract

Task embeddings are low-dimensional representations that are trained to capture task properties. In this paper, we propose MetaEval, a collection of $101$ NLP tasks. We fit a single transformer to all MetaEval tasks jointly while conditioning it on learned embeddings. The resulting task embeddings enable a novel analysis of the space of tasks. We then show that task aspects can be mapped to task embeddings for new tasks without using any annotated examples. Predicted embeddings can modulate the encoder for zero-shot inference and outperform a zero-shot baseline on GLUE tasks. The provided multitask setup can function as a benchmark for future transfer learning research.
BibTeX
@inproceedings{sileo2022analysis,
  author = {Damien Sileo and Marie-Francine Moens},
  title = {{Analysis and Prediction of NLP Models Via Task Embeddings}},
  booktitle = {LREC},
  year = {2022},
  url = {https://openreview.net/forum?id=LpHFWsJk4f},
}
Figure