MLT-DR: Multi-Lingual/Task Demonstration RetrievalAn Attempt towards Generalized Retriever for In-Context Learning

Kazuma Hashimoto, Arjun Akula, Karthik Raman, Michael Bendersky2024 · DOI

Abstract

This paper presents Multi-Lingual/Task Demonstration Retrieval (MLT-DR) for in-context learning with Large Language Models (LLMs). Our goal is to investigate how dense demonstration retrieval models are generalized across languages and tasks. We first convert 81 tasks into a common format, covering various languages, task types, and domains. For 8 English-based tasks among them, we use machine translation to create synthetic multi/cross-lingual tasks, by translating the examples into non-English languages to explicitly cover more than 130 languages. We then use an instruction-tuned LLM to estimate utility of demonstrations for all the tasks to train the demonstration retrieval models. In our experiments, we show an interesting counterintuitive observation; to compute embeddings of demonstrations, using both the input and ground-truth output hurts the generalization ability of the retriever on unseen tasks whose output space is quite different from those in the seen task set. We also examine that our retriever robustly works even with LLMs that we did not touch during the development of the models.

Introduction

In-Context Learning (ICL) has revolutionized how we interact with Large Language Models (LLMs), allowing them to perform new tasks by simply providing a few examples in the prompt. However, the effectiveness of ICL is notoriously sensitive to the specific examples chosen. While dense retrievers have been developed to select optimal demonstrations, they often struggle to generalize across different languages or tasks with novel output formats. This research introduces MLT-DR, a framework designed to build a truly generalized retriever capable of handling over 130 languages and diverse task types.

Methodology and Data Scope

The MLT-DR framework utilizes a bi-encoder architecture trained via contrastive learning. To bridge the gap between simple text similarity and actual model performance, the authors employed an "LLM-in-the-loop" approach. An instruction-tuned model (Flan-PaLM2) was used to score the utility of potential demonstrations, providing a ground-truth signal for training the retriever.

To ensure broad coverage, the researchers curated a massive dataset of 81 "seen" tasks, ranging from machine translation to sentiment analysis.

Table 1: The 81 'seen' tasks across various languages and domains used to train the MLT-DR retriever.

Furthermore, to address the scarcity of multilingual data, they applied machine translation to 8 core English tasks, synthetically expanding the language coverage to over 130 languages.

The "Output Overfitting" Phenomenon

One of the most significant contributions of this study is the investigation of encoding strategies. The researchers compared three approaches for representing demonstrations in the vector space:

  1. STD (Standard): [Instruction + Input + Output]
  2. DESC (Description): [Instruction + Input + Output Description]
  3. NO (Input Only): [Instruction + Input]

Conventional wisdom suggests that including the ground-truth output (STD) should provide the most context. However, the results revealed a surprising counterintuitive trend: the NO strategy frequently outperformed the standard approach on unseen tasks.

Table 2: Performance comparison of encoding strategies on unseen tasks, demonstrating that the NO strategy (ignoring outputs) often generalizes better than the standard (STD) approach.

As shown in the evaluation of unseen tasks like CLINC150 (a fine-grained intent classification task), the standard encoding strategy actually led to performance decreases (e.g., -1.34 to -1.96), while the NO strategy provided consistent gains (up to +2.35). This suggests that retrievers can "overfit" to specific label spaces; by ignoring the output, the retriever focuses on the semantic nature of the query, which generalizes far better to tasks with novel output formats.

Results and Generalization

Beyond task generalization, MLT-DR demonstrated remarkable success in multilingual and cross-lingual settings. The MT-augmented models showed significant improvements in low-resource languages, such as a +5.72 gain on Hindi and Turkish variants of the ATIS-intent task.

Importantly, the retriever proved to be "LLM agnostic." Although trained using feedback from Flan-PaLM2, the retrieved examples remained highly effective when used with Gemini 1.5 Pro, a model not involved in the training process. This indicates that the characteristics of a "good" demonstration are largely consistent across different state-of-the-art LLMs.

Conclusion

MLT-DR represents a significant step toward a universal demonstration retriever. By identifying the "output overfitting" trap and leveraging massive multilingual data augmentation, the authors provide a framework that is both robust and scalable. Future work will focus on dynamic weighting strategies to determine when output information is beneficial and addressing the remaining challenges in zero-shot language performance.

Summary updated

Abstract

This paper presents Multi-Lingual/Task Demonstration Retrieval (MLT-DR) for in-context learning with Large Language Models (LLMs).Our goal is to investigate how dense demonstration retrieval models are generalized across languages and tasks.We first convert 81 tasks into a common format, covering various languages, task types, and domains.For 8 English-based tasks among them, we use machine translation to create synthetic multi/cross-lingual tasks, by translating the examples into non-English languages to explicitly cover more than 130 languages.We then use an instruction-tuned LLM to estimate utility of demonstrations for all the tasks to train the demonstration retrieval models.In our experiments, we show an interesting counterintuitive observation; to compute embeddings of demonstrations, using both the input and ground-truth output hurts the generalization ability of the retriever on unseen tasks whose output space is quite different from those in the seen task set.We also examine that our retriever robustly works even with LLMs that we did not touch during the development of the models.
BibTeX
@misc{hashimoto2024mlt,
  author = {Kazuma Hashimoto and Arjun Akula and Karthik Raman and Michael Bendersky},
  title = {{MLT-DR: Multi-Lingual/Task Demonstration RetrievalAn Attempt towards Generalized Retriever for In-Context Learning}},
  year = {2024},
  doi = {10.18653/v1/2024.mrl-1.27},
}
Figure