One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, Tao Yu

Abstract

We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets.

Introduction: The Challenge of Specialized Embeddings

Text embeddings serve as the foundational "universal glue" in modern NLP, converting sentences into dense vectors for downstream applications like semantic search, clustering, and classification. However, traditional embedding models are often highly specialized. A model optimized for semantic similarity (e.g., identifying that "How to bake a cake" and "Cake baking instructions" are related) frequently underperforms on retrieval tasks, such as matching a specific question to a relevant document.

Furthermore, these models often struggle with domain shifts, failing to generalize from news data to specialized fields like medicine or finance. Historically, the solution has been to fine-tune separate models for each specific use case. INSTRUCTOR addresses this by asking: Can we build a single embedder that follows natural language instructions to produce the appropriate representation for any task?

Methodology: Instruction-Based Omnifarious Representations

The core innovation of INSTRUCTOR is instruction-finetuning for embeddings. Unlike standard encoders that process only the input text $x$, INSTRUCTOR takes a concatenation of a task instruction $I$ and the text $x$.

Figure 1: INSTRUCTOR generates task-specific embeddings by prepending natural language instructions to the input text.

The model uses GTR (Generalizable T5-based Retrieval) as its backbone and is trained using a contrastive loss objective. This objective encourages the model to minimize the distance between "positive" pairs (e.g., a question and its answer) while maximizing the distance between "negative" pairs. Instructions follow a structured template: “Represent the [Domain] [Text Type] for [Task Objective]:”. For instance, to retrieve documents, the instruction might be: "Represent the Wikipedia question for retrieving supporting documents:".

Data: The MEDI Collection

To train a truly versatile model, the researchers curated MEDI (Multitask Embedding Data with Instructions), a massive multitask mixture comprising 330 datasets with human-annotated instructions. MEDI combines 300 datasets from Super-NaturalInstructions with 30 classic embedding benchmarks like MS MARCO and SNLI.

Evaluation was conducted on the MTEB (Massive Text Embedding Benchmark), covering 56 datasets across seven categories, alongside "Billboard" and "Prompt Retrieval" tasks. Notably, 66 of the 70 evaluation datasets were entirely unseen during the training phase, testing the model's zero-shot generalization capabilities.

Results and Discussion

INSTRUCTOR achieves state-of-the-art performance, with an average improvement of 3.4% over previous best results across the 70 evaluation datasets.

Scaling and Efficiency

One of the most significant findings is the model's efficiency. INSTRUCTOR-XL (1.5B parameters) outperformed Sent-T5-XXL (4.8B parameters), a model more than three times its size. This suggests that high-quality training data and explicit task awareness are more critical than raw parameter count.

Figure 6: INSTRUCTOR consistently outperforms the GTR baseline across various model sizes, demonstrating the effectiveness of instruction finetuning.

Instructions as a Conflict Resolver

The research highlights that training a single model on diverse data—such as both symmetric similarity and asymmetric retrieval—can actually degrade performance if instructions are omitted. Instructions provide the necessary context to resolve these conflicting objectives, allowing the model to effectively leverage large-scale, diverse data.

Robustness and Generalization

By incorporating the diverse tasks from Super-NaturalInstructions, INSTRUCTOR became highly robust to instruction phrasing. It also demonstrated exceptional domain adaptation; in "out-of-distribution" domains like Geography and Biology, INSTRUCTOR showed relative gains of up to 21.8% compared to the GTR baseline.

Limitations and Future Work

While INSTRUCTOR sets a new benchmark, there are areas for improvement. Due to hardware constraints, the authors used only four negative examples per positive pair; contrastive learning typically scales well with more negatives. Additionally, while the model is robust, it performs best when instructions adhere to the specific "Domain/Type/Objective" template. Future research could explore even more flexible natural language interfaces and scaling the method to even larger models (e.g., 4.8B+ parameters).

Summary updated

Abstract

We introduce INSTRUCTOR, a new method for computing text embeddings given task instructions: every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). Unlike encoders from prior work that are more specialized, INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. We first annotate instructions for 330 diverse tasks and train INSTRUCTOR on this multitask mixture with a contrastive loss. We evaluate INSTRUCTOR on 70 embedding evaluation tasks (66 of which are unseen during training), ranging from classification and information retrieval to semantic textual similarity and text generation evaluation. INSTRUCTOR, while having an order of magnitude fewer parameters than the previous best model, achieves state-of-the-art performance, with an average improvement of 3.4% compared to the previous best results on the 70 diverse datasets. Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets. Our model, code, and data are available at https://instructor-embedding.github.io.
BibTeX
@inproceedings{su2023one,
  author = {Hongjin Su and Weijia Shi and Jungo Kasai and Yizhong Wang and Yushi Hu and Mari Ostendorf and Wen-tau Yih and Noah A. Smith and Luke Zettlemoyer and Tao Yu},
  title = {{One Embedder, Any Task: Instruction-Finetuned Text Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=t7_OyqQm8K},
}
Figure