Meta-Task Prompting Elicits Embedding from Large Language Models

Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, Andrew YatesOpenReview 2024 · OpenReview

Abstract

In this work, we introduce a new unsupervised embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning or task-specific engineering. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks yield competitive performance on Semantic Textual Similarity (STS) benchmarks and excel in downstream tasks, surpassing contrastive-trained models. Our findings suggest a new scaling law for embedding generation, offering a versatile, resource-efficient approach for embedding extraction across diverse sentence-centric scenarios.
BibTeX
@misc{lei2024meta,
  author = {Yibin Lei and Di Wu and Tianyi Zhou and Tao Shen and Yu Cao and Chongyang Tao and Andrew Yates},
  title = {{Meta-Task Prompting Elicits Embedding from Large Language Models}},
  year = {2024},
  url = {https://openreview.net/forum?id=GpaBHv9mby},
  howpublished = {OpenReview},
}

Loading related content...

Figure