Prompt Tuning Can Simply Adapt Large Language Models to Text Encoders

Kaiyan Zhao, Qiyu Wu, Zhongtao Miao, Yoshimasa TsuruokaNAACL 2025

TL;DR

This paper introduces a simple and highly efficient method called Suffix Prompt Tuning (SPT) to adapt large language models (LLMs) for sentence embedding. SPT achieves competitive performance with existing methods like LoRA fine-tuning, but with a drastic reduction in trainable parameters (less than 0.001%), demonstrating that causal attention in decoder-only LLMs is sufficient for this task.

What Problem Does This Paper Address?

Adapting massive, pre-trained Large Language Models (LLMs) for specific downstream tasks is a significant challenge. For sentence embedding, a common approach involves transforming decoder-only LLMs (like GPT) into text encoders. Traditional methods often require complex multi-step pipelines, including enabling bi-directional attention (which is unnatural for decoder-only models) and performing additional post-training tasks like Masked Next Token Prediction (MNTP). These methods, while effective, are computationally expensive due to the sheer size of LLMs. Furthermore, the necessity of bi-directional attention and these post-training steps for achieving good sentence embedding performance was questioned. This paper aims to find a simpler, more parameter-efficient way to adapt LLMs for sentence embedding, specifically by leveraging their inherent causal attention mechanism.

Methods

The core contribution is Suffix Prompt Tuning (SPT). Instead of modifying the LLM's architecture or performing extensive post-training, SPT appends a small number of trainable "soft prompt" tokens to the input sequence. These soft prompts are then optimized while keeping the rest of the LLM frozen.

Here's how it works:

  • Causal Attention: The method leverages the natural causal attention mechanism of decoder-only LLMs. This means each token can only attend to previous tokens.
  • Soft Prompts: A set of learnable continuous vectors (soft prompts) are prepended to the input text. These prompts are trained to guide the LLM towards generating sentence embeddings.
  • Parameter Efficiency: The key innovation is that only these few soft prompt tokens are trained. This drastically reduces the number of trainable parameters, often to less than 0.001% of the total model parameters, making it significantly more efficient than methods like LoRA.
  • Sentence Embedding: The output of the last soft prompt token (or an average of soft prompt tokens, depending on the configuration) is used as the sentence embedding.

The paper also explores variants like SPT-bi. (using bi-directional attention with post-training) and SPT v2 (which inserts trainable tokens into each layer, not just the input embedding layer) to further investigate the impact of these factors.

Datasets

The primary dataset used for training was the NLI dataset from Gao et al. (2021), which provides anchor sentences with one positive and one hard negative example.

For evaluation, the paper used a diverse set of benchmarks from the MTEB (Massive Text Embedding Benchmark):

  • Retrieval: QuoraRetrieval dataset.
  • Semantic Textual Similarity (STS): STS12-16, STS-B, and SICK-R datasets.
  • Clustering: Twenty Newsgroup Clustering dataset.
  • Classification: Tweet Sentiment Extraction Classification dataset.

Key Insights and Ideas

  • Causal Attention is Sufficient: The most striking insight is that decoder-only LLMs, with their inherent causal attention, can be effectively adapted for sentence embedding without needing to force bi-directional attention. This simplifies the adaptation process significantly.
  • Prompt Tuning's Efficiency: SPT demonstrates that a tiny fraction of trainable parameters (less than 0.001%) can yield performance comparable to, and sometimes even better than, full fine-tuning or LoRA-based methods that train a much larger percentage of parameters.
  • Post-training is Often Unnecessary: The paper's experiments suggest that additional post-training tasks like MNTP, often used to help LLMs adapt to bi-directional attention, do not provide significant benefits when SPT is applied with causal attention. This further streamlines the adaptation process.
  • Optimal Prompt Length Matters: While a single soft prompt token is highly efficient, performance can be further improved by increasing the prompt length up to an optimal point, after which performance may degrade.

Key Results

  • Competitive Performance: SPT achieved performance comparable to or better than strong baselines like LoRA fine-tuning and the reproduced LLM2Vec on retrieval, STS, clustering, and classification tasks.
  • Drastic Parameter Reduction: SPT reduced trainable parameters to less than 0.001%, a significant improvement over LoRA (which typically trains 0.1-1% of parameters).
  • Base Models: For OPT-125M, SPT with a prompt length of 16 achieved performance close to fully fine-tuned models, with only 0.0098% trainable parameters.
  • Larger Models (7B/8B): SPT on LLaMA2-7B and LLaMA3-8B with optimal prompt lengths consistently outperformed reproduced LLM2Vec (LoRA-bi.) models, demonstrating its effectiveness on larger architectures. For LLaMA3-8B, SPT with k=5 achieved the best scores across retrieval, STS, and classification.
  • SPT v2: Implementing SPT v2 (inserting prompts into each layer) further boosted performance, surpassing LoRA-based fine-tuning even with a similar number of trainable parameters.

Limitations and Future Work

  • Task Generalization: The authors acknowledge that while SPT is effective for sentence embedding, its generalization to other LLM adaptation tasks remains an open question.
  • Optimal Prompt Length Search: Finding the optimal soft prompt length requires an additional search step, which can vary based on the dataset and model size.
  • Multilingual Scenarios: The current work focuses on English. Exploring SPT's effectiveness in multilingual settings is suggested for future work.
  • More Trainable Parameters: While SPT is highly parameter-efficient, the paper briefly touches upon p-tuning v2 as a way to introduce more trainable parameters by extending prompts to all layers, suggesting this as a direction for further performance gains.

Summary updated

BibTeX
@inproceedings{zhao2025prompt,
  author = {Kaiyan Zhao and Qiyu Wu and Zhongtao Miao and Yoshimasa Tsuruoka},
  title = {{Prompt Tuning Can Simply Adapt Large Language Models to Text Encoders}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure