Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation

Yingchaojie Feng, Yiqun Sun, Yandong Sun, Minfeng Zhu, Qiang Huang, Anthony K. H. Tung, Wei Chen

Introduction

Traditional text embedding models generate static vector representations that fail to capture the multi-faceted nature of language. A single document may contain various semantic dimensions, such as sentiment, topic, or intent. While recent "instruction-following" models allow users to specify the desired facet via natural language prompts, they suffer from extreme computational inefficiency. These models require a full forward pass through a large Transformer for every text-instruction pair, leading to $O(m \times n)$ complexity that prevents real-time interaction with large-scale datasets.

The GSTransform Framework

To address these scalability challenges, we introduce Guided Space Transformation (GSTransform). This framework operates on the principle that generic embeddings already contain the necessary semantic information; the challenge lies in prioritizing the relevant dimensions for a given task.

GSTransform decouples instruction understanding from data encoding through a two-stage process:

  1. Instruction-based Label Construction: A small representative sample of the corpus (e.g., 3,000 texts) is processed by a Large Language Model (LLM). The LLM summarizes these texts according to the user's instruction and generates a task-specific taxonomy of labels.
  2. Label-guided Embedding Transformation: A lightweight linear encoder-decoder is trained on this small labeled subset. The encoder uses a contrastive loss to pull semantically similar texts (based on the new labels) together, while the decoder ensures the original semantic integrity is preserved via a reconstruction loss.
Embedding space reoriented by user intent (shape-based clustering).

By applying this learned linear transformation to pre-computed generic embeddings, GSTransform can reorient millions of vectors in seconds. This allows the same dataset to be viewed through different "lenses" almost instantaneously, such as switching from an intent-based view to an emotion-based view.

Embedding space reoriented by user emotion (color-based clustering).

Experimental Evaluation

We evaluated GSTransform across nine datasets covering clustering, semantic textual similarity (STS), and triplet alignment tasks. Using popular backbones like UAE, Mxbai, and BGE, the framework demonstrated significant improvements:

  • Superior Performance: GSTransform achieved an average score of 66.01 across all tasks, significantly outperforming the strongest baseline, InBedder-Llama2 (55.31).
  • Massive Speedups: On the Big Patent dataset, baseline models took nearly 8 hours to process a new instruction. GSTransform finished in just 87 seconds—a roughly 340x improvement.
  • Constant Latency: Unlike previous methods where wait times grow linearly with the number of documents, GSTransform’s latency remains nearly constant because the learning phase happens on a fixed-size sample regardless of the total database size.

Conclusion

GSTransform proves that "latent knowledge" in generic embeddings is sufficient for complex instruction following. By replacing heavy re-encoding with efficient linear transformations, it enables interactive, multi-faceted exploration of massive text corpora without the prohibitive cost of traditional methods. Future work may explore more complex architectures for difficult instructions, though the current linear approach provides an optimal balance of speed and accuracy.

Summary updated

BibTeX
@inproceedings{feng2025don,
  author = {Yingchaojie Feng and Yiqun Sun and Yandong Sun and Minfeng Zhu and Qiang Huang and Anthony K. H. Tung and Wei Chen},
  title = {{Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=M5QudhkueX},
}
Figure