Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation

Philip Lippmann, Jie Yang

TL;DR

ZEST (Zero-shot Embeddings via Synthetic Context) allows context-aware embedding models to adapt to new domains without ever seeing the actual target corpus. By using an LLM to generate a small, representative "proxy" corpus from just five example documents, ZEST achieves retrieval performance within 0.5% of models that have full access to the target data.

What Problem Does This Paper Address?

Standard "context-agnostic" embedding models (like BGE or GTE) represent documents as fixed vectors regardless of the surrounding corpus. While "context-aware" models (like CDE) have recently emerged to fix this—adjusting embeddings based on neighboring documents to capture domain-specific terminology and topical patterns—they have a major Achilles' heel: they require direct access to the target corpus during inference.

In the real world, this is often impossible. Privacy constraints might prevent a model from "looking" at a sensitive medical or legal database, or the sheer scale of the corpus might make real-time neighborhood lookups computationally prohibitive. The authors tackle this "zero-access" barrier, seeking a way to get the benefits of contextual adaptation without the privacy and infrastructure headaches of accessing the real data.

Methods

The authors introduce a two-phase framework:

  1. Offline Phase (Hierarchical Synthesis): Instead of accessing the real corpus, ZEST uses an LLM (like GPT-4o) to build a "synthetic proxy corpus" ($D_{synth}$).
  2. Domain Anchors: Starting with just $k=5$ exemplar documents from a similar domain, the LLM generates "domain anchors"—concise thematic seeds that capture distinct topical or stylistic facets of the target domain.
  3. Expansion: Each anchor is then expanded into dozens of full-length synthetic documents. This hierarchical approach ensures the synthetic corpus is diverse and doesn't "collapse" into a single repetitive theme.
  4. Online Phase (Inference): The system uses a frozen, pre-trained context-aware model (specifically CDE). During inference, instead of feeding the model neighbors from the real (inaccessible) corpus, they feed it the pre-computed embeddings of the synthetic proxy corpus.

Because the synthetic corpus is generated once and its embeddings are cached, the online computational cost is identical to standard context-aware retrieval, but with zero calls to the actual target database.

Datasets

The researchers evaluated ZEST using the MTEB (Massive Text Embedding Benchmark), which covers tasks like classification, clustering, and retrieval.

To simulate a realistic "zero-shot" scenario, they sourced their 5 exemplar documents ($D_{ex}$) from BEIR tasks that were domain-similar but distinct from the evaluation sets. They carefully ensured no data leakage by checking for 20-token span overlaps between the exemplars and the test sets.

Key Insights and Ideas

  • LLMs as Statistical Simulators: The core "aha moment" is that an LLM doesn't need to see the actual documents to simulate the statistical vibe of a domain. By generating text that mimics the term co-occurrence and stylistic patterns of a field (e.g., "biomedical research"), the LLM creates a "signal" that the frozen encoder can use to calibrate its embeddings.
  • Diversity is Key: The paper finds that a flat generation of synthetic documents is less effective than the hierarchical "anchor" method. Forcing the LLM to first define different sub-topics (anchors) ensures the proxy corpus covers the "breadth" of the domain, which is crucial for the encoder to understand the broader context.
  • The Power of Five: Remarkably, the "signal" from just five example documents is enough to recover nearly all the performance lost by not having access to the full corpus.

Key Results

  • Near-Parity with Full Access: On the MTEB benchmark, ZEST (using 512 synthetic documents) achieved an average score of 64.07, compared to 64.36 for the CDE model using the actual full target corpus. This means ZEST recovers 87.6% of the performance gap between context-agnostic models and full-access models.
  • Outperforming Baselines: ZEST significantly outperformed strong context-agnostic bi-encoders like GTE v1.5 (62.03) and BGE v1.5 (61.31).
  • Robustness: The method proved robust across different LLMs. While GPT-4o performed best, using an open-source model like Llama-3-70B still yielded strong results, outperforming simpler synthetic generation methods.

Limitations and Future Work

  • LLM Dependency: The quality of the synthetic context is tethered to the LLM's generative capabilities. If the LLM produces biased or overly generic text, the embedding quality may suffer.
  • Exemplar Selection: While the authors used a simple keyword-based mapping to pick exemplars, they note that the selection of these $k$ examples introduces variability. Future work could involve automating the selection of the most "informative" exemplars.
  • Cost: While the online phase is efficient, the offline phase incurs LLM API costs. The authors suggest that fine-tuning smaller, open-source LLMs specifically for high-fidelity context generation could mitigate this.
  • Bias Amplification: There is a risk that biases present in the LLM or the small exemplar set could be amplified in the synthetic corpus, requiring careful auditing for sensitive applications.

Summary updated

Abstract

Context-aware embedding methods boost retrieval accuracy by conditioning on corpus statistics (e.g., term co-occurrence and topical patterns) extracted from neighboring documents.However, this context-aware approach requires access to the target corpus or requires domain-specific finetuning, posing practical barriers in privacy-sensitive or resourceconstrained settings.We present ZEST, a zeroshot contextual adaptation framework that replaces real corpus access with a one-time offline synthesis of a compact proxy.Given only a handful of exemplar documents representative of the general target domain, we use a multi-step hierarchical procedure to generate a synthetic context corpus of several hundred documents that aims to emulate key domainspecific distributions.At inference, the frozen context-aware encoder uses this proxy corpus -without any finetuning or target corpus access -to produce domain-adapted embeddings.Across the MTEB benchmark, ZEST's zeroshot synthetic context adaptation using only five example documents performs within 0.5% of models leveraging full target corpus access -demonstrating remarkable efficacy without any retraining.ZEST thus provides a practical method for deploying high-performance, adaptable embeddings in constrained environments.
BibTeX
@misc{lippmann2025zero,
  author = {Philip Lippmann and Jie Yang},
  title = {{Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation}},
  year = {2025},
  eprint = {2506.23662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.23662},
}
Figure