OpticalRAG: Pixel-Space Compression for Token-Efficient Retrieval-Augmented Generation

Senhao Liu, Yuheng Zhang, Chunyu Wei, Yueguo Chen, Xinran Zhang

TL;DR

OpticalRAG shifts document compression for Retrieval-Augmented Generation (RAG) from the 1D text token space to 2D pixel space. By rendering documents as images and using a frozen vision encoder, it distills massive contexts into as few as 16 "injected tokens," achieving state-of-the-art efficiency and accuracy on long-context benchmarks without expensive cross-modal training.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) is the standard way to give Large Language Models (LLMs) access to external knowledge. However, it faces a fundamental "token bottleneck." Feeding thousands of retrieved text tokens into an LLM is computationally expensive and slow.

Existing solutions generally fall into two camps:

  1. Token-space compression: Pruning or summarizing text. This often leads to irreversible information loss at high compression ratios because 1D text sequences are sparse.
  2. Dense-vector compression: Squeezing an entire document into a single embedding. This often collapses fine-grained details necessary for precise answering.

The authors of OpticalRAG argue that the bottleneck isn't information density itself, but the modality of compression. Humans scan 2D pages efficiently; OpticalRAG aims to give LLMs a similar "visual" shortcut.

Figure 1: Illustration of OpticalRAG. (a) Token-space methods prune or summarize the 1D text sequence, causing permanent information loss at high compression ratios. (b) Dense-vector methods squeeze each document into a single embedding, collapsing fine-grained details. (c) Our OpticalRAG renders documents into page images and encodes them into compact visual tokens via a frozen ViT. At query time, a cross-attention module distills retrieved visual tokens into a small set of injected tokens conditioned on the user query.

Methods

OpticalRAG operates in three distinct stages:

  1. Optical Encoding: Instead of tokenizing text, the system renders document chunks into page images. These images are processed by a frozen vision encoder (specifically a "DeepEncoder" combining SAM and CLIP features) into a set of visual tokens.
  2. Encoder-Bridged Retrieval (EBR): To find relevant information, the system must match a text query to these visual tokens. Rather than using expensive contrastive training (like CLIP), EBR learns a lightweight projection that maps the LLM's internal query states directly into the visual feature space. This allows for "retrieval-without-recognition"—finding the right "pixels" without needing to OCR them first.
  3. Query-Driven Distillation (QDD): After retrieving the top-$K$ visual tokens, a cross-attention module (similar to a Q-Former) distills them into a tiny set of $M$ (typically 16) "distilled tokens." Crucially, this distillation is conditioned on the user's query, ensuring the 16 tokens capture exactly what is needed to answer the question.

The LLM (Qwen2.5-7B-Instruct) and the Vision Encoder remain frozen throughout. Only the small projection and distillation modules are trained.

Figure 2: OpticalRAG framework overview. ①Optical Encoding: document chunks are rendered as page images and encoded by a frozen VLM encoder into visual tokens; each visual token is indexed individually. ②Encoder-Bridged Retrieval: the query is projected into visual space via Wq and matched to the index by inner product to retrieve top-K visual tokens. ③Query-Driven Distillation: a cross-attention module with learned compression queries distills retrieved visual tokens into a compact prefix fed into the frozen LLM.

Datasets

The model was trained in two stages:

  • Stage 1 (Reconstruction): Pretraining on Wikipedia and C4 to teach the compressor how to faithfully represent document content in a few tokens.
  • Stage 2 (Task-aware): Fine-tuning on reading comprehension datasets including SQuAD v2, DROP, and CoQA.

Evaluation was performed on three challenging long-context benchmarks: LongBench-v2, BAMBOO (choice subset), and LooGLE-v2 (choice subset).

Key Insights and Ideas

  • The 2D Advantage: Visual encoders can pack the semantic content of a full text page into roughly 100 visual tokens with minimal loss. This "pixel-space" density offers a fidelity-compression tradeoff that 1D text methods simply cannot match.
  • Query-Conditioning is Non-Negotiable: To reach extreme compression (like 16 tokens), the system must know the query during the compression phase. A static summary of a document might miss the one specific fact a user asks about; a query-driven distillation "routes" only the relevant information into the bottleneck.
  • Theoretical Sufficiency: The authors provide a theoretical proof (Theorem 5.1) showing that query-driven distillation can be near-lossless if the number of injected tokens matches the "intrinsic rank" of the answer distribution. For multiple-choice questions, this rank is naturally very low, explaining why 16 tokens perform so well.

Key Results

OpticalRAG achieves the best Compression-Aware Accuracy (CAP)—a metric that balances raw task score with token efficiency—across all tested benchmarks.

  • Efficiency: While standard RAG and textual compressors often feed thousands of tokens to the LLM, OpticalRAG uses only 16 tokens.
  • Accuracy: On LongBench-v2, it achieved a raw score of 35.79, outperforming the closed-book backbone (30.62) by 5.17 points and beating much "heavier" baselines like RECOMP and LLMLingua-v2 in efficiency-adjusted scores.
  • Consistency: As shown in the results table, OpticalRAG provides a massive boost in CAP compared to both standard RAG and other "extreme" compression methods like xRAG.
Table 1: Main results on LongBench-v2, Bamboo, and LooGLE-v2.

Limitations and Future Work

The authors identify a few key areas for improvement:

  • Fixed Budgets: A single 16-token budget might be too restrictive for complex, multi-hop queries that require evidence from many different parts of a massive corpus.
  • Task Scope: The current evaluation focuses on multiple-choice QA. Extending this to open-ended generation is a primary goal for future work.
  • Adaptive Budgets: Exploring ways to dynamically adjust the number of injected tokens based on query complexity could further optimize the performance-efficiency frontier.

Summary updated

Abstract

Retrieval-augmented generation (RAG) is bottlenecked by the token budget of large language models: feeding long documents into the context window is expensive, while aggressive textual compression discards fine-grained information that is irreversibly lost in a one-dimensional token sequence. We argue that this bottleneck is not fundamental to information density but to the choice of compression \emph{modality}. Modern visual encoders can pack the contents of a rendered text page into roughly a hundred visual tokens with little semantic loss, opening a new compression-fidelity tradeoff that purely textual methods cannot reach. We present \textbf{OpticalRAG}, a framework that renders documents as page images, encodes them with a frozen vision encoder, and distills the retrieved visual tokens into as few as $16$ injected tokens before passing them to the LLM. Two ingredients make this work without expensive cross-modal training: (i) \emph{Encoder-Bridged Retrieval}, which projects text-query states into the visual feature space of the frozen encoder, through a projection shared with the compressor, to retrieve visual tokens from a text query, with no contrastive training; and (ii) \emph{Query-Driven Distillation}, a lightweight cross-attention compressor that condenses the retrieved visual tokens into a small set of query-conditioned prefix tokens. On LongBench-v2, BAMBOO, and LooGLE-v2, OpticalRAG achieves the best compression-aware accuracy (CAP) on all three benchmarks while injecting only $16$ tokens, and improves the closed-book Qwen2.5-7B-Instruct backbone by $5.17$ points on LongBench-v2.Code is available at https://github.com/brighterr/OpticalRAG.
BibTeX
@misc{liu2026opticalrag,
  author = {Senhao Liu and Yuheng Zhang and Chunyu Wei and Yueguo Chen and Xinran Zhang},
  title = {{OpticalRAG: Pixel-Space Compression for Token-Efficient Retrieval-Augmented Generation}},
  year = {2026},
  eprint = {2611.13675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13675},
}
Figure