Text or Pixels? Evaluating Efficiency and Understanding of LLMs with Visual Text Inputs

Yanhong Li, Zixuan Lan, Jiawei ZhouEMNLP 2025

TL;DR

This paper proposes a novel method to compress long text inputs for Large Language Models (LLMs) by rendering them as images. This "text-as-image" approach significantly reduces the number of tokens processed by the LLM decoder, leading to substantial efficiency gains without sacrificing performance on retrieval and summarization tasks.

What Problem Does This Paper Address?

Large Language Models (LLMs) are computationally expensive, especially when processing long texts. The Transformer architecture's self-attention mechanism has a quadratic complexity with respect to input length, making inference on lengthy documents, dialogues, or complex reasoning tasks prohibitively costly. While LLMs are increasingly capable of handling longer contexts, reducing the token count of inputs without losing crucial information is a critical challenge for improving efficiency, throughput, and cost-effectiveness.

Methods

The core idea is to leverage the multimodal capabilities of LLMs (specifically, their ability to "read" text from images) to compress textual information. The authors developed a pipeline called ConTexImage that takes long text inputs and renders them into a single image. This image is then fed to a multimodal LLM (MLLM) along with the original query. The MLLM's vision encoder processes the image, converting it into a fixed, small number of visual tokens. This significantly reduces the number of tokens that the LLM's decoder needs to process, effectively compressing the input. The ConTexImage pipeline involves:

  1. Preprocessing: Normalizing text and escaping special LaTeX characters.
  2. LaTeX Rendering: Embedding the text into a LaTeX document and compiling it to a PDF.
  3. Adaptive Font Optimization: Adjusting font size to ensure optimal readability and a target fill ratio within the image.
  4. Rasterization: Converting the PDF into a rasterized image.

This approach is "off-the-shelf," meaning it doesn't require fine-tuning the MLLM itself.

Datasets

The paper evaluates its method on two distinct benchmarks:

  • RULER S-NIAH (Single Needle-in-a-Haystack): A long-context retrieval task designed to test the model's ability to retain specific information within a long passage.
  • CNN/DailyMail: A document summarization task, used to assess performance on a genuine compression task where all input sentences contribute to the final output.

The authors also mention experiments on the BABILONG 1k Benchmark, which is designed to test the limits of long-context reasoning by extending classical bAbI tasks to longer sequences.

Key Insights and Ideas

  • Visual Text as Implicit Compression: Off-the-shelf multimodal LLMs inherently possess a form of token compression by processing images. This paper demonstrates that by rendering text as an image, we can explicitly leverage this capability for significant input compression.
  • Near-Optimal Token Allocation: The research suggests that a near-equal balance between visual tokens (from the image) and textual tokens (for the query) is a practical and effective operating point for multimodal reasoning, achieving high compression without performance degradation.
  • Model Scale Matters: Larger multimodal LLMs (like Qwen2.5-VL-72B) are more robust to text-as-image compression and can tolerate higher compression ratios compared to smaller models (like Qwen2.5-VL-7B), which show steeper performance drops.
  • Orthogonal to Other Compression Methods: The text-as-image approach is complementary to existing token-level pruning techniques, suggesting potential for combining methods to achieve even greater compression.

Key Results

  • Significant Token Savings: The text-as-image method achieved substantial token savings, often cutting decoder token usage by nearly half (38-58% reduction on RULER).
  • Performance Preservation: On the RULER task, GPT-4.1-mini and Qwen2.5-VL-72B sustained 97-99% accuracy with up to 58% fewer decoder tokens. On CNN/DailyMail summarization, the method outperformed two token-pruning baselines at similar compression rates.
  • Latency Improvements: For larger models like Qwen2.5-VL, the reduced sequence length outweighed the vision processing overhead, leading to 25-45% faster inference times. For GPT-4.1-mini, the overhead was modest (<1.5s).
  • Text-Token Tolerance: The maximum number of text tokens that could be preserved without significant accuracy loss (text-token tolerance) showed a strong positive correlation with the visual token budget.

Limitations and Future Work

  • Extremely Large Contexts: The current study focused on short-to-medium context scenarios. Evaluating the impact of text-as-image prompting on extremely large contexts (tens of thousands of tokens or more) is an open question, potentially requiring chunking or retrieval strategies.
  • Domain and Task Generalization: The experiments were conducted on retrieval and summarization tasks. Further validation is needed for other domains (e.g., medical, legal) and tasks (e.g., coding, translation).
  • Combining Compression Techniques: The authors suggest future work could explore combining token-level pruning before rendering text into an image to achieve even greater compression.
  • Other Modalities: Expanding the approach to other domains where most prompt tokens are critical and difficult to prune at the token level is also a promising direction.

Summary updated

BibTeX
@inproceedings{li2025text,
  author = {Yanhong Li and Zixuan Lan and Jiawei Zhou},
  title = {{Text or Pixels? Evaluating Efficiency and Understanding of LLMs with Visual Text Inputs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure