VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning

Yibo Wang, Yongcheng Jing, Shunyu Liu, Hao Guan, Rong-cheng Tu, Chengyu Wang, Jun Huang, Dacheng Tao

TL;DR

VTC-R1 introduces "optical memory" for long-context reasoning by rendering lengthy textual "Chain of Thought" traces into compact images. This approach achieves a 3.4× token compression and a 2.7× end-to-end inference speedup while actually improving accuracy on complex mathematical benchmarks compared to standard text-only models.

What Problem Does This Paper Address?

The recent "reasoning revolution" (led by models like OpenAI o1 and DeepSeek-R1) relies on generating massive internal chains of thought (CoT) to solve complex problems. However, this creates a massive efficiency bottleneck. Because the computational complexity of Transformers grows quadratically with sequence length, these long reasoning traces lead to high memory costs and sluggish inference speeds.

Existing solutions usually involve:

  1. Pruning/Summarization: Discarding "unimportant" tokens, which risks losing fine-grained logical details (like a single minus sign that changes an entire math proof).
  2. External Models: Using a second, smaller model to compress the text, which adds architectural complexity and latency.

The authors ask: How can we achieve efficient reasoning while preserving every fine-grained detail without relying on external models?

Methods

The authors propose Vision-Text Compression (VTC). Instead of keeping thousands of text tokens in the active context window, the model renders its intermediate reasoning steps into a compact image.

The VTC-R1 Loop: 1. Iterative Generation: The model generates a segment of reasoning (e.g., the first 4k tokens). 2. Lightweight Rendering: This text is rendered into a PNG image using a standard rendering pipeline (adjusting font, size, and layout). 3. Optical Memory: The image is fed back into the Vision-Language Model (VLM). The model "sees" its previous work as a visual input rather than reading it as text. 4. Continuation: The model continues reasoning, conditioned on the question and the "optical memory" of its previous steps.

This design is model-free (it doesn't need a separate compression model) and lightweight (rendering takes ~0.12s). By using vision tokens to represent text, the model can "summarize" the state of a problem in a fraction of the sequence length required by raw text.

Datasets

  • Training: The authors created OpenR1-Math-Inf, a 61K subset of the OpenR1-Math-220K corpus. They partitioned these solutions into segments to create 106K training instances and 105K rendered images.
  • Evaluation: The model was tested on five rigorous benchmarks:
  • GSM8K (Grade school math)
    • MATH500 (High school competitions)
    • AIME25 & AMC23 (Advanced math olympiads)
    • GPQA-Diamond (Graduate-level science questions, used for out-of-distribution testing).

Key Insights and Ideas

  • Vision as a High-Density Code: Text is sparse; vision is dense. A few hundred vision tokens can represent the semantic content of thousands of text tokens. This is especially true for math, where layout and symbolic structure are naturally "visual."
  • Training is Faster, Too: Because VTC-R1 processes shorter sequences (thanks to image compression), the total training time was reduced by ~48% compared to standard supervised fine-tuning (SFT) on the same data.
  • Overcoming Context Limits: Standard models are often limited by the context window they were trained on (e.g., 8k tokens). VTC-R1 can effectively "extend" its memory indefinitely by stacking images, allowing it to solve problems that require reasoning traces far longer than its training window.
  • The "Aha" Moment on GPQA: The most significant accuracy jump occurred on the hardest, out-of-distribution benchmark (GPQA-Diamond). This suggests that the visual representation helps the model maintain global coherence better than a massive, fragmented text string.

Key Results

  • Efficiency: Achieved a 3.4× compression ratio (181M text tokens reduced to 54M vision tokens).
  • Latency: Delivered a 2.7× speedup in end-to-end inference.
  • Accuracy Gains:
    • On MATH500, VTC-R1 (using Qwen3-VL-8B) improved accuracy from 85.4% to 90.0%.
  • On GPQA-Diamond, it saw a massive jump from 37.4% to 48.5%.
  • Ablation Proof: When the authors removed the image inputs during inference, accuracy plummeted (e.g., a 25.4% drop on GPQA), proving the model is actively "reading" its optical memory to solve the problems.

Limitations and Future Work

  • VLM Requirement: This method requires a Vision-Language Model with strong OCR and document-understanding capabilities. It cannot be applied to "text-only" LLMs without a vision encoder.
  • Fixed Rendering: The current approach uses a specific rendering style. The authors suggest that future work could explore "learned rendering" or more optimized visual layouts to further increase density.
  • Domain Specificity: While highly effective for math and science (which are symbol-heavy), the paper does not extensively explore how well this "optical memory" works for purely narrative or creative long-form writing.

Summary updated

Abstract

Long-context reasoning has significantly empowered large language models (LLMs) to tackle complex tasks, yet it introduces severe efficiency bottlenecks due to the computational complexity. Existing efficient approaches often rely on complex additional training or external models for compression, which limits scalability and discards critical fine-grained information. In this paper, we propose VTC-R1, a new efficient reasoning paradigm that integrates vision-text compression into the reasoning process. Instead of processing lengthy textual traces, VTC-R1 renders intermediate reasoning segments into compact images, which are iteratively fed back into vision-language models as "optical memory." We construct a training dataset based on OpenR1-Math-220K achieving 3.4x token compression and fine-tune representative VLMs-Glyph and Qwen3-VL. Extensive experiments on benchmarks such as MATH500, AIME25, AMC23 and GPQA-D demonstrate that VTC-R1 consistently outperforms standard long-context reasoning. Furthermore, our approach significantly improves inference efficiency, achieving 2.7x speedup in end-to-end latency, highlighting its potential as a scalable solution for reasoning-intensive applications. Our code is available at https://github.com/w-yibo/VTC-R1.
BibTeX
@misc{wang2026vtc,
  author = {Yibo Wang and Yongcheng Jing and Shunyu Liu and Hao Guan and Rong-cheng Tu and Chengyu Wang and Jun Huang and Dacheng Tao},
  title = {{VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning}},
  year = {2026},
  eprint = {2601.22069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.22069},
}
Figure