ReAccess: Look Once, Stay Focused with Efficient Test Time Memory

Tianhao Chen, Yilong Fan, Yuheng Wu, Xiaobin Hu, Dongman LeearXiv 2026 · arXiv

TL;DR

ReAccess is a lightweight framework that solves the "visual forgetting" problem in Vision-Language Models (VLMs) by using a compact, question-conditioned associative memory. By "looking once" during a single visual prefill and progressively retrieving evidence to "steer" the model's internal representations, it achieves state-of-the-art fine-grained understanding with only a 6% latency overhead—making it up to 33x faster than existing reasoning-intensive approaches.

What Problem Does This Paper Address?

Modern VLMs are impressive, but they suffer from a frustrating paradox: they often "see" the evidence needed to answer a question during their initial image scan, yet fail to "use" it during the final reasoning step.

Current solutions generally fall into two camps:

  1. Visual Methods: These involve re-processing the image (e.g., zooming into crops). While effective, they are computationally expensive and can cause the model to lose the "big picture" context.
  2. Language Methods: These use Chain-of-Thought (CoT) or intermediate descriptions. However, the authors find that as a model generates more text, its attention shifts away from the image and toward its own words, a phenomenon they call evidence attention decay.

The core challenge is bridging the gap between the rich visual information encoded by the model and its effective utilization during reasoning without blowing up the computational budget.

Analysis of how visual evidence influence decays as reasoning text accumulates.

Methods

The authors propose ReAccess, which treats visual evidence not as a static input, but as an active memory that evolves with the conversation.

1. Memory Construction (The "Write")

During the initial image prefill, ReAccess builds a compact $d \times d$ associative memory matrix for each Transformer layer. Crucially, this construction is question-conditioned: it uses the question's embedding to weight which visual features are most important to store. This ensures that fine-grained details relevant to the specific query are prioritized.

2. Memory Utility (The "Read" and "Steer")

As the model generates its answer token-by-token, it performs Progressive Evidence Recall. Instead of a single lookup, it uses two sequential reads: the first retrieves broad context, which then informs a second, more focused search for local details.

The retrieved evidence is then used for Latent Steering. Rather than appending text, ReAccess directly modifies the model's internal attention queries and hidden states. This keeps the visual evidence "top of mind" for the model throughout the entire generation process.

Overview of the ReAccess framework, showing memory construction during prefill and steering during generation.

3. Training Recipe

ReAccess is trained using a "Self-Guided Evidence Reconstruction" objective. A frozen VLM acts as a teacher (seeing high-res crops) to generate descriptions, while the ReAccess-augmented student tries to reconstruct those descriptions using only its internal memory and the original full image. This forces the memory to capture the "missing" fine-grained details.

Datasets

The model was trained on VisCoT and evaluated across a massive battery of 15 benchmarks, including:

  • General Understanding: GQA, MMBench, MMStar, MMMU.
  • Perception & OCR: TextVQA, DocVQA, ChartQA, OCRBench.
  • Fine-Grained/High-Res: HR-Bench (4K/8K), V*, and CV-Bench.

Key Insights and Ideas

  • The Utilization Gap: The authors prove that task-relevant information is often present in the model's hidden states but is ignored. They show that low-dimensional PCA projections of internal states can recover visual details better than the full states themselves, suggesting significant redundancy that a compact memory can exploit.
  • Attention Competition: Figure 1(f) reveals that generated text "competes" with visual input for the model's attention. ReAccess bypasses this competition by injecting evidence directly into the latent space.
  • Global-Local Synergy: Isolated crops actually reduce accuracy compared to full images because they lack context. ReAccess succeeds by preserving the global context in the original pathway while providing "shortcuts" to local evidence via memory.

Key Results

  • Broad Generalization: ReAccess improved performance across eight different VLM backbones (including Qwen2.5-VL, LLaVA-NeXT, and InternVL3.5), with relative gains of up to 16.6%.
  • Fine-Grained Mastery: On the challenging HR-Bench 8K, ReAccess outperformed the base Qwen2.5-VL-7B by nearly 5 percentage points, rivaling multi-turn methods that are far slower.
  • Extreme Efficiency: ReAccess adds only ~58M trainable parameters (0.7% of a 7B model). It incurs only a 6% latency overhead, whereas competing "active reasoning" methods like Thyme-RL or DeepEyes are 10x to 33x slower.
Inference latency comparison showing ReAccess maintains near-original speeds while significantly outperforming baselines.

Limitations and Future Work

  • Training Requirement: Unlike purely training-free visual prompting, ReAccess requires a lightweight training phase (though only one epoch on a standard dataset).
  • Base Model Dependency: If the base vision encoder completely fails to capture a feature (e.g., due to extremely low resolution), the memory cannot "hallucinate" it back into existence.
  • Future Directions: The authors suggest extending this associative memory approach to video understanding and long-context multimodal dialogues where "visual forgetting" is even more pronounced.

Summary updated

Abstract

Despite recent advances, vision-language models (VLMs) still struggle with fine-grained visual understanding, often failing to effectively utilize critical evidence already encoded in their visual representations. Existing approaches address this limitation through repeated visual processing or explicit reasoning, introducing substantial computational overhead. In this paper, we propose ReAccess, a lightweight framework that preserves and reuses question-relevant visual evidence through compact associative memory. Constructed during a single visual prefill, the memory supports progressive evidence retrieval that steers both attention queries and hidden representations, enabling effective visual grounding without repeated image encoding or explicit reasoning traces. Extensive experiments demonstrate consistent improvements in both general visual understanding and fine-grained perception, while achieving substantially faster inference than existing reasoning-intensive approaches, demonstrating a promising path to efficient multimodal reasoning.
BibTeX
@misc{chen2026reaccess,
  author = {Tianhao Chen and Yilong Fan and Yuheng Wu and Xiaobin Hu and Dongman Lee},
  title = {{ReAccess: Look Once, Stay Focused with Efficient Test Time Memory}},
  year = {2026},
  eprint = {2611.20936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20936},
}
Figure