StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval

Dayoon Ko, Yichuan Wang, Soyeong Jeong, Joseph E. Gonzalez, Gunhee Kim, Sewon MinarXiv 2026 · arXiv

TL;DR

StreamEmbed is a novel system for real-time visual indexing of screen history that enables "pixel-perfect" retrieval without the massive computational overhead of standard vision models. By exploiting the inherent redundancy of computer screens—where most pixels don't change between frames—it achieves up to a 3.7x speedup in encoding, making local, background indexing practical for the first time.

What Problem Does This Paper Address?

We spend a significant portion of our lives staring at screens, yet our digital "visual memory" is surprisingly poor. If you want to find a specific chart you saw last week or remember which app had a certain blue button, current tools fail you.

Existing solutions generally fall into two camps:

  1. OCR-based indexing: Fast and lightweight, but "blind" to anything that isn't text (like icons, layout, or UI state).
  2. Full visual embeddings: Highly accurate but computationally expensive, often taking over two seconds per screenshot. On a standard laptop, this would consume the GPU for nearly half of every hour just to keep up with a 5-second capture interval.

The challenge is: can we maintain the high quality of visual retrieval while making the encoding process lightweight enough to run silently in the background?

Methods

StreamEmbed treats screen history as a continuous stream rather than a series of isolated images. It introduces three core techniques to slash computation:

  • Spatial Token Pruning: Within a single screenshot, many blocks of pixels are identical to their neighbors (think of large white margins or empty desktop space). StreamEmbed identifies these and simply skips them, reducing the workload by 33–41%.
  • Temporal KV (Key-Value) Reuse: Across consecutive screenshots, most of the screen remains static. StreamEmbed reuses cached internal representations (keys and values in the Transformer layers) for unchanged blocks, recomputing only the regions that actually moved or changed (e.g., a blinking cursor or newly typed text).
  • Consistency Adapter: Pruning and reusing tokens can slightly "shift" the final embedding compared to a full-image scan. To fix this, the authors trained a tiny LoRA (Low-Rank Adaptation) module that acts as a "consistency adapter," ensuring the fast, incremental embeddings remain faithful to the original model's logic.
The StreamEmbed pipeline: from pixel-level change detection to incremental encoding.

Datasets

To evaluate the system, the authors introduced ScreenMemEval (SME), a comprehensive benchmark for screenshot retrieval.

  • Scope: 10,426 questions covering 184 hours of screen history.
  • Sources: It includes data from two human users (Windows) and one autonomous "computer-use" agent (Ubuntu).
  • Task: Unlike previous benchmarks that might only look at text, SME requires retrieving specific screenshots based on visual and temporal cues.

Key Insights and Ideas

  • Screens are "Lazier" than Video: In natural video, camera motion and lighting changes mean almost every pixel changes every frame. In contrast, 67–77% of blocks in a screen recording are either identical to a neighbor or unchanged from the previous frame. StreamEmbed proves that "pixel-level laziness" is a feature, not a bug.
  • Pixel Differences are Enough: You don't need complex motion estimation or optical flow to optimize screen encoders. Simple, raw pixel comparisons are sufficient to guide token pruning and reuse effectively.
  • The Pareto Frontier Shift: By combining these techniques, StreamEmbed doesn't just make a trade-off; it moves the entire "accuracy vs. latency" curve. It can be more accurate than OCR-based methods while running at a similar speed.
Figure 1. STREAMEMBED moves the accuracy–latency frontier. Recall@10 against indexing latency per screenshot (log scale) on an M2 Pro GPU, on the three histories of SCREENMEMEVAL. STREAMEMBED runs on Qwen3-VL-Embedding-2B at several input resolutions and on SigLIP 2. STREAMEMBED and the +LoRA baselines are fine-tuned on the training split of SME-H1, and all results are on held-out test sets (§5.3). We run baselines as released, while STREAMEMBED uses a linear patch embedding for Qwen-VL (§6.1).

Key Results

  • Speed: StreamEmbed is 2.8x to 3.7x faster than encoding full screenshots from scratch.
  • Accuracy: It maintains over 98% of the accuracy of a full-screenshot encoder and outperforms OCR-based indexing by up to 6.3 Recall@10 points at the same latency.
  • Efficiency: Using binary quantization, the system can store a simulated year of screenshots in just 0.13 GB of RAM, making it feasible to keep years of history searchable on a standard consumer laptop.
  • Redundancy: The system effectively ignores the 70.6% of blocks that repeat spatially and the 28.5% that are temporally unchanged, focusing its "attention" on the tiny 0.9% of the screen that actually changes.
Visualization of spatial and temporal redundancy in a typical screenshot.

Limitations and Future Work

While StreamEmbed is a major step forward, the authors note a few areas for growth:

  • Fixed Intervals: The current system relies on a set capture interval (e.g., every 5 seconds). Future versions could potentially "learn" when to take a screenshot based on activity levels.
  • Dynamic Content: While great for productivity apps, the speedup might diminish on highly dynamic content like full-screen video or gaming, where redundancy is lower.
  • Semantic Pruning: Currently, pruning is based on raw pixels. Future work could explore "semantic" pruning—learning to ignore UI elements that are visually different but semantically irrelevant to memory.

Summary updated

Abstract

Screen history can serve as searchable visual memory for people and agents, but unlike an offline corpus, it must be encoded and indexed as each screenshot arrives. OCR-based indexing is fast but loses visual information; visual encoding takes over two seconds. Yet screens change incrementally: most pixels repeat within and across screenshots. This raises a question: do visual encoders need to recompute their representations from scratch for every new screenshot? We present StreamEmbed, a system for incremental visual embedding that exploits this redundancy. Guided solely by pixel differences, it combines spatial token pruning, which skips blocks that repeat a neighbor, with temporal KV reuse, which reuses cached keys and values for blocks unchanged across screenshots. A small learned adapter compensates for pruning and reuse to preserve fidelity to full-screenshot embeddings. On ScreenMemEval, our benchmark of 10,426 questions over 184 hours of screen histories from two humans and a computer-use agent, StreamEmbed sets a new accuracy-latency Pareto frontier on an M2 Pro GPU: it encodes up to 3.7x faster than full-screenshot encoding without sacrificing accuracy and improves accuracy over OCR-based indexing by up to 6.3 Recall@10 points at matched latency. A memory-efficient vector index stores a simulated year of screenshots in 0.13 GB of RAM. Together, these results make real-time indexing of screen history in its native visual form practical, laying a foundation for fully local visual assistants.
BibTeX
@misc{ko2026streamembed,
  author = {Dayoon Ko and Yichuan Wang and Soyeong Jeong and Joseph E. Gonzalez and Gunhee Kim and Sewon Min},
  title = {{StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval}},
  year = {2026},
  eprint = {2611.18516},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18516},
}
Figure