Déjà View: A Shared Working-Memory Model for Visual Tasks

Mranmay Shetty, Gabriel KreimanarXiv 2026 · arXiv

TL;DR

Researchers have developed a single working-memory model that can flexibly switch between five different visual tasks—from simple matching to complex repetition detection—using a combination of dynamic neural activity and rapidly changing "synaptic" associations. Their findings reveal that while the passage of time has little effect on memory, the sheer number of intervening images significantly increases "false alarms," suggesting that interference, rather than decay, is the primary driver of working-memory errors.

What Problem Does This Paper Address?

Working memory (WM) is the brain's ability to hold and manipulate information over short periods. However, WM isn't just a passive storage bin; it’s a flexible system. Depending on the goal, you might need to remember a specific "sample" image while ignoring others, or you might need to flag every time you see a repeated image in a stream.

Most computational models of WM are specialized for a single task. Furthermore, there is an ongoing debate in neuroscience about the mechanism of WM: is it maintained through persistent neural activity (neurons firing continuously) or through "silent" synaptic plasticity (temporary changes in the strength of connections between neurons)? This paper introduces a unified model to explore how these two mechanisms contribute to flexibility across diverse visual demands.

Methods

The authors propose a model that integrates two types of temporary memory:

  1. Activity State ($h$): A dynamic vector representing current neural firing.
  2. Associations ($F$): A "fast-weight" matrix that stores temporary links between image features, mimicking short-term synaptic plasticity.

The model uses a pretrained, frozen DINOv2 encoder to turn images into feature vectors. Crucially, the model is "cued": it receives explicit signals telling it which task to perform (e.g., "Is this the sample?" or "Have I seen this before?") and what role the current image plays (e.g., "This is a distractor").

The model architecture combines activity-based memory with an associative matrix, guided by task and role cues.

The system was trained using a multi-objective loss function to handle five distinct tasks simultaneously, ensuring the same set of learned parameters could generalize across different instructions.

Datasets

  • COCO (Common Objects in Context): Used for training the model weights, calibrating response rules, and initial development.
  • Open Images: Used for a specific "fixed memory" readout experiment to ensure the model's performance wasn't just due to over-fitting specific image sets.
  • Masked Foreground Objects: For evaluation, the authors used objects from these datasets placed on neutral backgrounds to focus strictly on visual working memory of objects rather than scene context.

Key Insights and Ideas

  • Interference Over Decay: One of the most striking findings is that "blank" delays (time passing with no images) hardly affected performance. However, presenting intervening images (distractors) caused a sharp increase in false alarms. This suggests that the "clutter" of new information interferes with old memories more than the simple passage of time.
  • Task-Specific Memory Strengths: Through ablation studies (removing one memory type), the authors found that activity-based memory is superior for simple tasks like remembering a single sample. In contrast, associative memory is essential for "online familiarity" (detecting repetitions in a long stream).
  • The Cost of Multi-tasking: In Task 3 (categorizing distractors while remembering a sample), the model's ability to answer all questions correctly dropped significantly as the number of distractors increased, even though its memory of the original sample remained relatively stable.
  • History Matters: Even when controlling for how recently a target image was seen, having a longer "history" of previous novel images made the model more likely to incorrectly claim a new image was a repeat.

Key Results

The model successfully performed all five tasks using a single set of weights.

The model's performance across tasks. Note how accuracy in repetition tasks (e, f) drops as the sequence length increases, compared to the stability of sample matching across delays (a).
  • Task 1 (Sample Match): Performance remained high (~94%) even with long delays of 256 time units.
  • Task 4 & 5 (Repetition Detection): Performance was highly sensitive to sequence length. In Task 4, balanced accuracy dropped from 97.9% (4 images) to 52.1% (64 images).
  • False Alarms: In Task 4, the false-alarm rate skyrocketed from 4.2% to 91.7% as the list length grew to 64 images, highlighting the massive impact of image history on perceived familiarity.

Limitations and Future Work

While the model is highly flexible, it relies on explicit cues for task and image roles—information humans often have to infer from context. Additionally, the study focused on identical image repetitions. In the real world, we recognize objects across different angles, lighting, and distances; the authors note that testing "semantic" or "transformed" matching is a logical next step.

Future work could also explore how the model behaves when task boundaries are less clear or when it must learn to manage its own memory "reset" signals rather than receiving them as inputs.

Summary updated

Abstract

A working-memory model must remember stimuli and use its memory flexibly to perform different tasks. Here we introduce a model of working memory for images that combines dynamic activity with temporary associations between image features. We train the model on five tasks, including remembering a sample while other images are presented, categorizing distractor images while remembering the sample, and determining whether an image is new or has been presented before. The same trained model performs all tasks, guided by explicit cues about which response is required. We use this flexible memory model to ask how delay, distraction, and image history affect performance. Delays have little effect on sample-memory performance for the delays we tested, while seeing more images leads to more false alarms even when the target image appeared equally recently in all cases. On the sample-memory-plus-categorization task, the probability of correct responses on all questions decreases as more questions are asked, despite robust sample-memory performance. Finally, the model maintains stable performance across 40 brief trials of each task under the tested conditions. Together, these results demonstrate how a single working-memory system can be flexibly used to perform different tasks, and how its errors depend on both previous images and task demands.
BibTeX
@misc{shetty2026deja,
  author = {Mranmay Shetty and Gabriel Kreiman},
  title = {{Déjà View: A Shared Working-Memory Model for Visual Tasks}},
  year = {2026},
  eprint = {2611.23177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23177},
}
Figure