SE-Mem: Semantic-Episodic Memory for Scene-Consistent Generative View Synthesis

Changsong Lei, Yihan Li, Yu-Hui Wen, Huankun Sheng, Ming Li, Yeying Fan, Minnan Luo, Yong-Jin LiuarXiv 2026 · arXiv

TL;DR

SE-Mem is a novel framework that brings long-term scene consistency to generative view synthesis by mimicking human semantic and episodic memory. By combining structured 3D geometric cues with semantic importance scores, it ensures that video generators "remember" and faithfully reconstruct previously seen details, even after complex camera movements and viewpoint revisiting.

What Problem Does This Paper Address?

Generative View Synthesis (GVS) is the art of creating novel-view videos from a single reference image along a user-defined path. While modern diffusion models are great at hallucinating plausible new regions, they often suffer from "amnesia." When a camera moves through a scene and then returns to a previously visited spot, the generated content often drifts, loses fine-grained details, or becomes geometrically inconsistent.

Existing solutions typically use a "memory" of past frames, but they usually select these frames based purely on camera poses or geometric co-visibility. This approach has a major flaw: it treats every pixel equally. Large, redundant background regions (like a plain wall) might be over-represented in the memory, while small but semantically critical objects (like a decorative vase) are overlooked. This leads to blurry or inconsistent reconstructions of the most important parts of a scene.

Methods

The authors introduce SE-Mem (Semantic-Episodic Memory), a framework that organizes historical observations into a dual-structured system inspired by cognitive science:

  1. Episodic Memory (The "Where"): This is a structured 3D representation (a voxelized point cloud) that stores geometric cues. It tracks which parts of the 3D space have been seen and which historical frames captured them.
  2. Semantic Memory (The "What Matters"): Every point in the episodic memory is augmented with a Hierarchical Importance Score. This score is calculated using three cues:
  3. Semantic Distinctiveness: Using a SAM2 vision encoder to find regions that deviate from the "average" scene appearance.
  4. Object-Level Information Density: Measuring internal visual variation within object masks.
  5. Local Structural Details: Using image gradients (Sobel operator) to capture fine-grained edges and textures.
Figure 2. Overview of SE-Mem. Given a sequence of target camera poses {ctgt}, SE-Mem maintains a Semantic-Episodic Memory that combines 3D episodic observations with semantic importance scores. Historical frames are processed to obtain 3D point representations and importance maps, which are integrated into a voxelized memory representation. During synthesis, a two-stage retrieval strategy selects reference frames by considering target-view visibility and voxel importance. The retrieved frames and their corresponding camera conditions are then provided to a camera-conditioned video generator to synthesize novel views, while newly generated observations are incrementally incorporated into the memory.

Two-Stage Memory Retrieval

To provide the best context to the video generator, SE-Mem uses a clever two-stage retrieval strategy:

  • Primary Retrieval: Ranks frames by their overall support for the target view, weighted by semantic importance and visibility.
  • Complementary Retrieval: Specifically identifies frames that contain informative details missing from the primary selection. This ensures the generator has a comprehensive "cheat sheet" of the scene before it starts rendering.

Datasets

The researchers evaluated SE-Mem on several challenging benchmarks:

  • RealEstate10K: A large-scale dataset of monocular indoor videos with diverse camera trajectories.
  • Tanks-and-Temples: Features large-scale indoor and outdoor scenes with significant viewpoint variations.
  • DL3DV: Used to test the model's ability to generalize to entirely new types of scenes.
  • In-the-Wild Images: Qualitative testing on random internet images to prove real-world robustness.

Key Insights and Ideas

  • Semantics Over Geometry: The core "aha" moment is that visibility $\neq$ relevance. By prioritizing "informative" voxels over "visible" ones, the model avoids wasting its limited conditioning budget on redundant background pixels.
  • Voxelization for Efficiency: Instead of storing every point from every frame (which would explode memory usage), SE-Mem merges repeated observations into a voxel grid, keeping only the most reliable geometric and semantic data.
  • Plug-and-Play Compatibility: SE-Mem isn't a new video generator; it's a memory module. The authors demonstrated it could improve the performance of several different backbones, including SEVA, ViewCrafter, and MVGenMaster, without changing their underlying architectures.
Figure 3. Visualization of importance maps. Given an input scene (left), our hierarchical importance scoring module (Eq. (4)) produces an importance map (right), where brighter regions indicate higher importance values. This map guides our SE-Mem to prioritize semantically informative scene elements during both memory construction and retrieval.

Key Results

SE-Mem consistently outperformed state-of-the-art baselines, particularly in "Viewpoint Revisiting" scenarios where the camera returns to its starting point.

  • Visual Fidelity: On RealEstate10K, SE-Mem achieved a PSNR of 14.22, significantly higher than the baseline SEVA (13.20) and VMem (13.64).
  • Perceptual Quality: It showed consistent improvements in LPIPS, suggesting that the semantic-aware memory helps preserve the "look and feel" of the original scene.
  • Geometric Consistency: It reduced rotation and translation errors, proving that better memory leads to more stable camera control.
  • Efficiency: Despite the extra semantic processing, SE-Mem maintains a manageable peak GPU memory (~18.15 GB) and is significantly faster than some previous memory-augmented methods like VMem.
Qualitative comparison showing SE-Mem successfully recovering the original environment at t=200 compared to other methods.

Limitations and Future Work

  • Reconstruction Dependency: The system relies on an off-the-shelf point map estimator (CUT3R). If the initial 3D reconstruction is poor, the "indexing" of the memory becomes inaccurate, leading to the retrieval of irrelevant frames.
  • Error Propagation: While SE-Mem helps with consistency, it doesn't magically fix artifacts that were already generated in previous frames; those errors can still persist and accumulate.
  • Dynamic Scenes: The current framework assumes a static scene. Adapting this semantic-episodic memory to handle moving objects or changing lighting remains an open challenge.

Summary updated

Abstract

Generative view synthesis aims to synthesize novel-view videos from reference observations along user-defined camera trajectories. Maintaining global scene consistency throughout this process remains challenging for camera-conditioned interactive video generators. Existing approaches typically select reference frames based solely on camera poses, overlooking whether the content remains semantically relevant or even visible. To address these limitations, we draw inspiration from semantic and episodic memory in human cognition, in which semantic memory captures generalized knowledge across experiences, whereas episodic memory preserves information tied to specific past events and contexts. Motivated by the complementary roles of semantic and episodic memory, we introduce SE-Mem, a Semantic-Episodic Memory framework that organizes historical observations in a structured 3D representation and augments its elements with semantic importance scores. A two-stage memory retrieval strategy is designed to identify the most informative frames to provide contextual guidance. Taking into account both semantic significance and geometric visibility, SE-Mem preserves semantic coherence and fine-grained visual details during novel view synthesis. Extensive evaluations show that our approach improves visual quality and achieves competitive geometric consistency across multiple benchmarks.
BibTeX
@misc{lei2026se,
  author = {Changsong Lei and Yihan Li and Yu-Hui Wen and Huankun Sheng and Ming Li and Yeying Fan and Minnan Luo and Yong-Jin Liu},
  title = {{SE-Mem: Semantic-Episodic Memory for Scene-Consistent Generative View Synthesis}},
  year = {2026},
  eprint = {2611.00658},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00658},
}
Figure