TL;DR
SE-Mem is a novel framework that brings long-term scene consistency to generative view synthesis by mimicking human semantic and episodic memory. By combining structured 3D geometric cues with semantic importance scores, it ensures that video generators "remember" and faithfully reconstruct previously seen details, even after complex camera movements and viewpoint revisiting.
What Problem Does This Paper Address?
Generative View Synthesis (GVS) is the art of creating novel-view videos from a single reference image along a user-defined path. While modern diffusion models are great at hallucinating plausible new regions, they often suffer from "amnesia." When a camera moves through a scene and then returns to a previously visited spot, the generated content often drifts, loses fine-grained details, or becomes geometrically inconsistent.
Existing solutions typically use a "memory" of past frames, but they usually select these frames based purely on camera poses or geometric co-visibility. This approach has a major flaw: it treats every pixel equally. Large, redundant background regions (like a plain wall) might be over-represented in the memory, while small but semantically critical objects (like a decorative vase) are overlooked. This leads to blurry or inconsistent reconstructions of the most important parts of a scene.
Methods
The authors introduce SE-Mem (Semantic-Episodic Memory), a framework that organizes historical observations into a dual-structured system inspired by cognitive science:
- Episodic Memory (The "Where"): This is a structured 3D representation (a voxelized point cloud) that stores geometric cues. It tracks which parts of the 3D space have been seen and which historical frames captured them.
- Semantic Memory (The "What Matters"): Every point in the episodic memory is augmented with a Hierarchical Importance Score. This score is calculated using three cues:
- Semantic Distinctiveness: Using a SAM2 vision encoder to find regions that deviate from the "average" scene appearance.
- Object-Level Information Density: Measuring internal visual variation within object masks.
- Local Structural Details: Using image gradients (Sobel operator) to capture fine-grained edges and textures.
Two-Stage Memory Retrieval
To provide the best context to the video generator, SE-Mem uses a clever two-stage retrieval strategy:
- Primary Retrieval: Ranks frames by their overall support for the target view, weighted by semantic importance and visibility.
- Complementary Retrieval: Specifically identifies frames that contain informative details missing from the primary selection. This ensures the generator has a comprehensive "cheat sheet" of the scene before it starts rendering.
Datasets
The researchers evaluated SE-Mem on several challenging benchmarks:
- RealEstate10K: A large-scale dataset of monocular indoor videos with diverse camera trajectories.
- Tanks-and-Temples: Features large-scale indoor and outdoor scenes with significant viewpoint variations.
- DL3DV: Used to test the model's ability to generalize to entirely new types of scenes.
- In-the-Wild Images: Qualitative testing on random internet images to prove real-world robustness.
Key Insights and Ideas
- Semantics Over Geometry: The core "aha" moment is that visibility $\neq$ relevance. By prioritizing "informative" voxels over "visible" ones, the model avoids wasting its limited conditioning budget on redundant background pixels.
- Voxelization for Efficiency: Instead of storing every point from every frame (which would explode memory usage), SE-Mem merges repeated observations into a voxel grid, keeping only the most reliable geometric and semantic data.
- Plug-and-Play Compatibility: SE-Mem isn't a new video generator; it's a memory module. The authors demonstrated it could improve the performance of several different backbones, including SEVA, ViewCrafter, and MVGenMaster, without changing their underlying architectures.
Key Results
SE-Mem consistently outperformed state-of-the-art baselines, particularly in "Viewpoint Revisiting" scenarios where the camera returns to its starting point.
- Visual Fidelity: On RealEstate10K, SE-Mem achieved a PSNR of 14.22, significantly higher than the baseline SEVA (13.20) and VMem (13.64).
- Perceptual Quality: It showed consistent improvements in LPIPS, suggesting that the semantic-aware memory helps preserve the "look and feel" of the original scene.
- Geometric Consistency: It reduced rotation and translation errors, proving that better memory leads to more stable camera control.
- Efficiency: Despite the extra semantic processing, SE-Mem maintains a manageable peak GPU memory (~18.15 GB) and is significantly faster than some previous memory-augmented methods like VMem.
Limitations and Future Work
- Reconstruction Dependency: The system relies on an off-the-shelf point map estimator (CUT3R). If the initial 3D reconstruction is poor, the "indexing" of the memory becomes inaccurate, leading to the retrieval of irrelevant frames.
- Error Propagation: While SE-Mem helps with consistency, it doesn't magically fix artifacts that were already generated in previous frames; those errors can still persist and accumulate.
- Dynamic Scenes: The current framework assumes a static scene. Adapting this semantic-episodic memory to handle moving objects or changing lighting remains an open challenge.
Summary updated