When a network sharpens a low-resolution video, it faces two costs at once. The first is compute: models that process space and time jointly with 3D convolutions (filters that slide over height, width and frames together) are expensive. The second is a trade-off in what the output looks like. A network trained to minimize per-pixel error averages over all plausible high-resolution answers, so the output comes out blurry. A network trained to produce realistic texture invents plausible detail instead, which looks fine on grass but produces unsettling distortions on faces. Current video restorers spend the same compute and apply the same loss everywhere in the frame. Human memory does not work that way. Viewers recognize a scene later mostly by one or two "anchor" regions, usually a person or a distinctive object, and largely ignore uniform backgrounds.
We propose to use a predicted memorability map to route both budgets. The map estimates, for each location in the frame, how likely that region is to trigger later recognition. Regions the map marks as memorable get expensive spatiotemporal processing and a loss that favors fidelity over invented detail. Everything else gets a cheap frame-wise path and a texture-synthesis loss, where invented detail is harmless. The claim we test is concrete. At a fixed compute budget, a memorability-routed restorer should preserve human recognition memory of the original clip better than uniform restorers. It should also do this better than the same routing driven by ordinary saliency or object masks. We treat that last comparison as the result that decides whether memorability adds anything.
Why this is open, and what already exists
Three existing results motivate the proposal. Understanding and Predicting Image Memorability at a Large Scale turned a memorability network into a fully convolutional map and blurred selected regions by hand. Doing so changed how memorable the images were, so detail in the right places matters for memory. That work used blur only as a diagnostic after the fact. It never asked what a restoration objective should do with the information. EnhanceNet explains the blur-versus-hallucination trade-off. Per-pixel losses regress to the mean, while its perceptual, texture and adversarial losses won a 91.0% user preference over PSNR-optimized methods. It also reports that invented detail on faces pushes viewers back toward blurry outputs, and that frame-by-frame application flickers. Continuous Space-Time Video Super-Resolution with 3D Fourier Fields, called V³, represents video as learned 3D sinusoids. This avoids dense 3D convolutions and optical flow, and V³ gains over 2 dB PSNR on Adobe240 relative to earlier implicit representations, at 1.27s versus 3.03s per patch. Its authors state that it still softens at large scale factors under reconstruction losses, and they leave generative losses to future work. None of these three papers allocates either compute or loss type non-uniformly.
The memorability papers supply the routing signal and constrain its design. Defining Image Memorability using the Visual Memory Schema introduced the Visual Memory Schema (VMS): a 2D probability map of the regions people say triggered their recognition. Human agreement on these maps is about 0.67. Predicting Visual Memory Schemas with Variational Autoencoders predicts 224×224 VMS maps at a 2D Pearson correlation of 0.57. That paper also carries a warning we take seriously. Its maps correlate only 0.147 with a global memorability score, and accuracy collapses in large open scenes. Seeing Eye to AI shows that attention in a video memorability transformer favors objects over background, and that it aligns with human gaze (AUC-P 82.91 on Memento10k). That finding cuts both ways. It supports routing by memorability, but it also suggests that memorability may reduce to "objectness," so we design the experiments to catch that.
On the efficiency side, factorizing 3D filters into 2D spatial and 1D temporal parts (Factorized Spatio-Temporal Convolutional Networks) and expanding a small network along cheap axes (X3D) both cut compute uniformly across pixels. We reuse factorized blocks for the expensive path. Our change is to apply them only where the map says they matter.
Core Claim
Restoration detail has uneven value to viewers. The value is concentrated where memory is anchored. A restorer that concentrates spatiotemporal compute and fidelity-oriented supervision on those regions should do two things at matched FLOPs (floating-point operations). It should raise recognition memory of the restored clip toward that of the original. It should also avoid both blurring the anchors and inventing false detail on them. The comparison against object masks and saliency maps at matched coverage decides whether "memorability" is the right routing signal or just a proxy for "things."
Research Program
We build a two-path restorer for 4× spatial super-resolution of short clips.
The cheap path runs everywhere. It is a 2D per-frame convolutional network with zero-cost temporal channel shifting, the idea explored in Temporal Shift Reinforcement Learning.
The expensive path is a stack of factorized spatiotemporal blocks (2D spatial followed by 1D temporal). It runs only on space-time tubes selected by a routing map $M(x,y,t)\in[0,1]$, using sparse gather and scatter. We sweep the fraction of voxels covered, $k$, from 5% to 40%. We produce $M$ with a VMS predictor retrained on VISCHEMA, applied per frame and smoothed along tracks so that tubes stay coherent across frames.
The training loss splits by region:
$$\mathcal{L} = \sum_{x,y,t} M\,\big(\lambda_f \mathcal{L}_1 + \mathcal{L}_P\big) + (1-M)\,\big(\mathcal{L}_P + \mathcal{L}_T + \mathcal{L}_A\big) + \lambda_t \mathcal{L}_{\text{temp}}$$
Here $\mathcal{L}_1$ is per-pixel absolute error, and $\mathcal{L}_P$, $\mathcal{L}_T$, $\mathcal{L}_A$ are EnhanceNet's perceptual, Gram-texture and adversarial terms. $\mathcal{L}_{\text{temp}}$ penalizes frame-to-frame change in the residual detail, to suppress flicker. The weights $\lambda_f$ and $\lambda_t$ are tuned on a validation split. In anchor regions, the adversarial term is deliberately absent and the fidelity term dominates. This follows directly from EnhanceNet's finding about faces.
No video VMS data exists; the file confirms that no spatiotemporal schema dataset has been established. We therefore collect a small validation set rather than a training set. We will take about 300 Memento10k clips and adapt the VISCHEMA protocol. Participants view clips in a continuous recognition task. After each recognized repeat, they mark one to three regions on a keyframe grid. We aggregate these marks into maps. This set serves two purposes: it checks whether the per-frame image predictor transfers to video, and it provides an oracle routing map for the upper-bound ablation.
Baselines
The naive reference is bicubic upsampling, and next to it a uniform per-pixel ($\mathcal{L}_1$) network. That network uses the cheap path everywhere, widened until its FLOPs match the routed model's. It defines the "blurry but faithful" corner of the trade-off.
The strongest existing method on standard restoration quality is V³ (Continuous Space-Time Video Super-Resolution with 3D Fourier Fields). It reports a gain of over 2 dB PSNR against prior implicit representations on Adobe240. The file gives no absolute PSNR value, so we will reproduce V³ from its released configuration and report our own measured number rather than cite one. We run it spatial-only at 4×, on the same training clips and the same degradation. We also run the variant its authors propose, V³ with a uniform adversarial and perceptual loss, since that is the most direct competitor to our loss change.
The strongest perceptual competitor is a uniform EnhanceNet-style loss (EnhanceNet). We apply it to the same two-path architecture with a uniform map ($M\equiv$ const), plus the same temporal term, so that only the routing differs.
The ablations that isolate the key ingredient keep the architecture, coverage $k$, FLOPs and tuning budget fixed. They change only the source of $M$: random tubes, a bottom-up saliency map, a class-agnostic "things" segmentation mask, predicted memorability, and the human video-VMS oracle on the annotated subset. We also separate the two routed components. One variant routes compute only, with a uniform loss. The other routes the loss only, with uniform compute. Every model gets the same number of tuning trials.
Beating the baselines means the following. On human recognition memory, the routed model must beat the uniform EnhanceNet-style model and V³, with confidence intervals that exclude zero. It must also beat the things-mask variant. On standard metrics, its PSNR must not fall below the uniform perceptual model's.
Why not just…?
Why not just route with saliency or object masks, which need no memory experiments? This is the real threat. The evidence favoring memorability is suggestive only: Spatial Mapping of Visual Memorability and What Makes an Object Memorable? argue that memorability differs from saliency. Memorability-trained attention also favors "things" (Seeing Eye to AI), and that is exactly what an object mask gives for free. The matched-coverage ablation settles the question. If the things mask ties predicted memorability on the human memory test, we will report that memorability adds nothing beyond objectness. The paper then becomes a claim about object-routed restoration, and the routing result still holds. Memorability can only win where the two signals disagree: scenes with several objects, of which only one is remembered. We will analyze that stratum separately.
Why not just use a uniform perceptual and adversarial loss with more capacity? EnhanceNet itself documents failures on faces and flicker on video. Anchors are disproportionately faces and people, so uniform invented detail should hurt exactly where memory is formed. The matched-FLOPs uniform baseline measures this directly. Any gain it shows over routing would count against our claim.
Why not just add a generative loss to V³, as its authors suggest? That fixes softness but reintroduces invented detail on anchors. It also leaves the uniform basis budget untouched. We run this variant. If it matches our model on memory preservation at lower cost, the routing idea is unnecessary for continuous representations, and we would say so.
Why not just use an efficient uniform network in the style of X3D? A uniform network might reach the same quality at the same FLOPs without routing. The loss-only routed variant with uniform compute answers this. If that variant matches the full model, compute routing contributes nothing and only the loss split matters.
Proposed Experiments
Training uses Memento10k clips degraded by 4× downsampling with mild compression. Standard evaluation runs on Adobe240 at 4× spatial scale. The file reports no prior restoration results on Memento10k itself; no reported result found on Lacuna for super-resolution on Memento10k. We measure PSNR, a learned perceptual distance, and a temporal flicker score. We report each separately inside and outside human-marked anchor regions on the annotated subset. The decisive prediction is that anchor-region fidelity rises while background realism holds.
The decisive check is a human memory experiment. We run a Memento10k-style continuous recognition game in which each target clip appears as the original, the routed restoration, or one of the baselines, with lags spanning the dataset's 9 to 200 interleaved clips. The primary metric is recognition hit rate relative to the original clip. The secondary metric is the false-alarm rate on lure clips that share a category with the target. It tests whether invented anchor detail breeds false recognition, a question motivated by the prototype-driven false schemas in Predicting Visual Memory Schemas with Variational Autoencoders. Human hit rates on Memento10k are consistent across observers at a Spearman correlation of 0.730 (Multimodal Memorability). That ceiling tells us how many participants per clip we need.
As a cheaper proxy, we compare predicted memorability of restored versus original clips. The predictor is a CLIP-plus-transformer model that is never used for routing, so the proxy is not circular. We report agreement between the proxy and the human study, and only then rely on the proxy for the larger sweeps over coverage $k$.
We also report two failure analyses. The first covers large open scenes, where VMS prediction is known to collapse. The second checks whether the frame-wise map transfers to video, measured as 2D correlation against our collected video maps and compared with the 0.57 image-level figure.
References
- Understanding and Predicting Image Memorability at a Large Scale
- EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
- Continuous Space-Time Video Super-Resolution with 3D Fourier Fields
- Defining Image Memorability using the Visual Memory Schema
- Predicting Visual Memory Schemas with Variational Autoencoders
- Seeing Eye to AI: Comparing Human Gaze and Model Attention in Video Memorability
- Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks
- X3D: Expanding Architectures for Efficient Video Recognition
- Temporal shift reinforcement learning
- What Makes an Object Memorable?
- Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability