Encoder-decoder networks for video restoration have a weak point. The skip connections, which copy high-resolution early features straight to the decoder, pass along whatever is in the current frame: blur, holes, sensor noise. Temporal methods have a matching weak point. They decide which neighboring frames to trust by measuring how similar each neighbor is to the current frame, so when the current frame is the corrupted one, the trust signal is corrupted too. We propose to use the second mechanism to repair the first, and to do it the other way around from usual. At every resolution of a video U-Net, the skip connection for frame $t$ is checked against a consensus built from aligned neighboring frames. Where the current frame's early features disagree with that consensus, the skip passes a similarity-weighted blend of neighbor features instead of the raw feature. Where they agree, the current frame's own detail passes through unchanged.
In two sentences: skip connections should not decide alone what high-frequency detail reaches the decoder, because their only evidence is the frame that may be damaged; other frames of the same video are a better judge. We gate each skip, at each scale and location, by how well the current frame agrees with its temporally aligned neighbors, and substitute neighbor evidence where it does not.
Core Claim
Two literatures each hold half of this argument. On the skip side, UCTransNet showed that a U-Net with no skips can beat one with all skips on GlaS. It fixes this with channel cross-attention driven by the decoder's own features. TSDN and FADeR showed that skips let defects bypass the bottleneck, and they attenuate them with internally predicted masks. FreeU rescales diffusion skips with fixed Fourier factors. Partial convolutions identified holes being carried through skips in inpainting. These papers motivate the gap. Every one of them filters using signals from the same single image, and none uses time.
On the temporal side, FGFA, STSN, MLFA and Adaptive Re-weighting (ARW) weight aligned neighbors by cosine similarity, but only at one deep feature level. TF-Blender and ARW's consistency term point the way around a corrupted reference: compare each frame against the group rather than against the target alone. We reuse that technique.
The new claim is specific and falsifiable. In video restoration with spatially and temporally varying corruption, raw multi-scale skips are a major path by which per-frame corruption reaches the output. A gate driven by cross-frame consensus removes that path better than an equally sized single-frame gate. It also beats temporal aggregation confined to the bottleneck. And it does both without the blurring that comes from replacing skips wholesale with temporal averages.
The closest prior work is DCP-MP. It also corrects skip features using temporal information, but it predicts entries of a precompiled facial dictionary with 3D convolutions. It is therefore restricted to faces, uses no pairwise feature similarity, and makes no keep-versus-replace decision per location. Hara et al. aggregate aligned frames by similarity before a completion network whose skips stay raw. Brehm et al. avoid the problem by removing downsampling, at 400 ms per video frame on a V100.
Research Program
The module sits on every skip of a standard video U-Net. Let $e_t^l(p)$ be the encoder feature of frame $t$ at scale $l$ and location $p$. Let $e_{j\to t}^l(p)$ be the feature of neighbor $j$ after it has been warped onto frame $t$'s grid. Let $m^l(p)$ be the median of all aligned features in the window, including frame $t$. The median is used so that no single frame, the target included, acts as the anchor. Neighbor weights and the gated skip are
$$w_j = \mathrm{softmax}_j\big(\cos(e_{j\to t}^l, m^l)/\tau\big),\qquad a^l = \sum_j w_j\, e_{j\to t}^l,$$
$$\tilde e_t^l = g^l\, e_t^l + (1-g^l)\, a^l,\qquad g^l = \sigma\big(h^l[\,e_t^l,\ a^l,\ \cos(e_t^l, m^l)\,]\big).$$
Here $\tau$ is a temperature, $a^l$ is the consensus aggregate, $h^l$ is a small convolution, $\sigma$ is the logistic function, and $\tilde e_t^l$ replaces the raw skip feature. The gate keeps genuine motion and new content where neighbors cannot vouch for it. It substitutes neighbor evidence only where the current frame is the outlier.
To control cost, alignment offsets are estimated once at quarter resolution with deformable convolution, as in STSN and TCENet. They are then rescaled to the other levels rather than recomputed per scale. Coarse-to-fine refinement follows Zhou et al., whose code (Revisiting-Temporal-Alignment, with VDB and VSR configurations) we build on.
There is also a diagnostic. We measure "leakage" by inserting a localized corruption into exactly one frame of a clip: a blur patch, an occluder, or a noise burst. We then measure how much error appears in that frame's output at that location compared with a clean-frame run. Repeating this with each scale's skip zeroed in turn attributes the leaked error to specific scales. The same diagnostic tests whether the UCTransNet paradox, where no skips beat all skips, appears in video deblurring at all. If it does not, the motivation weakens and we will report that.
Baselines
The naive reference is the same video U-Net with raw skips and temporal fusion only at the bottleneck. It is trained with identical data, schedule and parameter count. We also run its no-skip variant, to check the skip paradox in this setting.
The strongest existing method on restoration is Zhou et al.'s IAM+ARW pipeline (Revisiting Temporal Alignment). It is the strongest because it already combines careful long-range alignment with consensus-aware similarity weighting, so any gain we show cannot simply be credited to "using similarity." Its reported 37.84 dB PSNR is on Vimeo-90K-T super-resolution. For REDS no reported result found on Lacuna for Zhou et al.'s numeric PSNR, so we retrain it from the released code on REDS deblurring under our budget. For deblurring we also cite Brehm et al.'s reported 34.44 dB PSNR on REDS (High-Resolution Dual-Stage) as the external reference point. For inpainting the reference is Hara et al.'s reported 34.17 PSNR / 0.941 SSIM on DAVIS (Video Inpainting with Deformable Convolution).
All retrained models share data splits, input window length, GPU-hours and a fixed hyperparameter search of equal size. Beating the strongest baseline means a gain of at least 0.3 dB PSNR on REDS at no more than 1.3x its latency, together with a clear reduction in measured leakage. Without the leakage reduction, a PSNR gain alone would not support the mechanism.
The key ablation swaps our gate's input for a single-frame signal. One version uses a UCTransNet-style channel cross-attention gate on each skip. The other uses a learned soft mask in the style of FADeR. Both are matched in parameters. This isolates the claim that cross-frame consensus, not gating capacity, does the work. A second ablation sets $g^l=0$, which always uses the aggregate (multi-scale ARW on skips). A third uses the target frame instead of the median as anchor.
Why not just…?
Why not just gate skips with a single-frame attention module, as UCTransNet does? Because its evidence comes from the same frame that is damaged. That is the "lack of external guidance" gap noted for CCA and FNE. Still, a learned single-frame gate might infer blur from local statistics well enough. The matched single-frame ablation settles this directly. The prediction is that the gap is small on uniformly mild blur and large on one-frame localized corruption, where a single image cannot tell missing structure from flat texture.
Why not just do temporal aggregation at the bottleneck and leave skips raw? This is what ARW and FGFA-style methods do. TSDN's ablation shows how much raw skips can undo a bottleneck: adding standard skips dropped MVTec AD image AUC from 87.3% to 70.7% (TSDN). That is a different task, though, so the per-scale leakage attribution is what will show whether high-resolution skips carry the residual corruption in video.
Why not just replace every skip with a similarity-weighted temporal average (multi-scale ARW)? This is the most serious threat, because it is simpler. Our expectation is that pure averaging blurs real motion and new content and causes ghosting, the failure reported for StreamV2V under fast motion. If the $g^l=0$ ablation matches the full model on REDS4-Fast and in fast-motion clips, we will drop the gate and report multi-scale consensus aggregation as the contribution.
Why not just remove skips or rescale them like FreeU? Removing downsampling cost Brehm et al. 400 ms per video frame. Fixed scaling cannot follow corruption that changes from frame to frame. Both appear as baselines: the no-skip U-Net, and a per-scale FreeU-style constant scaling tuned on validation.
Proposed Experiments
The primary test is video deblurring on REDS, with REDS4-Fast for high motion and GoPro as secondary. Metrics are PSNR, SSIM, a temporal-consistency warp error, and latency. The second regime is video inpainting on DAVIS with irregular masks, where raw skips carry hole features (Partial convolutions). There we report PSNR and SSIM against Hara et al.'s 34.17 / 0.941.
The decisive checks are three. First, on the single-frame localized-corruption protocol, the consensus gate must cut leaked error substantially more than the matched single-frame gate. Second, the median anchor must beat the target anchor most when the target frame is the corrupted one. This directly tests the reference-quality failure that one-to-all similarity weighting suffers from. Third, the full gate must beat always-aggregate on fast-motion clips, or we simplify the method as stated above. We will also visualize $g^l$ per scale, expecting low values concentrated at the finest scales over blurred or masked regions. That pattern would show the gate is doing what the mechanism predicts rather than acting as generic extra capacity.
References
- UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-Wise Perspective with Transformer
- Two-Stream Decoder Feature Normality Estimating Network for Industrial Anomaly Detection
- Feature Attenuation of Defective Representation Can Resolve Incomplete Masking on Anomaly Detection
- FreeU: Free Lunch in Diffusion U-Net
- Image Inpainting for Irregular Holes Using Partial Convolutions
- Flow-Guided Feature Aggregation for Video Object Detection
- Object Detection in Video with Spatiotemporal Sampling Networks
- High-Resolution Dual-Stage Multi-Level Feature Aggregation for Single Image and Video Deblurring
- Revisiting Temporal Alignment for Video Restoration
- TF-Blender: Temporal Feature Blender for Video Object Detection
- Dynamic Content Prediction with Motion-aware Priors for Blind Face Video Restoration
- Video Inpainting by Frame Alignment with Deformable Convolution
- Temporal Context Enhanced Feature Aggregation for Video Object Detection
- Looking Backward: Streaming Video-to-Video Translation with Feature Banks