When a video is too large for a neural network to process in one pass, as with 4K or 8K footage, the usual fix is to cut each frame into tiles, run the network on each tile, and stitch the outputs back together. Overlapping the tiles and blending them hides the seams within a single frame. In video, though, each frame is blended separately, so the seams shimmer from frame to frame. A separate body of work on video editing makes footage stable over time in a different way. Instead of smoothing pixels, the model predicts a few interpretable settings per frame, such as brightness, contrast, or a projection angle, and penalizes those settings for changing quickly. This proposal joins the two. For spatially varying enhancement of high-resolution video, each overlapping tile predicts a small vector of filter settings. We then solve one small least-squares problem that makes neighbouring tiles agree in the regions where they overlap and makes each tile's settings change slowly over time. The weight on the temporal penalty is set separately for each tile and frame, by how much the overlapping tiles disagree there.
The key reframing is that seam artifacts and temporal flicker share a cause. Both come from noisy estimates of a low-dimensional quantity, made from too little context. Seams are those errors seen across space, and flicker is the same errors seen across time. Overlap is normally pure overhead: a 50% overlap in both directions costs roughly 4x compute. Here it doubles as a free uncertainty signal that decides where temporal smoothing is needed. If this works, overlap can be smaller, seams and flicker are removed together, and the smoothing step costs almost nothing because it operates on a handful of numbers per tile rather than on pixels.
Core Claim
Consider patch-wise video enhancement whose output is a parametric filter, meaning a fixed differentiable image operation controlled by a few numbers. In that setting, jointly regularizing per-tile parameters over space and time should beat pixel-space overlap blending followed by temporal smoothing on two fronts. It should give lower temporal flicker and lower seam visibility at equal or better per-frame fidelity, and it should do so with a smaller overlap ratio. Using overlap disagreement as a per-tile confidence for the temporal penalty is the ingredient that should matter most. It lets the method smooth heavily where tiles are uncertain and lightly where content genuinely changes.
Where This Sits
The overlapping-patch literature recombines redundant predictions in pixel space. The simplest version averages them uniformly. Others weight them by distance from the tile centre (Zhou et al., Eggimann et al.), use spline windows (Benjdira et al.), use Gaussian windows (Edirimuni), or crop away tile borders entirely (Ho et al., Guan et al.). These supply our pixel-space baselines. None of them treats the disagreement between overlapping tiles as information. All of them operate per frame, and the research file names temporal flicker at seams as an open problem of exactly this design.
PatchVSR works on video, blending overlapping patches of a video diffusion model with spatial weight maps. UniMIE averages the noise estimates of overlapping patches at every diffusion step. Both still reconcile tiles in a high-dimensional signal space, and neither regularizes a compact per-tile parameter over time.
Continuous Patch Stitching shows that a network with no convolutional padding and correctly sized overlaps reproduces full-frame output exactly. This constrains the proposal: when an architecture can be redesigned that way, spatial seams can be eliminated rather than regularized. We therefore target settings where that redesign is not available.
The temporal side supplies the method. Harmonizer predicts six filter parameters from a downsampled frame and smooths them with an exponential moving average, a running average that gives recent frames more weight. Our method is a tiled, spatially varying generalization of this idea. The 360° projection work (Automatic Content-Aware Projection for 360deg Videos) and stereo rectification (Self Image Rectification for Uncalibrated Stereo Video) penalize changes in parameters between consecutive frames. Multi-Cali Anything penalizes parameter variance over a window of frames. Piecewise tone mapping scales its temporal weight by scene content. Boosting VOS via scale inconsistency uses disagreement between predictions at different resolutions as an uncertainty signal. Our confidence weighting borrows that idea, swapping disagreement across resolutions for disagreement across overlapping tiles. Throughout this work, temporal regularization is applied to one parameter vector per frame, or to a fixed rig of cameras. None of it regularizes a field of parameters defined on a grid of overlapping tiles.
The research file's section on closest existing work was not available to us. The novelty judgment above rests only on the papers described here.
Research Program
A lightweight backbone, of the EfficientNet-B0 type used in Harmonizer, sees each tile $i$ of frame $t$ together with a coarse thumbnail of the whole frame for global context. It predicts filter parameters $\hat\theta_{i,t}$ for brightness, contrast, saturation, colour temperature, highlights, and shadows. The final parameters $\theta_{i,t}$ come from solving
$$ \min_{\theta}\ \sum_{i,t} \|\theta_{i,t}-\hat\theta_{i,t}\|^2 \;+\; \alpha \sum_{t}\sum_{(i,j)\in\mathcal{N}} \|f(x^{ij}_t;\theta_{i,t})-f(x^{ij}_t;\theta_{j,t})\|^2 \;+\; \sum_{i,t} \lambda_{i,t}\,\|\theta_{i,t}-\theta_{i,t-1}\|^2 . $$
Here $\mathcal{N}$ is the set of pairs of tiles that overlap, $x^{ij}_t$ is the overlap region between tiles $i$ and $j$ at frame $t$, $f$ is the differentiable filter, and $\alpha$ sets how strongly overlapping tiles must agree. The temporal weight is
$$ \lambda_{i,t}=\beta\, g\big(d_{i,t}\big)\, s_t , $$
where $d_{i,t}$ is the mean disagreement, measured in pixels, between tile $i$ and its overlapping neighbours under the raw predictions $\hat\theta$, and $g$ is an increasing function so that more disagreement means more smoothing. The factor $s_t$ falls toward zero at detected scene cuts, in the spirit of the content-adaptive weighting in piecewise tone mapping.
The output image is produced by interpolating $\theta$ bilinearly between tile centres and applying $f$ at full resolution. Because the parameters vary smoothly across the frame, no pixel-space blending is needed. Within a sliding window of frames, the problem is a sparse least-squares solve over a few thousand variables at 4K, which is negligible next to running the backbone.
Camera motion is the main risk. Tiles are fixed in image coordinates, so a temporal penalty on the parameters of a fixed tile will lag behind content as it pans across. We will test a variant that first warps the previous frame's parameter field by estimated global motion before applying the penalty.
Baselines
The naive reference processes each tile independently, averages overlaps uniformly, and does no temporal processing. It shows the size of the problem.
The strongest pixel-space competitor applies Gaussian centre weighting (Edirimuni) to the predicted output tiles, then an exponential moving average over pixels and per-tile parameters, as in Harmonizer and the 360° projection work. It will share our backbone, tile grid, and tuning budget. Its smoothing constant gets the same number of validation sweeps as our $\alpha$ and $\beta$.
The strongest existing video method is CO2Net, which reports fMSE 186.72 and Temporal Loss 5.11 on HYouTube; fMSE is the mean squared error on the foreground, and Temporal Loss measures frame-to-frame flicker. Beating it means lower Temporal Loss at fMSE no worse than 186.72 under the same evaluation protocol.
A non-tiled global baseline runs Harmonizer on a downsampled frame with its moving average, predicting one parameter vector per frame. It shows whether spatially varying parameters are needed at all.
Three ablations isolate the ingredients. The first fixes $\lambda$ to a constant, removing the confidence from overlap disagreement. The second sets $\alpha=0$, so there is temporal smoothing but no spatial agreement term. The third removes the temporal term entirely. We will repeat all methods at 50%, 25%, and 12.5% overlap.
Why not just…?
Why not just smooth the stitched video in pixel space afterwards? Pixel smoothing over time blurs genuine motion and costs memory in proportion to resolution. CO2Net's own motivation for anchoring on colour mappings instead of per-pixel flow points the same way. The decisive comparison is the pixel-space baseline above, evaluated at equal fidelity. If it matches our Temporal Loss and seam metrics, the parameter-space claim fails.
Why not just use a padding-free network so seams vanish exactly, as Continuous Patch Stitching does? This is a real alternative wherever the architecture can be rebuilt. However, exact equivalence to full-frame processing only reproduces the per-frame output. It does nothing about flicker over time, and it does not carry over to backbones with attention or global pooling. We concede the spatial term is redundant for networks with exact equivalence. Our claim rests on the temporal and confidence-weighting terms, which the ablations test separately.
Why not just predict one global parameter vector from a downsampled frame, as Harmonizer does? For global colour shifts this may well win, and HYouTube may not reward tiling at all. That is why the decisive test below uses spatially varying targets. If the global baseline is competitive even there, the tiled approach is unnecessary.
Why not just run an exponential moving average on each tile's parameters independently? That is the constant-$\lambda$, $\alpha=0$ ablation. Neighbouring tiles' running averages drift apart, and the seams then flicker slowly instead of quickly. We expect the joint solve to show measurably lower seam variance over time.
Proposed Experiments
On HYouTube, at native resolution and also upsampled to stress the tiling, we report fMSE, MSE, and Temporal Loss against CO2Net's reported figures. Because HYouTube targets are largely global, this mainly checks that tiling costs nothing.
The decisive test is synthetic. We will take high-resolution source video and apply known parameter fields that vary smoothly across space and over time, for example a moving light gradient or a regional exposure ramp. The task is to recover the clean video. The research file lists no reported result for a 4K patch-wise video enhancement benchmark, so we will build this set ourselves and release it. Because the true parameters are known, we can measure parameter error directly, alongside PSNR. We will also measure seam visibility as the gradient jump across tile boundaries relative to tile interiors, and seam flicker as the temporal variance of pixels near tile boundaries relative to interior pixels.
The central prediction is threefold. At 25% overlap, our method should match or beat the pixel-space baseline at 50% overlap on both seam measures. Removing the overlap-confidence weighting should increase seam flicker on panning clips. And on clips with scene cuts and fast motion, the adaptive weight $\lambda_{i,t}$ should avoid the lag that a fixed weight produces.
If the constant-$\lambda$ ablation performs as well as the full method, the confidence idea is wrong and the contribution shrinks to a careful tiled version of Harmonizer. We will report it as such.
References
- Deep Image Debanding
- Hydra: An Accelerator for Real-Time Edge-Aware Permeability Filtering in 65nm CMOS
- Streamlined Global and Local Features Combinator (SGLC) for High Resolution Image Dehazing
- IterativePFN: True Iterative Point Cloud Filtering
- Nuclei Segmentation of Fluorescence Microscopy Images Using Three Dimensional Convolutional Neural Networks
- PreP-OCR: A Complete Pipeline for Document Image Restoration and Enhanced OCR Accuracy
- PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolution
- A diffusion model for universal medical image enhancement
- Continuous Patch Stitching for Block-Wise Image Compression
- Harmonizer: Learning to Perform White-Box Image and Video Harmonization
- Automatic Content-Aware Projection for 360deg Videos
- Self Image Rectification for Uncalibrated Stereo Video with Varying Camera Motions and Zooming Effects
- Multi-Cali Anything: Dense Feature Multi-Frame Structure-from-Motion for Large-Scale Camera Array Calibration
- Intra-and-Inter-Constraint-Based Video Enhancement Based on Piecewise Tone Mapping
- Boosting Video Object Segmentation Based on Scale Inconsistency
- Deep Video Harmonization With Color Mapping Consistency