Causal Space-Time Coupling for Streaming Invertible Video Rescaling

Invertible networks can shrink a video for transmission and later restore it. An invertible network is a single model that runs exactly forward and backward with the same weights. The downscaler packs the detail it removes into a hidden code, and the upscaler unpacks it. Today these systems handle time poorly. Frame-by-frame models flicker. The models that do use time either bolt a non-invertible recurrent network onto the upscaler, or wait for a whole group of frames before doing anything. We propose to put temporal modeling inside the invertible transform itself, in factorized form. The network alternates purely spatial coupling steps with temporal coupling steps. Each temporal step conditions on past frames only, through cheap one-dimensional convolutions along the time axis. The result should be an exactly invertible video rescaler that streams frame by frame, flickers less, and needs far fewer parameters than group-of-frames models.

The bet is that the two directions fit together naturally. A coupling layer splits features into two halves and transforms one half using a function of the other. That function never needs to be invertible, so it can be any network, including a factorized "(2+1)D" convolution: a 2D spatial filter followed by a separate 1D temporal filter. Making that temporal filter causal (looking only backward in time) preserves exact invertibility and removes the buffering delay that grouped-frame methods impose.

Core Claim

Flicker and latency in invertible video rescaling come from where temporal information enters the model, not from a lack of capacity. In frame-wise invertible rescaling, as in Invertible Image Rescaling (IRN), the per-frame high-frequency code is drawn independently from a Gaussian. 3R-INN applies such a model frame by frame and reports visible grain flicker. It names temporal modeling as the missing piece.

Video Rescaling Networks With Joint Optimization Strategies offers two ways to add time, and each has a cost:

  • LSTM-VRN adds a bidirectional LSTM to the upscaler only. The downscaler stays blind to motion, and the model is no longer a single bijection.
  • MIMO-VRN processes a group of frames jointly and reaches 33.79 dB PSNR on Vid4 at 4×. It pays with 19M parameters and the latency of buffering the whole group.

We claim that a bijection built from alternating spatial and causal-temporal coupling layers can do better than both on the right axis. It should roughly match MIMO-VRN's restoration quality at a fraction of its parameters and with no look-ahead. It should also reduce temporal flicker below both frame-wise IRN and MIMO-VRN.

Concretely, a temporal coupling step acts on a frame-indexed feature stack $x_{1:T}$ split by channel into $a$ and $b$:

$$y^{a}_t = a_t \odot \exp\big(s(b_{t-k+1:t})\big) + m(b_{t-k+1:t}), \qquad y^{b}_t = b_t,$$

Here $s$ and $m$ are causal (2+1)D convolution stacks over the last $k$ frames. Inversion only requires dividing by the scale and subtracting the shift. The upscaler can therefore invert frame $t$ as soon as frames $t-k+1$ through $t$ have arrived.

A second, smaller change replaces the independent Gaussian code with a causal conditional prior. The mean and scale of frame $t$'s code are predicted from the low-resolution frames up to $t$. LF-VSN argued for a similar move in steganography: its Redundancy Prediction Module predicts residual detail from the carrier video instead of assuming noise. Our version differs in being causal and in serving temporal coherence rather than payload capacity.

Research Program

Base design. We build on IRN: a Haar wavelet downsampling step followed by coupling blocks.

Insertion point. Every other block becomes temporal. Its conditioning network follows the factorized residual block of Fast Spatio-Temporal Residual Network (FSTRN), a spatial 1×3×3 convolution followed by a temporal 3×1×1 convolution. We pad only on the past side. FSTRN reports a 55.86% parameter and 55.48% FLOP reduction for this block relative to a full 3D residual block, which is what makes temporal context affordable inside multi-stage coupling stacks. CSTVR and LF-VSN both name memory as their limit on temporal context: Continuous Space-Time Video Resampling for its 3D blocks and 3D Swin layers, and LF-VSN for its 3-frame window.

Training. Training follows the IRN recipe on Vimeo-90K: a low-resolution guidance loss, a reconstruction loss, and a likelihood term on the code. The likelihood term is now under the causal conditional prior. Quantization of the low-resolution output is handled with the straight-through estimator as in IRN. We also test a lossy-codec stage between downscaling and upscaling, since every video paper in the file flags codec fragility. That stage is a secondary measurement, not the main claim.

What is new. The novelty is a specific structural position, not "add (2+1)D convolutions somewhere":

  • Versus RISTN. Residual Invertible Spatio-Temporal Network uses invertible blocks only as a spatial feature extractor inside a non-invertible super-resolution net, with a 130+ layer ConvLSTM handling time. Our temporal modeling sits inside the bijection.
  • Versus CSTVR. CSTVR inserts a small invertible module into an otherwise non-invertible 3D pipeline trained in two stages. Ours is invertible end to end.
  • Versus LF-VSN and MIMO-VRN. Both feed concatenated frames to 2D coupling. Ours makes time an explicit, causal axis of the conditioning networks.

We deliberately exclude a temporal Haar wavelet. It pairs frames, which reintroduces look-ahead, so it would undercut the streaming claim.

Baselines

Naive reference: frame-wise IRN. This is the IRN architecture trained on Vimeo-90K frames and applied independently per frame. The file reports about 31.29 dB PSNR on Vid4 at 4× in the MIMO-VRN paper's comparison. We retrain it in our pipeline so that only architecture differs. A second variant samples the code as all zeros instead of drawing it randomly. That variant removes sampling flicker for free, so it is the honest frame-wise floor for temporal-consistency metrics.

Strongest existing method: MIMO-VRN. It is the strongest competitor because it is the only cited invertible rescaler that models time in both directions. The file gives its result as 33.79 dB PSNR on Vid4 at 4×. We report that published figure. We also retrain MIMO-VRN under our data, training-iteration, and tuning budget, so that parameter count, look-ahead latency, and flicker are measured identically for both models.

LSTM-VRN. This is the same paper's 2D invertible downscaler with a non-invertible bidirectional LSTM upscaler. It tests whether temporal context in the upscaler alone suffices. No reported Vid4 number for it was found on Lacuna, so we will retrain it under the same budget.

What counts as beating MIMO-VRN. We call the proposal successful if it does three things together:

  • comes within 0.3 dB of MIMO-VRN on Vid4;
  • uses at most half its parameters and zero future frames;
  • shows lower temporal warping error than MIMO-VRN.

A clear PSNR gain would be a bonus, not the claim.

Ablations. Three ablations isolate the key ingredient:

  • Temporal kernel of size one. Same depth and nearly the same parameters, but the temporal steps degenerate into spatial ones. This separates "temporal coupling" from "more coupling layers."
  • Centered, non-causal temporal kernel. This measures what causality costs.
  • Monolithic 3×3×3 kernel at matched FLOPs. This tests the separability assumption.

Finally, we swap the causal conditional prior for IRN's independent Gaussian, to see how much of any flicker reduction belongs to the prior versus the coupling layers.

Why not just…?

Why not just drop (2+1)D convolutions into MIMO-VRN's coupling networks? This is the obvious combination, and it may well match our PSNR, because MIMO-VRN already sees future frames. It does not remove group-of-frames buffering, which is the latency we target. The non-causal ablation is essentially this model in our codebase. If it beats the causal model by a wide margin on both quality and flicker, the streaming design costs too much, and we would report that plainly.

Why not just run frame-wise IRN with the code fixed to zero? This is the most serious threat to the flicker claim. A deterministic code removes random grain flicker without any temporal modeling. It cannot recover detail that is consistent with motion, though, and IRN itself notes that ignoring instance-specific detail produces artifacts. We will compare temporal warping error and PSNR jointly. If zero-code IRN matches us on consistency, our flicker contribution reduces to the prior, and the paper would be about efficiency alone.

Why not just add a post-hoc temporal deflickering network or a ConvLSTM to the upscaler? LSTM-VRN is exactly the second option, and the downscaler then never sees motion. RISTN needed more than 130 recurrent layers and reached 26.13 dB on Vid4 at 4× as a pure super-resolution model. The LSTM-VRN baseline answers this directly for the rescaling setting. If it matches us at similar parameters, temporal context inside the bijection is not paying for itself.

Why not just use full 3D convolutions or 3D attention inside the coupling blocks? CSTVR identifies quadratic attention cost in its 3D Swin upsampler as its main bottleneck, and FSTRN shows factorization cuts parameters and FLOPs by more than half. Coupled, non-rigid motion is still where separable kernels can lose. The matched-FLOP monolithic ablation will show whether that happens on Vid4's camera-motion sequences.

Proposed Experiments

Data and metrics. All models train on Vimeo-90K and are evaluated at 4× on Vid4 following MIMO-VRN's protocol. Generalization is checked at 2× on Vimeo-90K test clips. We report:

  • PSNR and SSIM;
  • the per-frame PSNR fluctuation within a clip, the quantity MIMO-VRN's center loss targets;
  • flow-warped temporal error between consecutive restored frames;
  • parameters, FLOPs, and frames of look-ahead.

Codec stress test. We H.264-encode the low-resolution video at several quality settings before upscaling. No reported codec-robustness number for any invertible rescaler was found on Lacuna, so we report the curves for all retrained models.

Decisive checks. Three outcomes settle the claim. First, if the size-one temporal kernel ablation matches the full model on warping error, temporal coupling is not doing the work. Second, if causal and non-causal variants differ by under 0.2 dB, streaming is nearly free, which would be the headline result. Third, if the causal model matches MIMO-VRN's 33.79 dB within the stated margin at half the parameters, the efficiency case is made. If instead it trails by more than a decibel, the factorized causal bijection is too weak, and the gap itself shows how much look-ahead is worth in invertible rescaling.