MIMT: Masked Image Modeling Transformer for Video Compression

Jinxi Xiang, Kuan Tian, Jun ZhangICLR 2023 · OpenReview

Introduction

Neural video compression has made significant strides, but a major bottleneck remains: the entropy model. Traditionally, these models are autoregressive, predicting pixels or tokens one by one in a fixed "raster-scan" order (like reading a book from top-left to bottom-right). This unidirectional constraint ignores useful context that might appear later in the sequence and is incredibly slow for high-resolution video.

To address this, researchers have introduced MIMT (Masked Image Modeling Transformer). MIMT reimagines video compression by replacing rigid raster-scan entropy models with a flexible, bidirectional transformer. By treating video decoding as an iterative "mask-and-predict" task—similar to how BERT handles text—it achieves state-of-the-art efficiency, outperforming the latest H.266 (VVC) standard with significant bitrate savings.

The MIMT Architecture

MIMT overcomes the limitations of sequential decoding by leveraging several novel techniques designed to capture both spatial and temporal dependencies efficiently.

MIMT Architecture Overview
  • Bidirectional Transformer: Unlike standard autoregressive models, MIMT uses bidirectional attention. During training, it learns to predict randomly masked tokens by looking at all other available tokens in the frame, capturing a holistic view of the context.
  • Manifold Priors: To give the transformer the best possible starting point, the framework feeds it three types of information:
  • Side Information: A hierarchical spatial estimate of the current frame.
  • Temporal Prior: The previously decoded frame.
  • Recurrent Prior: A hidden state from a ConvLSTM that accumulates information over a long range of previous frames.
  • Shifted Window Attention: To keep the transformer's computational cost manageable for 1080p video, MIMT utilizes Swin-Transformer-style shifted windows, restricting attention to local neighborhoods rather than computing attention across the entire frame at once.

Iterative "Easy-to-Hard" Decoding

A standout feature of MIMT is its inference strategy. The model abandons the traditional top-to-bottom scanning order, which acts as an arbitrary bottleneck. Instead, it employs an Iterative Decoding Scheduler.

Iterative Decoding Process

At inference, the model starts by predicting all tokens in parallel. It keeps the most "confident" ones (those with the lowest entropy) and uses them as context to refine the remaining masked tokens in subsequent steps. Using a sine-function scheduler, this mimics a "sketch-to-detail" artistic process. This "easy-to-hard" decoding naturally resolves simpler parts of the image first (like static backgrounds) and uses that foundation to predict harder parts (like fast-moving objects or complex textures).

Performance and Results

Trained on the Vimeo-90k dataset and evaluated on standard benchmarks (HEVC Class B, UVG, and MCL-JCV), MIMT achieves impressive gains over both traditional and neural baselines:

  • State-of-the-Art Compression: Compared to the VTM (H.266/VVC Reference), MIMT achieves an average 22.7% bitrate saving (PSNR) and a massive 56.3% saving (SSIM). It also consistently outperforms top-tier neural codecs like DMC and DCVC.
  • Decoding Efficiency: While slower than purely parallel models, MIMT is significantly faster than full autoregressive models. For 1080p video, it decodes in roughly 1.3 seconds per frame on an NVIDIA P40 GPU—a 100x speedup over sequential autoregressive decoding.
  • The Power of Recurrence: The inclusion of a recurrent prior (ConvLSTM) allows the model to "remember" details from further back in the video, significantly boosting overall compression performance compared to models that only look at the immediate previous frame.

Limitations and Future Work

Despite its massive efficiency gains over sequential models, MIMT's 1.3 seconds per frame decoding time is still far from the ~0.03 seconds required for real-time 30fps playback. Furthermore, the architecture is highly sophisticated, combining CNNs, Transformers, and LSTMs, which may make hardware-level implementation on mobile chips challenging. Future work will likely focus on optimizing the number of decoding steps and exploring even more efficient attention mechanisms to bridge the gap toward real-time neural video playback.

Summary updated

Abstract

Deep learning video compression outperforms its hand-craft counterparts with enhanced flexibility and capacity. One key component of the learned video codec is the autoregressive entropy model conditioned on spatial and temporal priors. Operating autoregressive on raster scanning order naively treats the context as unidirectional. This is neither efficient nor optimal, considering that conditional information probably locates at the end of the sequence. We thus introduce an entropy model based on a masked image modeling transformer (MIMT) to learn the spatial-temporal dependencies. Video frames are first encoded into sequences of tokens and then processed with the transformer encoder as priors. The transformer decoder learns the probability mass functions (PMFs) \emph{conditioned} on the priors and masked inputs. Then it is capable of selecting optimal decoding orders without a fixed direction. During training, MIMT aims to predict the PMFs of randomly masked tokens by attending to tokens in all directions. This allows MIMT to capture the temporal dependencies from encoded priors and the spatial dependencies from the unmasked tokens, i.e., decoded tokens. At inference time, the model begins with generating PMFs of all masked tokens in parallel and then decodes the frame iteratively from the previously-selected decoded tokens (i.e., with high confidence). In addition, we improve the overall performance with more techniques, e.g., manifold conditional priors accumulating a long range of information, shifted window attention to reduce complexity. Extensive experiments demonstrate the proposed MIMT framework equipped with the new transformer entropy model achieves state-of-the-art performance on HEVC, UVG, and MCL-JCV datasets, generally outperforming the VVC in terms of PSNR and SSIM.
BibTeX
@inproceedings{xiang2023mimt,
  author = {Jinxi Xiang and Kuan Tian and Jun Zhang},
  title = {{MIMT: Masked Image Modeling Transformer for Video Compression}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=j9m-mVnndbm},
}
Figure