Introduction
Neural video compression has made significant strides, but a major bottleneck remains: the entropy model. Traditionally, these models are autoregressive, predicting pixels or tokens one by one in a fixed "raster-scan" order (like reading a book from top-left to bottom-right). This unidirectional constraint ignores useful context that might appear later in the sequence and is incredibly slow for high-resolution video.
To address this, researchers have introduced MIMT (Masked Image Modeling Transformer). MIMT reimagines video compression by replacing rigid raster-scan entropy models with a flexible, bidirectional transformer. By treating video decoding as an iterative "mask-and-predict" task—similar to how BERT handles text—it achieves state-of-the-art efficiency, outperforming the latest H.266 (VVC) standard with significant bitrate savings.
The MIMT Architecture
MIMT overcomes the limitations of sequential decoding by leveraging several novel techniques designed to capture both spatial and temporal dependencies efficiently.
- Bidirectional Transformer: Unlike standard autoregressive models, MIMT uses bidirectional attention. During training, it learns to predict randomly masked tokens by looking at all other available tokens in the frame, capturing a holistic view of the context.
- Manifold Priors: To give the transformer the best possible starting point, the framework feeds it three types of information:
- Side Information: A hierarchical spatial estimate of the current frame.
- Temporal Prior: The previously decoded frame.
- Recurrent Prior: A hidden state from a ConvLSTM that accumulates information over a long range of previous frames.
- Shifted Window Attention: To keep the transformer's computational cost manageable for 1080p video, MIMT utilizes Swin-Transformer-style shifted windows, restricting attention to local neighborhoods rather than computing attention across the entire frame at once.
Iterative "Easy-to-Hard" Decoding
A standout feature of MIMT is its inference strategy. The model abandons the traditional top-to-bottom scanning order, which acts as an arbitrary bottleneck. Instead, it employs an Iterative Decoding Scheduler.
At inference, the model starts by predicting all tokens in parallel. It keeps the most "confident" ones (those with the lowest entropy) and uses them as context to refine the remaining masked tokens in subsequent steps. Using a sine-function scheduler, this mimics a "sketch-to-detail" artistic process. This "easy-to-hard" decoding naturally resolves simpler parts of the image first (like static backgrounds) and uses that foundation to predict harder parts (like fast-moving objects or complex textures).
Performance and Results
Trained on the Vimeo-90k dataset and evaluated on standard benchmarks (HEVC Class B, UVG, and MCL-JCV), MIMT achieves impressive gains over both traditional and neural baselines:
- State-of-the-Art Compression: Compared to the VTM (H.266/VVC Reference), MIMT achieves an average 22.7% bitrate saving (PSNR) and a massive 56.3% saving (SSIM). It also consistently outperforms top-tier neural codecs like DMC and DCVC.
- Decoding Efficiency: While slower than purely parallel models, MIMT is significantly faster than full autoregressive models. For 1080p video, it decodes in roughly 1.3 seconds per frame on an NVIDIA P40 GPU—a 100x speedup over sequential autoregressive decoding.
- The Power of Recurrence: The inclusion of a recurrent prior (ConvLSTM) allows the model to "remember" details from further back in the video, significantly boosting overall compression performance compared to models that only look at the immediate previous frame.
Limitations and Future Work
Despite its massive efficiency gains over sequential models, MIMT's 1.3 seconds per frame decoding time is still far from the ~0.03 seconds required for real-time 30fps playback. Furthermore, the architecture is highly sophisticated, combining CNNs, Transformers, and LSTMs, which may make hardware-level implementation on mobile chips challenging. Future work will likely focus on optimizing the number of decoding steps and exploring even more efficient attention mechanisms to bridge the gap toward real-time neural video playback.
Summary updated