BayesCrop: Real-Time Probabilistic Multi-Crop Video Reframing

Cheng-Han Lee, Rajesh Sureddi, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan BovikarXiv 2026 · arXiv

TL;DR

BayesCrop is a real-time framework that treats video reframing as a probabilistic problem rather than a deterministic one. By modeling crop preferences as a two-component Gaussian Mixture Model (GMM), it captures multiple regions of interest and their local uncertainties, achieving 112 FPS throughput while significantly improving the coverage of human-preferred crops.

What Problem Does This Paper Address?

Converting landscape video to portrait (reframing) is a staple of modern social media, but it’s a task fraught with subjectivity. If two people are talking at opposite ends of a wide shot, which one should the camera follow? Most existing automated tools try to predict a single "optimal" bounding box, which oversimplifies the problem and often misses valid alternative viewpoints.

Furthermore, processing high-resolution video is computationally expensive. Running a heavy neural network on every single frame to find the "best" crop is often too slow for real-time applications or mobile devices. The authors of BayesCrop set out to create a system that is both probabilistically aware (handling multiple subjects) and computationally efficient (running in real-time).

Methods

BayesCrop introduces a few clever architectural and algorithmic shifts to solve these challenges:

  1. Probabilistic Multi-Crop Modeling: Instead of predicting coordinates directly, the model predicts the parameters $(\pi, \mu, \sigma)$ of a two-component spatially isotropic GMM. Each component represents a "mode" or a likely region of interest. The mean ($\mu$) gives the representative crop, while the variance ($\sigma$) represents the local uncertainty, allowing the system to sample additional plausible crop candidates.
  2. Anchor-and-Propagate Strategy: To save on compute, BayesCrop doesn't run its full neural predictor on every frame. Instead, it performs inference on sparse "anchor" frames. For the intermediate frames, it propagates the crop trajectories using motion vectors already available in the compressed video bitstream. This avoids the need for expensive optical flow or separate tracking networks.
  3. Architecture: The model uses a frozen DINOv2 backbone for robust feature extraction, followed by a multi-scale feature aggregator and a Transformer-based decoder (similar to DETR). Two learnable queries interact with these features to output the parameters for the two GMM components.
Overview of the BayesCrop pipeline, showing anchor-frame prediction and motion-vector propagation.

Datasets

The authors found existing datasets lacking for multi-crop evaluation, so they developed:

  • LIVE-YT-MC (Multi-Crop): A new dataset derived from human subjective annotations. It explicitly preserves multiple crop modes and their local variations, providing a richer ground truth for training probabilistic models.
  • Multi-Target Long Video Benchmark: A new evaluation set consisting of longer videos (avg. 24 seconds) with multiple distinct subjects, used to test temporal stability and viewpoint diversity.

They also evaluated on standard benchmarks like LIVE-YT-VC++ and RetargetVid.

Key Insights and Ideas

  • Embracing Ambiguity: By using a GMM, the model acknowledges that there isn't always one "right" answer. This allows for "dual-view" prediction where two distinct subjects can be tracked simultaneously.
  • Efficiency via Bitstream: Leveraging motion vectors from the compressed domain is a "free" way to maintain temporal coherence. It naturally handles camera motion and subject movement between neural inference steps.
  • Isotropic Simplification: The authors found that simplifying the GMM covariance to be spatially isotropic (where $x$ and $y$ share variance) reduced model complexity and improved training stability without sacrificing the ability to capture local uncertainty.
  • View Switching: For a single-output video, they use a lightweight "Periodic View Switch Agent" that alternates between the two predicted trajectories. This creates a more dynamic, "edited" feel compared to staying glued to a single subject.
Comparison of the probabilistic ROI prediction against human-annotated clusters.

Key Results

  • Speed: BayesCrop achieves a throughput of 112 FPS on a standard GPU, making it truly real-time.
  • Coverage: On the LIVE-YT-VC++ benchmark, BayesCrop achieved a 68.4% "Best-of-2" IoU. This means that if you look at both predicted modes, there is a very high likelihood that one of them matches what a human would choose.
  • Single-Output Quality: Even when forced to pick just one crop, BayesCrop remains competitive with state-of-the-art deterministic models (reaching 55.2 mIoU) while being significantly smaller and faster.
  • Human Preference: In a three-way study on long videos, BayesCrop was preferred 39.4% of the time, outperforming CenterCrop (33.3%) and even a powerful MLLM baseline like Gemini 3 Flash (27.2%).
Qualitative results showing how BayesCrop handles multiple subjects over time compared to baselines.

Limitations and Future Work

  • Shot Planning: The current view-switching strategy is periodic and lightweight. The authors note that a more sophisticated "semantic" shot planner (perhaps using an MLLM) could further improve the narrative flow of the final reframed video.
  • Component Limit: The model is currently limited to two GMM components ($K=2$). While this covers many common scenarios (like a two-person interview), extremely crowded scenes might benefit from a higher $K$.
  • Motion Vector Noise: The reliance on compressed-domain motion vectors means performance could degrade if the input video has significant encoding artifacts or very low bitrates.

Summary updated

Abstract

Automatic video reframing is increasingly important for adapting landscape content to heterogeneous display formats. Existing methods typically predict a single deterministic crop, despite the subjective nature of reframing and the presence of multiple plausible regions of interest. We present \textbf{BayesCrop}, a real-time probabilistic framework for multi-crop video reframing. BayesCrop represents crop preferences using a two-component spatially isotropic Gaussian mixture over crop position and scale. Each component defines a representative reframing mode together with its local uncertainty, enabling both dual-view prediction and sampling of additional plausible crop candidates. For efficient video processing, we introduce an anchor-and-propagate strategy that performs neural prediction only on sparse anchor frames and propagates crop trajectories using motion vectors available from the compressed video bitstream. The resulting trajectories can be combined into a single output using a lightweight periodic view-switching strategy. Experiments on LIVE-YT-MC, LIVE-YT-VC++, and a multi-target long-video benchmark demonstrate competitive single-output localization, high multi-crop reference coverage, diverse yet coherent crop hypotheses, and stable perceptual quality. BayesCrop achieves real-time throughput of 112 FPS at the default operating point, while sampling from the learned distributions further improves reference coverage beyond the two representative crop means.
BibTeX
@misc{lee2026bayescrop,
  author = {Cheng-Han Lee and Rajesh Sureddi and Neil Birkbeck and Yilin Wang and Balu Adsumilli and Alan Bovik},
  title = {{BayesCrop: Real-Time Probabilistic Multi-Crop Video Reframing}},
  year = {2026},
  eprint = {2611.09100},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09100},
}
Figure