TL;DR
BayesCrop is a real-time framework that treats video reframing as a probabilistic problem rather than a deterministic one. By modeling crop preferences as a two-component Gaussian Mixture Model (GMM), it captures multiple regions of interest and their local uncertainties, achieving 112 FPS throughput while significantly improving the coverage of human-preferred crops.
What Problem Does This Paper Address?
Converting landscape video to portrait (reframing) is a staple of modern social media, but it’s a task fraught with subjectivity. If two people are talking at opposite ends of a wide shot, which one should the camera follow? Most existing automated tools try to predict a single "optimal" bounding box, which oversimplifies the problem and often misses valid alternative viewpoints.
Furthermore, processing high-resolution video is computationally expensive. Running a heavy neural network on every single frame to find the "best" crop is often too slow for real-time applications or mobile devices. The authors of BayesCrop set out to create a system that is both probabilistically aware (handling multiple subjects) and computationally efficient (running in real-time).
Methods
BayesCrop introduces a few clever architectural and algorithmic shifts to solve these challenges:
- Probabilistic Multi-Crop Modeling: Instead of predicting coordinates directly, the model predicts the parameters $(\pi, \mu, \sigma)$ of a two-component spatially isotropic GMM. Each component represents a "mode" or a likely region of interest. The mean ($\mu$) gives the representative crop, while the variance ($\sigma$) represents the local uncertainty, allowing the system to sample additional plausible crop candidates.
- Anchor-and-Propagate Strategy: To save on compute, BayesCrop doesn't run its full neural predictor on every frame. Instead, it performs inference on sparse "anchor" frames. For the intermediate frames, it propagates the crop trajectories using motion vectors already available in the compressed video bitstream. This avoids the need for expensive optical flow or separate tracking networks.
- Architecture: The model uses a frozen DINOv2 backbone for robust feature extraction, followed by a multi-scale feature aggregator and a Transformer-based decoder (similar to DETR). Two learnable queries interact with these features to output the parameters for the two GMM components.
Datasets
The authors found existing datasets lacking for multi-crop evaluation, so they developed:
- LIVE-YT-MC (Multi-Crop): A new dataset derived from human subjective annotations. It explicitly preserves multiple crop modes and their local variations, providing a richer ground truth for training probabilistic models.
- Multi-Target Long Video Benchmark: A new evaluation set consisting of longer videos (avg. 24 seconds) with multiple distinct subjects, used to test temporal stability and viewpoint diversity.
They also evaluated on standard benchmarks like LIVE-YT-VC++ and RetargetVid.
Key Insights and Ideas
- Embracing Ambiguity: By using a GMM, the model acknowledges that there isn't always one "right" answer. This allows for "dual-view" prediction where two distinct subjects can be tracked simultaneously.
- Efficiency via Bitstream: Leveraging motion vectors from the compressed domain is a "free" way to maintain temporal coherence. It naturally handles camera motion and subject movement between neural inference steps.
- Isotropic Simplification: The authors found that simplifying the GMM covariance to be spatially isotropic (where $x$ and $y$ share variance) reduced model complexity and improved training stability without sacrificing the ability to capture local uncertainty.
- View Switching: For a single-output video, they use a lightweight "Periodic View Switch Agent" that alternates between the two predicted trajectories. This creates a more dynamic, "edited" feel compared to staying glued to a single subject.
Key Results
- Speed: BayesCrop achieves a throughput of 112 FPS on a standard GPU, making it truly real-time.
- Coverage: On the LIVE-YT-VC++ benchmark, BayesCrop achieved a 68.4% "Best-of-2" IoU. This means that if you look at both predicted modes, there is a very high likelihood that one of them matches what a human would choose.
- Single-Output Quality: Even when forced to pick just one crop, BayesCrop remains competitive with state-of-the-art deterministic models (reaching 55.2 mIoU) while being significantly smaller and faster.
- Human Preference: In a three-way study on long videos, BayesCrop was preferred 39.4% of the time, outperforming CenterCrop (33.3%) and even a powerful MLLM baseline like Gemini 3 Flash (27.2%).
Limitations and Future Work
- Shot Planning: The current view-switching strategy is periodic and lightweight. The authors note that a more sophisticated "semantic" shot planner (perhaps using an MLLM) could further improve the narrative flow of the final reframed video.
- Component Limit: The model is currently limited to two GMM components ($K=2$). While this covers many common scenarios (like a two-person interview), extremely crowded scenes might benefit from a higher $K$.
- Motion Vector Noise: The reliance on compressed-domain motion vectors means performance could degrade if the input video has significant encoding artifacts or very low bitrates.
Summary updated