Retargeting Video With an End-to-End Framework

Thi Ngoc Hanh Le, Huiguang Huang, Yi-Ru Chen, Tong-Yee Lee

TL;DR

This paper introduces RETVI, a novel end-to-end deep learning framework for video retargeting that significantly speeds up the process while maintaining high visual quality. By eliminating computationally expensive pre-processing steps and learning a direct mapping from input to retargeted video, RETVI offers a practical solution for adapting videos to various aspect ratios.

What Problem Does This Paper Address?

In today's diverse digital landscape, videos need to be adaptable to a wide range of screen sizes and aspect ratios, from smartphones to large displays. Video retargeting aims to achieve this by resizing videos while preserving important visual content and avoiding undesirable artifacts like distortion or flickering. However, existing methods often rely on time-consuming pre-processing steps (like saliency map generation or optical flow estimation) and struggle with handling diverse video content, arbitrary aspect ratios, and real-time performance demands. This makes them impractical for applications like live streaming or on-demand video services.

Methods

RETVI tackles these challenges with an end-to-end framework composed of two key modules:

  • Content Feature Analyzer (CFA): This module acts as a replacement for traditional pre-processing. Inspired by the U-Net architecture, it learns a pixel-wise representation of the input video frame's content, capturing high-level semantic information that is robust to noise and occlusions. This allows it to identify important regions without relying on external, time-consuming analyses.
  • Adaptive Deforming Estimator (ADE): This module takes the features from the CFA and, along with a target resizing ratio, learns to estimate a deformation matrix. This matrix guides how the input frame is transformed to the desired aspect ratio. The ADE employs a hybrid approach, combining pixel-based and grid-based strategies to balance computational efficiency and detail preservation.

The framework is trained using a weighted combination of four loss functions: critical region loss (Lcri) to preserve important objects, global integrity loss (Lglo) for overall visual coherence, temporal consistency loss (Ltem) to prevent flickering, and fidelity loss (Lfid) to ensure high-quality output.

Datasets

The paper uses the DAVIS dataset for training, which contains 90 videos with foreground annotations. For evaluation, they use a set of nine diverse videos and the RetargetMe dataset for image retargeting quality assessment.

Key Insights and Ideas

  • End-to-End Efficiency: By eliminating the need for explicit pre-processing steps like saliency detection or optical flow, RETVI drastically reduces computational overhead, making real-time video retargeting feasible.
  • Robust Content Understanding: The CFA module's ability to learn high-level, pixel-wise features provides a more robust understanding of video content compared to traditional methods that rely on lower-level cues.
  • Unified Framework: RETVI handles both image and video retargeting within a single, unified framework, demonstrating its versatility.
  • Adaptive Deformation: The ADE's ability to learn a deformation matrix that adapts to arbitrary aspect ratios and content types is crucial for its flexibility.

Key Results

  • Speed: RETVI achieves a remarkable speedup, processing a video with 144 frames in approximately 2.3 seconds (0.016 seconds per frame), compared to conventional methods that can take minutes per frame due to pre-processing.
  • Quality: Experimental results show that RETVI outperforms previous methods in terms of visual quality, effectively preserving important content, reducing distortion, and minimizing flickering. Human user studies also indicate a preference for RETVI's results in many scenarios.
  • Retargetability: The method demonstrates good performance even on images with low retargetability scores, producing plausible results where other methods might fail due to cropping or distortion.

Limitations and Future Work

  • Dense/Landscape Scenes: The paper notes that RETVI may not perform optimally on very dense or landscape scenes, where its performance can become similar to linear scaling. This is attributed to the ADE module's difficulty in differentiating importance in such scenarios.
  • Global Integrity Loss (Lglo): The performance of Lglo is dependent on an external feature extractor (ResNet), which can lead to instability in videos with multiple objects distributed across the entire frame.
  • Cropping Effect: In some cases, particularly with main objects located at the extreme edges of the frame, cropping might still occur.

Future work includes developing loss functions independent of external feature extractors, exploring automatic methods for defining important physical regions to mitigate cropping, and investigating text-driven frameworks for semantic understanding in retargeting. The authors also suggest extending the framework for stereo image/video retargeting.

Summary updated

Abstract

Video holds significance in computer graphics applications. Because of the heterogeneous of digital devices, retargeting videos becomes an essential function to enhance user viewing experience in such applications. In the research of video retargeting, preserving the relevant visual content in videos, avoiding flicking, and processing time are the vital challenges. Extending image retargeting techniques to the video domain is challenging due to the high running time. Prior work of video retargeting mainly utilizes time-consuming preprocessing to analyze frames. Plus, being tolerant of different video content, avoiding important objects from shrinking, and the ability to play with arbitrary ratios are the limitations that need to be resolved in these systems requiring investigation. In this paper, we present an end-to-end RETVI method to retarget videos to arbitrary aspect ratios. We eliminate the computational bottleneck in the conventional approaches by designing RETVI with two modules, content feature analyzer (CFA) and adaptive deforming estimator (ADE). The extensive experiments and evaluations show that our system outperforms previous work in quality and running time.
BibTeX
@article{le2024retargeting,
  author = {Thi Ngoc Hanh Le and Huiguang Huang and Yi-Ru Chen and Tong-Yee Lee},
  title = {{Retargeting Video With an End-to-End Framework}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2024},
  url = {https://openreview.net/forum?id=1PIHN9QfFn},
}
Figure