RVM+: An AI-Driven Vision Sensor Framework for High-Precision, Real-Time Video Portrait Segmentation with Enhanced Temporal Consistency and Optimized Model Design

Na Tang, Yuehui Liao, Yu Chen, Guang Yang, Xiaobo Lai, Jing ChenSensors 2025 · DOI

TL;DR

This paper introduces RVM+, an improved AI framework for real-time video portrait segmentation. By integrating ConvGRU for better temporal consistency and employing knowledge distillation for efficiency, RVM+ achieves state-of-the-art accuracy and speed, making it suitable for resource-constrained applications like AR and robotics.

What Problem Does This Paper Address?

Accurate segmentation of portraits in videos is crucial for many AI applications, including human-computer interaction, augmented reality, and autonomous navigation. However, real-world video presents significant challenges: rapid motion, objects occluding each other, and constantly changing backgrounds can all lead to inconsistent segmentation results across frames. Furthermore, many existing segmentation models are computationally expensive, making them unsuitable for real-time applications on devices with limited processing power. The paper aims to address these issues by developing a framework that is both highly accurate and computationally efficient, with a strong emphasis on maintaining temporal consistency in video sequences.

Methods

The core of this work is the enhancement of the existing Robust Video Matting (RVM) framework. The key innovation is the integration of Convolutional Gated Recurrent Units (ConvGRU). Unlike standard recurrent units, ConvGRUs combine convolutional operations with recurrent mechanisms, allowing them to effectively capture both spatial features within a frame and temporal dependencies across frames. This dual capability is crucial for maintaining consistent segmentation even as the scene or subject changes.

To address the computational cost, the researchers implemented a knowledge distillation strategy. This involves training a larger, more complex "teacher" model (RVM+) and then using its learned "soft targets" (probability distributions) to train a smaller, more efficient "student" model. This process transfers the knowledge and performance of the teacher to the student model with minimal loss in accuracy, significantly reducing model size and inference time.

The model also employs a Laplace pyramid loss for frame-wise segmentation and an additional temporal correlation loss to explicitly penalize inconsistencies between consecutive frames.

Datasets

The study utilized a combination of existing and augmented datasets for training and evaluation:

  • VideoMatte240K: A large dataset with high-resolution alpha masks and foregrounds.
  • YouTubeVIS: A dataset for video instance segmentation, filtered to include human clips.
  • PhotoMatte85: Contains half-length portraits without backgrounds.
  • Adobe Image Matting: Provides examples of portrait segmentation.
  • Supervisely Person Dataset: Includes labeled human body images.
  • VMD: Contains alpha masks and foreground video clips of people.
  • Distinction-646: Screened images containing people.
  • Background Videos (BG): Screened video sets without characters.

Additionally, the authors created a custom dataset, VM-datasets, by aggregating and augmenting these sources. This involved adding images from MulData [21-27] and 2000 background images to ensure diversity and realism, particularly for video applications. Data augmentation techniques like random rotation, scaling, and background fitting were applied to enhance the dataset's robustness.

Key Insights and Ideas

  • ConvGRU is a game-changer for temporal consistency: The paper strongly suggests that integrating ConvGRU into video segmentation architectures is highly effective for capturing long-range temporal dependencies, leading to significantly more stable and accurate results in dynamic scenes.
  • Knowledge distillation is a practical necessity: For real-time applications, especially on edge devices, knowledge distillation is not just a performance booster but a critical step to make complex models deployable without sacrificing too much accuracy. The minimal accuracy drop (1.0-1.5%) highlights its efficiency.
  • Balancing detail and speed is achievable: The RVM+ framework demonstrates that it's possible to achieve both high-fidelity segmentation (capturing fine details like hair) and real-time performance, a common trade-off in computer vision.
  • Dataset diversity is key for generalization: The creation and use of a diverse, augmented dataset (VM-datasets) were crucial for evaluating the model's performance across a wide range of real-world scenarios and ensuring its generalizability.

Key Results

  • Superior Segmentation Accuracy: RVM+ consistently outperformed other state-of-the-art models (BGMv2, ConnectNet, DeepLabV3, MODNet) across multiple datasets.
  • On the VS-test dataset, RVM+ achieved a Mean Intersection over Union (MIoU) of 0.974, a SAD of 5.81, and a dtSSD of 1.78.
  • For the challenging VMHD_TS dataset, RVM+ achieved an MIoU of 0.852 and a significantly improved SAD of 6.57 compared to other models.
  • On PhotoMatte85-TS, RVM+ achieved an MIoU of 0.895 and a dtSSD of 1.84.
  • Enhanced Temporal Consistency: Metrics like dtSSD and MSE showed significant improvements, indicating better stability and reduced errors across video frames. For example, the dtSSD for RVM+ on VS-test was 1.78, compared to 2.02 for the original RVM.
  • Real-time Performance: The model achieved high Frames Per Second (FPS) rates, reaching up to 32.3 FPS on the VS-test dataset, making it suitable for real-time applications.
  • Efficiency through Distillation: After knowledge distillation, the model maintained high accuracy while significantly reducing computational demands, with minimal accuracy loss.

Limitations and Future Work

  • Generalization Variability: While RVM+ performs well, the paper notes some variability in performance across different datasets, which is typical for model generalization. Future work could focus on further improving this generalizability.
  • Complex Multi-Subject/Multi-Modal Scenarios: The authors suggest future research will focus on enhancing the model's generalizability to handle more complex multi-subject and multi-modal scenarios.
  • Integration with Sensing Systems: The paper highlights exciting opportunities for integrating RVM+ with advanced sensing systems like IoT devices and wearable sensors for applications in augmented reality, human-computer interaction, and autonomous systems. This suggests a path towards more ubiquitous and integrated AI vision solutions.

Summary updated

Abstract

Video portrait segmentation is essential for intelligent sensing systems, including human-computer interaction, autonomous navigation, and augmented reality. However, dynamic video environments introduce significant challenges, such as temporal variations, occlusions, and computational constraints. This study introduces RVM+, an enhanced video segmentation framework based on the Robust Video Matting (RVM) architecture. By incorporating Convolutional Gated Recurrent Units (ConvGRU), RVM+ improves temporal consistency and captures intricate temporal dynamics across video frames. Additionally, a novel knowledge distillation strategy reduces computational demands while maintaining high segmentation accuracy, making the framework ideal for real-time applications in resource-constrained environments. Comprehensive evaluations on challenging datasets show that RVM+ outperforms state-of-the-art methods in both segmentation accuracy and temporal consistency. Key performance indicators such as MIoU, SAD, and dtSSD effectively verify the robustness and efficiency of the model. The integration of knowledge distillation ensures a streamlined and effective design with negligible accuracy trade-offs, highlighting its suitability for practical deployment. This study makes significant strides in intelligent sensor technology, providing a high-performance, efficient, and scalable solution for video segmentation. RVM+ offers potential for applications in fields such as augmented reality, robotics, and real-time video analysis, while also advancing the development of AI-enabled vision sensors.
BibTeX
@inproceedings{tang2025rvm,
  author = {Na Tang and Yuehui Liao and Yu Chen and Guang Yang and Xiaobo Lai and Jing Chen},
  title = {{RVM+: An AI-Driven Vision Sensor Framework for High-Precision, Real-Time Video Portrait Segmentation with Enhanced Temporal Consistency and Optimized Model Design}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25051278},
}
Figure