Virtual Multi-Modality Self-Supervised Foreground Matting for Human-Object Interaction

Bo Xu, Han Huang, Cheng Lu, Ziwen Li, Yandong Guo

Abstract

Image matting typically focuses on extracting human subjects, often neglecting the objects they interact with. We present Virtual Multi-Modality Foreground Matting (VMFM), a novel framework designed to extract high-quality alpha mattes of both humans and their interacted objects from a single RGB image. By leveraging "virtual" modalities—depth maps, segmentation masks, and interaction heatmaps—VMFM learns to group humans and objects without manual labels or specialized hardware. Through a self-supervised "Complementary Learning" strategy, the model achieves state-of-the-art performance on new large-scale benchmarks, LFM40K and UFM75K.

Introduction

Image matting is the high-precision task of separating foreground elements from their backgrounds. While deep learning has significantly advanced human matting, existing models often fail in Human-Object Interaction (HOI) scenarios. For instance, when a person holds a guitar or a tennis racket, standard models frequently treat the object as part of the background, "cutting it off" from the human subject.

Furthermore, high-quality matting usually relies on auxiliary inputs like trimaps or pre-captured background images, which are rarely available in "in the wild" photography. Creating ground-truth labels for HOI matting is also prohibitively expensive. VMFM addresses these challenges by using a self-supervised approach that requires only a single RGB image.

Methodology

The VMFM framework utilizes a dual-stream architecture that processes an RGB image alongside three "virtual" modalities generated by pre-existing models:

  1. Estimated Depth Map: Provides spatial continuity to group the human and object as a single entity.
  2. Segmentation Mask: Offers a coarse outline of the human body.
  3. Interaction Heatmap: Identifies likely contact points between the human and the object.
The VMFM framework architecture featuring virtual modality generation and the Complementary Learning module.

As shown in the architecture above, these modalities feed into two parallel networks: SFPNet (segmentation-based) and DFPNet (depth-based). To train on unlabeled data, the authors employ Complementary Learning (CL). In this self-supervised setup, the two networks supervise each other based on their confidence levels (deviation probability). If one stream is more certain about a pixel's classification, it provides the gradient to update the other. This allows the model to combine the strengths of different modalities—such as depth's ability to group objects and color's ability to define fine edges like hair. A final Refinement Network (RN) sharpens the matte by focusing on uncertain regions.

Datasets

To facilitate research in HOI matting, the authors introduced two major datasets:

  • LFM40K (Labeled Foreground Matting 40K): Over 40,000 labeled frames across 20 interaction classes (e.g., riding, carrying, kicking).
  • UFM75K (Unlabeled Foreground Matting 75K): 75,000 unlabeled images used for the self-supervised Complementary Learning phase.
  • Adobe Matting Dataset & Dist-646: Used for pre-training and benchmarking against existing methods.

Results and Key Insights

VMFM demonstrates a significant quantitative and qualitative leap over previous state-of-the-art (SOTA) methods like Background Matting (BGM).

Qualitative comparison between the baseline BGM and the proposed VMFM, showing better preservation of interacted objects.

On the LFM40K test set, the VMFM model (D-CL-RN) achieved a Mean Squared Error (MSE) of 2.08 ($10^{-2}$), nearly half that of BGM (4.28). Qualitatively, VMFM excels at preserving thin objects and fine details, such as golf clubs or hair, which are often erased by human-centric models. A user study confirmed these findings, with participants preferring VMFM's results over 80% of the time.

The core insight of VMFM is the use of depth as a "grouper." While RGB features might struggle to link a person to a visually distinct object (like a colorful backpack), depth maps provide the spatial continuity needed to recognize them as a single foreground unit. Additionally, interaction heatmaps provide a semantic prior that prevents the model from accidentally cropping out essential tools or equipment.

Conclusion and Future Work

VMFM provides a robust, self-supervised solution for matting in complex interaction scenarios without requiring manual labels or background images. While the model occasionally struggles when the foreground and background share very similar colors or textures, it sets a new standard for HOI matting. Future research may extend these multi-modal self-supervision techniques to 3D body and scene reconstruction, moving beyond human-only "voxels" to include the objects that define human activity.

Summary updated

Abstract

Most existing human matting algorithms tried to separate pure human-only foreground from the background. In this paper, we propose a Virtual Multi-modality Foreground Matting (VMFM) method to learn human-object interactive foreground (human and objects interacted with him or her) from a raw RGB image. The VMFM method requires no additional inputs, e.g. trimap or known background. We reformulate foreground matting as a self-supervised multi-modality problem: factor each input image into estimated depth map, segmentation mask, and interaction heatmap using three auto-encoders. In order to fully utilize the characteristics of each modality, we first train a dual encoder-to-decoder network to estimate the same alpha matte. Then we introduce a self-supervised method: Complementary Learning(CL) to predict deviation probability map and exchange reliable gradients across modalities without label. We conducted extensive experiments to analyze the effectiveness of each modality and the significance of different components in complementary learning. We demonstrate that our model outperforms the state-of-the-art methods.
BibTeX
@inproceedings{xu2021virtual,
  author = {Bo Xu and Han Huang and Cheng Lu and Ziwen Li and Yandong Guo},
  title = {{Virtual Multi-Modality Self-Supervised Foreground Matting for Human-Object Interaction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00049},
}
Figure