Motion-Appearance Interactive Encoding for Object Segmentation in Unconstrained Videos

Chun-Chao Guo, Jianhuang Lai, Xiaohua XieIEEE TCSVT 2017 · DOI · OpenReview

Abstract

Automatic video object segmentation (VOS) in unconstrained environments remains a significant challenge due to camera jitter, motion blur, and complex backgrounds. This paper presents Interactively Constrained Encoding (ICE), a framework that leverages the mutual constraints between motion and appearance cues. Unlike traditional methods that fuse these features late in the process, ICE allows them to assist each other during encoding, leading to more robust and accurate segmentation.

Introduction

Identifying and tracking moving objects in "unconstrained" video sequences—those characterized by unpredictable camera movement and cluttered scenes—is a fundamental task in computer vision. Standard approaches typically utilize two primary cues: motion (e.g., optical flow) and appearance (e.g., color and texture). However, these cues are often processed independently, leading to errors where motion blur causes "bleeding" across boundaries or where background objects mimic the target's appearance.

This research addresses these issues by recognizing that motion and appearance are homologous; they describe the same physical object and should therefore share structural patterns. By enforcing mutual constraints throughout the encoding process, the proposed ICE method achieves superior performance in challenging scenarios.

Methodology

Interactively Constrained Encoding (ICE)

The core of the proposed framework is the ICE module. Rather than treating motion and appearance as separate inputs to be fused later, ICE uses each cue to refine the other. For instance, appearance-based saliency is restricted by motion patterns to filter out static background clutter. Conversely, motion extraction is guided by appearance boundaries to prevent the "bleeding" effect common in raw optical flow. This interactive process ensures that the resulting probability maps are uniform and well-aligned with the object's true boundaries.

Visualization of the ICE pipeline: (a) Optical flow, (b) Region from motion, (c) Appearance saliency, (d) Region from appearance, (e) Accumulation of cues, and (f) the final ICE output.

Superpixel-Level Optimization

To ensure computational efficiency, the method operates on SLIC superpixels instead of individual pixels. This approach reduces the complexity of the spatiotemporal graph by approximately 100x. Despite this reduction, the semantic structure captured by ICE allows for high-precision segmentation that rivals or exceeds pixel-level methods.

Occlusion Handling and Energy Minimization

The framework incorporates a multi-object localization strategy using weighted bipartite graph matching (the Hungarian algorithm) to maintain consistent object IDs during inter-occlusion. The final segmentation is achieved by minimizing a spatiotemporal energy function that balances unary potentials (foreground likelihood) with pairwise potentials (spatial and temporal consistency).

Experimental Results

The ICE method was evaluated on several benchmark datasets, including SegTrack, MOViCS, and GaTech.

Quantitative Performance

On the SegTrack dataset, ICE achieved a state-of-the-art average per-frame pixel error of 587, a significant improvement over previous benchmarks (e.g., 877 and 592). Furthermore, the method demonstrated an exceptional average precision of 99.4%.

Qualitative Robustness

Qualitative analysis confirms the method's robustness. In sequences like "birdfall" or videos of running individuals, ICE successfully maintained accurate segmentation even when optical flow was degraded by rapid displacement or noise. Ablation studies comparing ICE to Separate Encoding (SE) consistently showed that the interactive mechanism is the primary driver of reduced pixel error.

Conclusion

The Interactively Constrained Encoding (ICE) framework demonstrates that mutual assistance between motion and appearance cues is vital for robust VOS in unconstrained videos. By integrating these cues early and throughout the encoding process, the method achieves high accuracy and efficiency. Future work will explore the incorporation of more complex geometric patterns and the use of wider temporal windows to further enhance stability.

Summary updated

Abstract

We present a novel method of integrating motion and appearance cues for foreground object segmentation in unconstrained videos. Unlike conventional methods encoding motion and appearance patterns individually, our method puts particular emphasis on their mutual assistance. Specifically, we propose using an interactively constrained encoding (ICE) scheme to incorporate motion and appearance patterns into a graph that leads to a spatiotemporal energy optimization. The reason of utilizing ICE is that both motion and appearance cues for the same target share underlying correlative structure, thus can be exploited in a deeply collaborative manner. We perform ICE not only in the initialization but also in the refinement stage of a two-layer framework for object segmentation. This scheme allows our method to consistently capture structural patterns about object perceptions throughout the whole framework. Our method can be operated on superpixels instead of raw pixels to reduce the number of graph nodes by two orders of magnitude. Moreover, we propose to partially explore the multi-object localization problem with inter-occlusion by weighted bipartite graph matching. Comprehensive experiments on three benchmark datasets (i.e., SegTrack, MOViCS, and GaTech) demonstrate the effectiveness of our approach compared with extensive state-of-the-art methods.
BibTeX
@article{guo2017motion,
  author = {Chun-Chao Guo and Jianhuang Lai and Xiaohua Xie},
  title = {{Motion-Appearance Interactive Encoding for Object Segmentation in Unconstrained Videos}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2017},
  doi = {10.1109/tcsvt.2019.2908779},
}
Figure