Abstract
Professional visual effects (VFX) production frequently requires the segmentation of partial objects or specific regions, a process known as rotoscoping. While Video Object Segmentation (VOS) has advanced significantly, existing models often struggle with non-semantic boundaries and long-term consistency. We present XMem++, a framework designed for high-precision, low-effort VOS in production environments. By incorporating a permanent memory module for user annotations and an automated frame-suggestion algorithm, XMem++ achieves state-of-the-art performance with up to 5x fewer manual annotations than previous methods.
Introduction
In the VFX industry, rotoscoping—the manual creation of masks frame-by-frame—remains a significant bottleneck. Standard VOS models are typically trained to segment entire semantic objects, such as a whole person or vehicle. However, professional workflows often demand the segmentation of "partial regions," such as the half of a face for digital makeup or a specific tattoo. These tasks are challenging because they lack clear semantic boundaries and are prone to "quality jumps" or temporal flickering when new annotations are introduced.
XMem++ addresses these challenges by bridging the gap between research-grade VOS and the high-precision needs of film production, focusing on long-term stability and minimal user intervention.
Methods
XMem++ builds upon the XMem architecture, introducing three key innovations to enhance performance and efficiency:
Permanent Memory Module
Unlike standard models that compress or discard older reference frames to manage VRAM, XMem++ introduces a dedicated permanent memory buffer. User-provided annotations are stored here in an uncompressed format as "ground-truth references." This ensures that the model never "forgets" the user's specific intent, maintaining consistency even in long video sequences and preventing the drift common in purely predictive models.
Attention-Based Frame Suggestion
To minimize manual labor, XMem++ includes an iterative algorithm that suggests the "next best frame" for annotation. By calculating the dissimilarity between existing annotations and the rest of the video through a "cycle consistency" check, the system identifies frames with the most diverse poses or lighting conditions. This automated selection is significantly faster and more effective than manual searching by a human expert.
In-Memory Augmentations
To maximize the utility of each annotated frame, XMem++ automatically generates 11 augmented versions (varying brightness, scale, rotation, etc.) and stores them in memory. This process effectively expands the training signal, teaching the model to recognize the target region under various environmental conditions without requiring additional user input.
Evaluation and Results
The framework was evaluated on standard benchmarks (LVOS, MOSE) and a new dataset, PUMaVOS (Partial and Unusual MAsk VOS), which contains 24 challenging clips featuring partial masks and unusual targets like reflections.
Key findings include:
- Efficiency: XMem++ achieved higher accuracy with 10 annotated frames than the baseline XMem did with 50, representing a 5x improvement in annotation efficiency.
- Performance: On the LVOS benchmark, XMem++ outperformed state-of-the-art models while using half the number of annotated frames.
- Real-time Capability: The system operates at approximately 32 FPS on an RTX 3090, supporting interactive studio workflows.
Discussion and Conclusion
A core insight of XMem++ is the prioritization of user intent. By separating permanent user memory from temporary model predictions, the system avoids the "semantic bias" of models that attempt to snap to whole objects. Instead, it focuses on the specific pixels defined by the user.
While XMem++ significantly advances the state of the art, limitations remain in handling extreme motion blur, identical distractors in close proximity, and extremely fine high-frequency details like hair. Future work may explore dense scene correspondences and on-the-fly generative data augmentation to further bolster robustness. XMem++ provides a robust solution for production-level video object segmentation, reducing the manual burden of rotoscoping while delivering high-precision results.
Summary updated