Abstract
Nowadays, video conference solutions are widely adopted for companies, education, and government. People segmentation is crucial for supporting virtual background, an essential video conference function to protect users' privacy. This paper demonstrated a people segmentation framework called CE-PeopleSeg, which employed an efficient segmentation method, structural pruning, and dynamic frame skipping techniques, leading to a fast inference speed on CPU. Our extensive experiments show that the proposed CE-PeopleSeg can achieve a high prediction mIoU of 87.9% on Supervised People Dataset while reaching a real-time inference speed of 32.40 fps on CPU with very low usage of 10%.
Introduction
Virtual backgrounds have become a staple of modern video conferencing, providing essential privacy for users working from home. However, the underlying technology—people segmentation—is computationally expensive. While state-of-the-art models can run in real-time on powerful GPUs, many remote workers and students rely on laptops or desktop PCs without dedicated graphics cards. Existing CPU-based solutions often consume so much processing power that the computer slows down, making it difficult to run meeting software or share a screen simultaneously. CE-PeopleSeg is designed to be a "CPU-efficient" framework that maintains high accuracy without hogging the processor, achieving real-time performance on standard mid-range hardware.
Methodology
The CE-PeopleSeg framework achieves its efficiency through a three-pronged approach: an optimized architecture, extended structural pruning, and a self-adaptive frame-skipping mechanism.
Efficient Architecture
The model utilizes a Feature Pyramid Network (FPN) with a MobileNetV2 backbone. While popular segmentation models like DeepLab often rely on "atrous" (dilated) convolutions, these are memory-intensive and less suited for CPU inference. In contrast, the FPN architecture efficiently merges low-level spatial details with high-level semantic information. The authors also identified a "sweet spot" for input resolution; while 320px provides peak accuracy, a 160px resolution offers a 4.5x speed boost with only a marginal 3.4% drop in mIoU.
Extended Structural Pruning
To further reduce the computational footprint, the team applied "network slimming." This technique uses the scaling factors in Batch Normalization (BN) layers as indicators of channel importance. Channels with low scaling factors are pruned. The authors specifically extended this to depthwise separable convolutions, the core building blocks of MobileNetV2. A key insight was the "BN2 Pruning Trick": by pruning based on the second BN layer in the bottleneck blocks, the model ensures that input and output channel counts remain consistent, avoiding the architectural conflicts that often plague automated pruning.
Dynamic Frame Skipping
Recognizing that participants in video calls are often relatively stationary, CE-PeopleSeg implements a motion-detection trigger. If the L1 norm difference between the current frame and the last processed frame is below a threshold, the system reuses the previous segmentation mask. To handle varying environments and camera distances, the threshold is self-adaptive, calculated based on the history of recent frame differences.
Experimental Results
The model was trained and evaluated on the Supervisely Person Dataset, consisting of 5,711 images with high-quality manual annotations.
On an Intel i5-8400 CPU (2.80GHz), CE-PeopleSeg achieved:
- Accuracy: A high prediction mIoU of 87.9%.
- Speed: A real-time inference speed of 32.40 FPS.
- Efficiency: Extremely low CPU usage of only 10%.
- Model Compression: Structural pruning reduced model parameters by approximately 44% (from 2.25M to 1.25M) with less than a 1% drop in accuracy.
Conclusion
CE-PeopleSeg demonstrates that high-quality, real-time people segmentation is achievable on consumer-grade CPUs with minimal resource consumption. By combining architectural optimization, targeted pruning, and temporal redundancy exploitation, the framework enables privacy features like virtual backgrounds without compromising system performance. Future work will focus on improving temporal consistency to reduce edge flickering and testing the model's robustness in more diverse lighting and background conditions.
Summary updated