Abstract
Video semantic segmentation is a critical yet computationally expensive task for real-time applications like autonomous driving. Traditional methods often suffer from unbalanced latency or inaccuracies due to optical flow warping. This paper introduces TDNet (Temporally Distributed Network), a novel framework that distributes the depth of a neural network across sequential video frames. By executing shallower sub-networks on each frame and aggregating features via an Attention Propagation Module, TDNet achieves high accuracy with significantly lower latency and a perfectly balanced computational load.
The Challenge: Real-Time Video Segmentation
Semantic segmentation requires assigning a category label to every pixel, a process that is inherently slow for high-resolution video. The "naive" approach of processing every frame independently is too slow for real-time use. Existing "keyframe" strategies attempt to mitigate this by running a heavy model on select frames and warping features to others using optical flow. However, this leads to:
- Unbalanced Latency: Computational spikes at keyframes make real-time guarantees difficult.
- Warping Errors: Optical flow is computationally heavy and prone to failure during occlusions or fast motion.
The TDNet Framework
TDNet addresses these issues through three core innovations:
1. Distributed Sub-networks
Instead of running one deep model at every time step, TDNet decomposes a deep network into several shallower sub-networks. At each frame $T$, only one sub-network is executed. The final feature representation is a composite of the current frame's output and cached outputs from previous frames. This allows the system to maintain the representational power of a deep network while only paying the per-frame cost of a shallow one.
2. Attention Propagation Module (APM)
To account for object motion between frames, TDNet uses an APM to align features. Unlike optical flow, which relies on local pixel movement, the APM uses a non-local attention mechanism to globally correlate features. Efficiency is maintained through Attention Downsampling (processing at lower resolutions) and Attention Propagation (chaining alignment across frames).
3. Grouped Knowledge Distillation (GKD)
To ensure sub-networks learn complementary features, the authors employ GKD. A large, pre-trained "teacher" model (e.g., PSPNet-101) guides the training of the "student" TDNet, penalizing it if the aggregated or individual sub-network outputs deviate from the teacher's high-quality features.
Experimental Results
The researchers evaluated TDNet on the Cityscapes, CamVid, and NYUD-v2 datasets. A key finding was that TDNet provides a superior trade-off between speed and accuracy compared to existing state-of-the-art methods.
- Efficiency: On Cityscapes,
TD2-PSP50achieves 79.9% mIoU at 178ms/f, nearly twice as fast as the standard PSPNet-101 (360ms/f) with comparable accuracy. - Real-time Speed: The
TD4-Bise18variant runs at 48 FPS (21ms/f), outperforming dedicated real-time models like ICNet. - Predictable Performance: TDNet eliminates "lag spikes" because the computational load is identical for every frame.
Key Insights
- Temporal Depth: The core "Aha!" moment is that network depth can be distributed across time rather than just space.
- Robustness: The attention-based alignment is more resilient to scene changes and occlusions than traditional optical flow.
- Resource Balance: Predictable per-frame latency is a major advantage for hardware deployment in robotics and autonomous systems.
Limitations and Future Work
While highly effective, TDNet has a few trade-offs:
- Sub-network Depth: Increasing the number of sub-networks ($N$) makes each one shallower. If $N$ is too large, the sub-networks may fail to extract sufficiently complex features.
- Memory Usage: Caching intermediate feature maps from previous frames increases GPU memory overhead.
- Expansion: Future research could apply this temporally distributed logic to other tasks like video object detection or instance segmentation.
Summary updated