Abstract
For the two-stream style methods in action recognition, fusing the two streams' predictions is always by the weighted averaging scheme. This fusion method with fixed weights lacks of pertinence to different action videos and always needs trial and error on the validation set. In order to enhance the adaptability of two-stream ConvNets and improve its performance, an end-to-end trainable gated fusion method, namely gating ConvNet, for the two-stream ConvNets is proposed in this paper based on the MoE (Mixture of Experts) theory. The gating ConvNet takes the combination of feature maps from the same layer of the spatial and the temporal nets as input and adopts ReLU (Rectified Linear Unit) as the gating output activation function. To reduce the over-fitting of gating ConvNet caused by the redundancy of parameters, a new multi-task learning method is designed, which jointly learns the gating fusion weights for the two streams and learns the gating ConvNet for action classification. With our gated fusion method and multi-task learning approach, a high accuracy of 94.5% is achieved on the dataset UCF101.
Introduction
In computer vision, "Two-Stream" networks are the standard for action recognition. One stream analyzes static RGB frames (spatial) to understand the scene's appearance, while the other analyzes optical flow (temporal) to capture motion. Traditionally, these streams are fused at the end by averaging their predictions—often with a fixed bias, such as weighting the temporal stream 1.5x more than the spatial stream.
However, this "one size fits all" approach is inherently flawed. Some actions are defined primarily by appearance (e.g., "playing cello"), while others are defined by specific motion patterns (e.g., "jumping"). A fixed weight cannot adapt to the specific content of a video or the varying quality of input frames. This paper addresses this limitation by introducing an adaptive, sample-specific fusion mechanism.
Proposed Method: Gated TSN
The core innovation of this work is the Gating ConvNet, integrated into a Temporal Segment Network (TSN) framework. Instead of manual weight tuning, the Gating ConvNet learns to predict the optimal fusion weights for each specific video clip.
Technical Components
- Feature-Level Input: The Gating ConvNet does not look at raw pixels. Instead, it takes intermediate feature maps from both the spatial and temporal streams (specifically from the
inception4elayer) as its input. This "middle-out" fusion provides a balance between raw detail and high-level semantic specialization. - ReLU Gating: While traditional Mixture of Experts (MoE) models often use Softmax, this model employs ReLU (Rectified Linear Unit). This allows the network to output non-negative weights that act as a "confidence ratio" without the strict sum-to-one constraint, facilitating faster training convergence.
- Multi-Task Learning: To prevent the gating network from overfitting—given it has millions of parameters but only outputs two scalar weights—the authors implemented a dual-branch training strategy. The model is trained to simultaneously predict the action class and the fusion weights. This forces the gating network to learn meaningful semantic features rather than memorizing noise.
Experimental Results
The model was evaluated on the UCF101 dataset, containing 13,320 video clips across 101 categories. The Gated TSN achieved a state-of-the-art 94.5% accuracy, outperforming the baseline TSN (94.0%) and even versions of TSN that utilized three modalities (94.2%).
Sample-Specific Trust
Qualitative analysis reveals that the Gating ConvNet successfully shifts trust between modalities based on video content. For instance, in videos where the background is cluttered but motion is clear, the network automatically increases the weight of the temporal stream. Conversely, if motion is ambiguous but objects are distinct, it relies more heavily on the spatial stream.
As shown in the figure above, for actions like "HeadMassage," traditional weighted averaging might incorrectly favor a motion-based prediction like "PlayingViolin." The gated fusion mechanism identifies the higher reliability of the spatial stream in this context, leading to a correct classification.
Conclusion
The Gating ConvNet provides a robust, adaptive solution for multi-stream fusion in action recognition. By replacing fixed weights with a learned, sample-specific confidence mechanism, the model achieves higher accuracy and better handles edge cases where one modality might be misleading. While the approach increases memory consumption during training due to the need to store dual-stream feature maps, its performance gains are significant. Future work may involve extending this adaptive gating technique to other domains, such as semantic segmentation, where multi-scale feature fusion is critical.
Summary updated