Universal-to-Specific Framework for Complex Action Recognition

Peisen Zhao, Lingxi Xie, Ya Zhang, Qi Tian

Abstract

Video-based action recognition has recently attracted much attention in the field of computer vision. To solve more complex recognition tasks, it has become necessary to distinguish different levels of interclass variations. Inspired by a common flowchart based on the human decision-making process that first narrows down the probable classes and then applies a "rethinking" process for finer-level recognition, we propose an effective universal-to-specific (U2S) framework for complex action recognition. The U2S framework is composed of three subnetworks: a universal network, a category-specific network, and a mask network. The universal network first learns universal feature representations. The mask network then generates attention masks for confusing classes through category regularization based on the output of the universal network. The mask is further used to guide the category-specific network for class-specific feature representations. The entire framework is optimized in an end-to-end manner. Experiments on a variety of benchmark datasets, e.g., the Something-Something, UCF101, and HMDB51 datasets, demonstrate the effectiveness of the U2S framework; i.e., U2S can focus on discriminative spatiotemporal regions for confusing categories. We further visualize the relationship between different classes, showing that U2S indeed improves the discriminability of learned features. Moreover, the proposed U2S model is a general framework and may adopt any base recognition network.

Introduction

Most deep learning models for action recognition are "one-pass" systems that extract features and classify actions in a single flow. While effective for distinct actions like "jumping" versus "running," these models often struggle with fine-grained interclass variations. In datasets like Something-Something, categories often share significant visual features—for instance, "bending something so it deforms" versus "bending something until it breaks."

Standard models fail here because they attempt to learn a "universal" feature representation that lacks the sensitivity to detect the subtle, decisive moments that distinguish these classes. Analysis shows a clear correlation: as the similarity between a category and others increases, the recognition accuracy typically decreases. This motivates the need for a "rethinking" mechanism that can narrow down possibilities and then focus specifically on the features that resolve ambiguity.

Figure 1: Analysis of category similarity versus recognition accuracy, showing that higher interclass similarity leads to lower performance.

Methodology

The U2S framework mimics the human decision-making process through three interconnected subnetworks:

  1. Universal Network (UN): Utilizing a ResNet-50 backbone with mixed 2D and 3D convolutions, the UN extracts general spatiotemporal features and provides an initial set of class predictions (logits).
  2. Mask Network (MN): This subnetwork identifies confusing classes using a Category Similarity Matrix (CSM), which is calculated based on the sparsity patterns of feature activations. If two actions activate similar neural channels, they are deemed confusing. The MN then generates a spatiotemporal "attention mask" highlighting regions likely to contain discriminative information for those specific classes.
  3. Category-Specific Network (CSN): The CSN shares 2D convolution parameters with the UN but maintains independent 3D parameters. It applies the masks from the MN to the video features, forcing the model to "rethink" the classification by focusing on the most relevant, fine-grained details.
Figure 2: The Universal-to-Specific (U2S) framework architecture, illustrating the flow from universal feature extraction to category-specific masking.

The entire system is trained end-to-end, utilizing a combined loss function that optimizes both the general accuracy of the UN and the specialized accuracy of the CSN. This collaborative learning approach allows the "rethinking" process to provide gradients that help the base model learn more discriminative features overall.

Experimental Results

The U2S framework was evaluated on several major benchmarks, demonstrating significant improvements in fine-grained recognition:

  • Something-Something V2: The model achieved a 58.47% Top-1 accuracy, a nearly 2% improvement over the strong one-pass baseline. The most dramatic gains occurred in confusing pairs; for example, accuracy for "Pouring something out of something" increased by 12.66%.
  • Generalization: Beyond video, the model improved Top-1 accuracy on the Stanford Dogs image dataset by 6.09%, proving the universal-to-specific logic applies to static fine-grained classification as well.
  • State-of-the-Art: When combined with Temporal Shift Modules (TSM), U2S outperformed many existing complex models while relying solely on RGB data, avoiding the computational expense of optical flow.

Conclusion

The U2S framework provides a robust method for addressing fine-grained variations in action recognition by incorporating a targeted rethinking mechanism. While the dual-branch architecture introduces some computational overhead, the gains in discriminative power are substantial. Future work will focus on improving the initial universal predictions, as "Oracle" tests (using ground-truth labels for masking) suggest a potential performance ceiling as high as 63.54%.

Summary updated

Abstract

Video-based action recognition has recently attracted much attention in the field of computer vision. To solve more complex recognition tasks, it has become necessary to distinguish different levels of interclass variations. Inspired by a common flowchart based on the human decision-making process that first narrows down the probable classes and then applies a "rethinking" process for finer-level recognition, we propose an effective universal-to-specific (U2S) framework for complex action recognition. The U2S framework is composed of three subnetworks: a universal network, a category-specific network, and a mask network. The universal network first learns universal feature representations. The mask network then generates attention masks for confusing classes through category regularization based on the output of the universal network. The mask is further used to guide the category-specific network for class-specific feature representations. The entire framework is optimized in an end-to-end manner. Experiments on a variety of benchmark datasets, e.g., the Something-Something, UCF101, and HMDB51 datasets, demonstrate the effectiveness of the U2S framework; i.e., U2S can focus on discriminative spatiotemporal regions for confusing categories. We further visualize the relationship between different classes, showing that U2S indeed improves the discriminability of learned features. Moreover, the proposed U2S model is a general framework and may adopt any base recognition network.
BibTeX
@article{zhao2021universal,
  author = {Peisen Zhao and Lingxi Xie and Ya Zhang and Qi Tian},
  title = {{Universal-to-Specific Framework for Complex Action Recognition}},
  journal = {IEEE Trans. Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=Fb58xWjkLY},
}
Figure