FineTea: A Novel Fine-Grained Action Recognition Video Dataset for Tea Ceremony Actions

Changwei Ouyang, Yun Yi, Hanli Wang, Jin Zhou, Tao TianJ. Imaging 2024 · DOI

TL;DR

This paper introduces FineTea, a novel dataset for fine-grained action recognition specifically targeting tea ceremony actions, which are characterized by subtle differences and strong temporal dependencies. To address the challenges of this domain, they propose TSM-ConvNeXt, a method that combines the powerful ConvNeXt backbone with a Temporal Shift Module (TSM) to effectively capture spatiotemporal information, achieving state-of-the-art results on their new dataset and the Diving48 benchmark.

What Problem Does This Paper Address?

Recognizing human actions in videos is a crucial task in computer vision with applications ranging from surveillance to human-computer interaction. While deep learning has made significant strides, existing action recognition datasets often focus on coarse-grained actions (e.g., "playing basketball"). This creates a gap for tasks requiring fine-grained analysis, where subtle differences in movements are critical. Tea ceremonies, with their elegant and nuanced actions, present a prime example of such a domain. The subtle variations between different tea ceremony steps, coupled with their strong temporal dependencies, make them particularly challenging for current models. Furthermore, a lack of specialized datasets for this domain hinders research progress.

Methods

The core of the proposed solution is the TSM-ConvNeXt model. It builds upon the ConvNeXt architecture, a recent convolutional neural network that adopts a Transformer-like design with depth-wise convolutions and improved regularization. To enhance temporal understanding, the Temporal Shift Module (TSM) is integrated. TSM works by shifting a portion of feature channels across adjacent frames, introducing temporal variations that help the model capture the dynamic nature of actions without requiring expensive 3D convolutions. The TSM block is strategically placed within the ConvNeXt block, before the depth-wise convolution.

Datasets

The primary contribution is the introduction of the FineTea dataset. This dataset comprises 2745 video clips of tea ceremony actions, meticulously annotated with 9 basic action classes and 31 fine-grained subclasses. The dataset is designed to be challenging due to the high similarity between actions and the need for precise temporal modeling. To ensure robust evaluation, a unique non-repeating appearance split strategy is employed, creating two sets (split 1 and split 2) for two-fold cross-validation.

The paper also uses the Diving48 dataset for comparative evaluation, a well-established benchmark for fine-grained action recognition.

Key Insights and Ideas

  • The subtlety of tea ceremony actions: The paper highlights that the differences between many tea ceremony actions are extremely subtle, often residing in small hand gestures or utensil manipulations, making them difficult to distinguish.
  • Temporal dependencies are paramount: The sequence, duration, and transitions between actions are as important as the individual movements themselves, posing a significant challenge for static frame-based recognition.
  • ConvNeXt as a strong backbone: The adoption of ConvNeXt as the backbone demonstrates its effectiveness in capturing spatial features, which, when combined with TSM, leads to superior performance.
  • TSM's efficiency for temporal modeling: The TSM module proves to be an effective and computationally efficient way to incorporate temporal information into 2D CNNs, outperforming baseline methods that rely solely on spatial features.
  • Green tea actions are easier: Analysis of the confusion matrices suggests that green tea ceremony actions are generally easier to recognize, likely due to more distinct movements compared to black or oolong tea preparations.

Key Results

  • The proposed TSM-ConvNeXt model achieved a 7.31% improvement over a baseline method using ResNet50 with TSM on the FineTea dataset.
  • On the FineTea dataset, TSM-ConvNeXt (sampling 16 frames) achieved the best experimental results among the evaluated state-of-the-art methods, reaching an average accuracy of 71.41%.
  • On the Diving48 dataset, TSM-ConvNeXt achieved an accuracy of 85.5%, outperforming other CNN-based and Transformer-based methods.
  • The TSM-ConvNeXt method demonstrated a good balance between training/testing time and accuracy, with TSM-ConvNeXt16 achieving higher accuracy than AIM despite longer processing times.

Limitations and Future Work

  • Dataset expansion: The authors plan to collect more videos of fine-grained tea ceremony actions to further expand the dataset.
  • Lightweight modules: Future research will focus on designing more lightweight fine-grained modules to improve the method's performance and efficiency.
  • Generalization: While the dataset split strategy aims to mitigate bias, further exploration into generalization across different environments or styles of tea ceremonies could be beneficial.
  • Hyperparameter sensitivity: The shift proportion parameter in TSM was found to be critical, with an optimal value of 1/4 identified. Further analysis into the sensitivity and robustness of this parameter across different datasets could be valuable.

Summary updated

BibTeX
@article{ouyang2024finetea,
  author = {Changwei Ouyang and Yun Yi and Hanli Wang and Jin Zhou and Tao Tian},
  title = {{FineTea: A Novel Fine-Grained Action Recognition Video Dataset for Tea Ceremony Actions}},
  journal = {J. Imaging},
  year = {2024},
  doi = {10.3390/jimaging10090216},
}
Figure