ARID: A New Dataset for Recognizing Action in the Dark

Yuecong Xu, Jianfei Yang, Haozhi Cao, Kezhi Mao, Jianxiong Yin, Simon SeearXiv 2020 · OpenReview

Abstract

The task of action recognition in dark videos is useful in various scenarios, e.g., night surveillance and self-driving at night. Though progress has been made in the action recognition task for videos in normal illumination, few have studied action recognition in the dark. This is partly due to the lack of sufficient datasets for such a task. In this paper, we explored the task of action recognition in dark videos. We bridge the gap of the lack of data for this task by collecting a new dataset: the Action Recognition in the Dark (ARID) dataset. It consists of over 3,780 video clips with 11 action categories. To the best of our knowledge, it is the first dataset focused on human actions in dark videos. To gain further understandings of our ARID dataset, we analyze the ARID dataset in detail and exhibited its necessity over synthetic dark videos. Additionally, we benchmarked the performance of several current action recognition models on our dataset and explored potential methods for increasing their performances. Our results show that current action recognition models and frame enhancement methods may not be effective solutions for the task of action recognition in dark videos.

Introduction

Action recognition has advanced significantly with datasets like Kinetics and UCF101, which primarily feature well-lit, high-contrast "web videos." However, real-world applications such as night-time autonomous driving and security surveillance must operate in low-light conditions. While specialized hardware like infrared sensors exists, it is often cost-prohibitive for large-scale deployment. This research introduces ARID (Action Recognition In the Dark), the first dataset specifically designed to address the lack of high-quality, real-world dark video data.

The ARID Dataset

The core contribution of this work is the ARID dataset, which contains over 3,780 clips across 11 action categories, including jumping, pouring, sitting, and waving. These clips were filmed in diverse indoor and outdoor settings using standard commercial cameras during night hours, without artificial lighting on the actors.

Sample frames from the ARID dataset illustrating various human actions in low-light environments.

The dataset is split into training and testing sets to facilitate robust model evaluation. The distribution of clips across categories is shown below, highlighting the balance and scale of the data provided for researchers.

Distribution of training and testing clips across the eleven action categories in the ARID dataset.

Methodology and Benchmarking

The researchers evaluated several State-of-the-Art (SOTA) architectures on ARID, including:

  • 3D-CNNs: Models like ResNet-101 and ResNext-101 that process video as a 3D volume to capture temporal dynamics.
  • Two-Stream Networks: Architectures that analyze both raw frames (spatial) and motion/optical flow (temporal).

Additionally, they tested five frame enhancement techniques, such as Histogram Equalization (HE) and deep-learning-based methods like KinD, to determine if pre-processing "brightening" improves recognition accuracy.

Key Insights

Real vs. Synthetic Darkness

A critical finding of the study is that real dark videos possess unique statistical properties—such as specific RGB distributions and sensor noise—that are not captured by mathematically darkening "daylight" videos. Models trained on synthetic dark data (like a darkened version of HMDB51) fail to generalize to the messy reality of the ARID dataset.

The Background Focus Problem

Using Class Activation Maps (CAMs), the authors observed that models often ignore the dark silhouette of the actor and instead focus on the background. The network identifies actions by noticing how the background is obscured or revealed by movement, rather than tracking the actor directly. This suggests that current models rely on different visual cues in the dark than they do in daylight.

The Role of Temporal Cues

In low-light conditions where spatial details are murky, temporal cues become vital. Incorporating optical flow significantly boosted performance, suggesting that motion is the primary signal when visual clarity is low.

Results and Conclusion

The 3D-ResNext-101 model emerged as the top performer, achieving a 74.73% Top-1 accuracy. Interestingly, frame enhancement proved to be a double-edged sword; while it helps human visibility, it can introduce artifacts that decrease model accuracy. Gamma Intensity Correction (GIC) provided the most consistent improvement, though it was modest (approx. 3.75%).

While ARID is a pioneering step, future work will focus on expanding the dataset's scale and developing specialized architectures inherently robust to low-contrast environments, reducing the reliance on problematic pre-processing steps.

Summary updated

Abstract

The task of action recognition in dark videos is useful in various scenarios, e.g., night surveillance and self-driving at night. Though progress has been made in the action recognition task for videos in normal illumination, few have studied action recognition in the dark. This is partly due to the lack of sufficient datasets for such a task. In this paper, we explored the task of action recognition in dark videos. We bridge the gap of the lack of data for this task by collecting a new dataset: the Action Recognition in the Dark (ARID) dataset. It consists of over 3,780 video clips with 11 action categories. To the best of our knowledge, it is the first dataset focused on human actions in dark videos. To gain further understandings of our ARID dataset, we analyze the ARID dataset in detail and exhibited its necessity over synthetic dark videos. Additionally, we benchmarked the performance of several current action recognition models on our dataset and explored potential methods for increasing their performances. Our results show that current action recognition models and frame enhancement methods may not be effective solutions for the task of action recognition in dark videos.
BibTeX
@misc{xu2020arid,
  author = {Yuecong Xu and Jianfei Yang and Haozhi Cao and Kezhi Mao and Jianxiong Yin and Simon See},
  title = {{ARID: A New Dataset for Recognizing Action in the Dark}},
  year = {2020},
  eprint = {2006.03876},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.03876},
}
Figure