Two-Streams: Dark and Light Networks with Graph Convolution for Action Recognition from Dark Videos (Student Abstract)

Saurabh Suman, Nilay Naharas, Badri Narayan Subudhi, Vinit JakhetiyaAAAI 2023 · DOI

Abstract

Action recognition in low-light environments is a critical challenge for computer vision applications such as autonomous robotics and night-time surveillance. Traditional models often fail due to low contrast and significant sensor noise. This paper presents a two-stream architecture that processes raw dark video alongside an enhanced light stream generated by a lightweight Self-Calibrated Illumination (SCI) module. By utilizing a weight-shared R(2+1)D backbone and a Graph Convolutional Network (GCN) for temporal feature fusion, the proposed method achieves state-of-the-art performance on the ARID dataset, reaching 95.86% Top-1 accuracy with improved parameter efficiency.

Introduction

Identifying human actions in video is a fundamental task in computer vision. However, most existing research assumes well-lit conditions, leaving a gap in performance for low-light scenarios. Dark videos suffer from reduced visibility, heavy noise, and a lack of distinct spatio-temporal features. Previous attempts to solve this often relied on computationally expensive image enhancement techniques that do not necessarily translate well to video action recognition. This work addresses these issues by proposing an efficient dual-stream approach that leverages both raw and enhanced data.

Proposed Methodology

The architecture consists of three primary components designed for efficiency and accuracy in dark environments.

Dual-Stream Processing

The network employs two parallel streams:

  1. Dark Stream: Processes raw, low-light video frames directly to preserve deep textural features that might be obscured or distorted by enhancement algorithms.
  2. Light Stream: Uses a Self-Calibrated Illumination (SCI) module to brighten frames. SCI is a cascaded, lightweight framework that provides illumination enhancement without the high computational overhead of traditional deep learning-based enhancers.

Weight-Shared Feature Extraction

Both streams utilize an R(2+1)D backbone for feature extraction. This architecture decomposes 3D convolutions into separate spatial and temporal components, which enhances learning capacity while reducing complexity. To minimize the model's footprint, the two streams share weights, allowing the network to learn a unified feature representation for both dark and enhanced inputs.

Temporal Graph Encoder

The extracted features from both streams are concatenated and fed into a three-layer Graph Convolutional Network (GCN). The GCN serves as a temporal graph encoder, modeling the relationships between frames over time. This step effectively "denoises" the temporal signal and selects the most salient spatio-temporal information for final classification.

Architecture of the proposed two-stream network featuring SCI and GCN fusion.

Experimental Evaluation

The model was benchmarked using the Action Recognition in Dark (ARID) dataset, which contains real-world low-light video sequences.

Performance Comparison

The proposed method was compared against 15 state-of-the-art (SOTA) models, including various 3D CNN architectures and previous dual-stream approaches.

Comparison of Top-1 and Top-5 accuracy across various models on the ARID dataset.

As shown in the results table, the proposed method achieves a Top-1 accuracy of 95.86%, significantly outperforming the previous best model, DarkLight-R(2+1)D-34, which reached 94.04%. Furthermore, the model achieved a Top-5 accuracy of 99.87%.

Efficiency Gains

Beyond accuracy, the proposed architecture is notably more efficient. By employing weight sharing and the R(2+1)D backbone, the model reduces the total parameter count by over one million compared to the DarkLight baseline. This makes it a more viable candidate for deployment on edge devices with limited computational resources.

Conclusion

This research demonstrates that combining raw and enhanced video streams through a GCN-based fusion mechanism provides a robust solution for action recognition in the dark. The synergy between the dark stream's texture preservation and the light stream's clarity, coupled with efficient temporal encoding, sets a new standard for the ARID benchmark. Future work will focus on hyperparameter optimization and improving robustness in near-zero light conditions.

Summary updated

BibTeX
@inproceedings{suman2023two,
  author = {Saurabh Suman and Nilay Naharas and Badri Narayan Subudhi and Vinit Jakhetiya},
  title = {{Two-Streams: Dark and Light Networks with Graph Convolution for Action Recognition from Dark Videos (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27030},
}
Figure