Information Elevation Network for Fast Online Action Detection

Sunah Min, Jinyoung MoonarXiv 2021 · arXiv

TL;DR

Researchers have developed the Information Elevation Network (IEN), a high-speed architecture for detecting actions in streaming video as they happen. By introducing a novel "Information Elevation Unit" (IEU), the model selectively "lifts" relevant past information to the present, significantly outperforming previous methods while remaining efficient enough for real-world deployment by using only RGB frames.

The Challenge of Online Action Detection

Online Action Detection (OAD) is the task of identifying an action in a video stream at the very moment it occurs. Unlike "offline" detection, where a model can look at the entire video at once, OAD models only see the past and the current frame.

The industry standard for processing such time-series data is the Long Short-Term Memory (LSTM) network. However, standard LSTMs suffer from a critical flaw in OAD: the forget gate. This gate decides what information to discard based only on the previous state and the immediate input ($x_t$). Because it doesn't "know" what the current ongoing action is, it often accidentally discards past information that would have been vital for identifying the current action as it unfolds.

Furthermore, many high-performing models rely on "optical flow" (calculating pixel movement), which is computationally expensive and prevents true real-time performance on standard hardware.

Methodology: The Information Elevation Unit (IEU)

The core innovation of this paper is the Information Elevation Unit (IEU). The IEU is a modified LSTM cell designed to maintain a "goal-oriented" memory.

Comparison of the Original LSTM and the proposed Information Elevation Unit (IEU).

As shown in the figure above, the IEU introduces a new elevation gate ($e_t$). This gate takes the current frame's information ($x_0$) as an "anchor" input at every timestep. By feeding the current information back into the processing of past chunks, the model can determine which parts of the past hidden state are relevant to the current action and "elevate" them directly into the cell state. This effectively reinforces forgotten information that is suddenly relevant again.

To ensure the model is fast enough for practical use (like CCTV monitoring), the IEN discards optical flow entirely. Instead, it uses a Temporal Shift Module (TSM) as a feature extractor, which captures motion information directly from RGB frames with minimal overhead.

System Architecture

The IEN processes video chunks sequentially. Each chunk is passed through a feature extractor and then into a series of IEU blocks. The architecture ensures that the current frame ($x_0$) is routed specifically to the elevation, input, and output gates, while keeping the forget gate identical to a standard LSTM to maintain stability.

The overall architecture of the Information Elevation Network (IEN).

Key Results and Performance

The model was evaluated on two gold-standard OAD benchmarks: THUMOS-14 (sports actions) and TVSeries (realistic, complex environments).

  • State-of-the-Art Accuracy: On THUMOS-14, the IEN achieved a mean Average Precision (mAP) of 60.4%. Remarkably, this RGB-only model outperformed existing "two-stream" models that use both RGB and expensive optical flow (which peaked at 55.7%). On TVSeries, it achieved a mean Calibrated Average Precision (mcAP) of 81.4%.
  • Real-Time Speed: The feature extraction process is incredibly fast, reaching 266 FPS for RGB frames on a Titan XP GPU. In contrast, calculating optical flow typically crawls at around 12 FPS, making it the primary bottleneck in older systems.
  • Efficiency Trade-off: Adopting the IEU instead of a standard LSTM resulted in a 5.78% performance boost with only a modest 12.5% decrease in processing speed—a trade-off that is highly beneficial for accuracy-critical applications.

Limitations and Future Work

While the IEN performs exceptionally well on distinct physical actions, it still struggles with subtle gestures (like "cricket shot" or "point") and actions that occur extremely quickly. Future work may involve refining the feature extractor to better capture these subtle visual cues without sacrificing the real-time speed that makes the IEN practical for real-world surveillance and monitoring.

Summary updated

Abstract

Online action detection (OAD) is a task that receives video segments within a streaming video as inputs and identifies ongoing actions within them. It is important to retain past information associated with a current action. However, long short-term memory (LSTM), a popular recurrent unit for modeling temporal information from videos, accumulates past information from the previous hidden and cell states and the extracted visual features at each timestep without considering the relationships between the past and current information. Consequently, the forget gate of the original LSTM can lose the accumulated information relevant to the current action because it determines which information to forget without considering the current action. We introduce a novel information elevation unit (IEU) that lifts up and accumulate the past information relevant to the current action in order to model the past information that is especially relevant to the current action. To the best of our knowledge, our IEN is the first attempt that considers the computational overhead for the practical use of OAD. Through ablation studies, we design an efficient and effective OAD network using IEUs, called an information elevation network (IEN). Our IEN uses visual features extracted by a fast action recognition network taking only RGB frames because extracting optical flows requires heavy computation overhead. On two OAD benchmark datasets, THUMOS-14 and TVSeries, our IEN outperforms state-of-the-art OAD methods using only RGB frames. Furthermore, on the THUMOS-14 dataset, our IEN outperforms the state-of-the-art OAD methods using two-stream features based on RGB frames and optical flows.
BibTeX
@misc{min2021information,
  author = {Sunah Min and Jinyoung Moon},
  title = {{Information Elevation Network for Fast Online Action Detection}},
  year = {2021},
  eprint = {2109.13572},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.13572},
}
Figure