Learning to Discriminate Information for Online Action Detection: Analysis and Application

Sumin Lee, Hyunjun Eun, Jinyoung Moon, Seokeon Choi, Yoonhyung Kim, Chanho Jung, Changick Kim

In the world of computer vision, "online action detection" is the equivalent of a human watching a live security feed or an autonomous car scanning the road: you need to know what is happening right now, without the luxury of seeing how the scene ends.

Traditional approaches use Recurrent Neural Networks (RNNs) to track temporal patterns. However, these models have a "hoarding" problem—they tend to accumulate all information from the past, including irrelevant background noise and unrelated actions. A new paper, "Learning to Discriminate Information for Online Action Detection: Analysis and Application," introduces a clever architectural fix to help AI focus on what actually matters.

TL;DR

The authors propose the Information Discrimination Unit (IDU), a modified recurrent unit that uses the current frame as a reference to filter out irrelevant past information. By combining this with a future-gazing Information Integration Unit (IIU), they achieved state-of-the-art results in both detecting ongoing actions and anticipating future ones on major benchmarks like TVSeries and THUMOS-14.

What Problem Does This Paper Address?

Most action detection research happens "offline," where an algorithm analyzes a completed video clip from start to finish. Online action detection (OAD) is much harder because the model must make a decision at every single frame.

The core challenge is information relevancy. A streaming video is full of "background" (someone sitting still) or "irrelevant actions" (someone walking past the main actor). Standard RNNs, like GRUs or LSTMs, are designed to remember the past, but they aren't very good at deciding which parts of the past are relevant to the current moment. This leads to "diluted" feature representations where the signal of the actual action is lost in the noise of the background.

Methods

The researchers introduced two novel recurrent units to tackle detection and anticipation:

1. The Information Discrimination Unit (IDU)

The IDU is an evolution of the Gated Recurrent Unit (GRU). While a standard GRU updates its internal state based on the previous state and the current input, the IDU adds a "current information" mechanism.

Fig. 1. Comparison between GRU [10] and the proposed Information Discrimination Unit (IDU) for online action detection. Our IDU extends GRU with two novel components, a mechanism utilizing current infor- mation (blue lines) and an early embedding module (red dash boxes). First, reset and update modules in our IDU additionally take the current information (i.e., x0) to consider whether the past information (i.e, ht−1 and xt) are relevant to an ongoing action such as x0. Second, the early embedding module is introduced to consider the relation between high- level features for both information.

As shown in Figure 1, the IDU uses the features of the current time step ($x_0$) to influence the reset and update gates. Essentially, the model asks: "Is what I saw 2 seconds ago relevant to what I'm seeing right now?" If the answer is no, the gates drop that information. They also added an Early Embedding Module that uses contrastive loss to ensure the features extracted are highly discriminative for specific action classes.

2. The Information Integration Unit (IIU)

For the task of Action Anticipation (predicting what happens next), the authors developed the IIU. Instead of just looking at raw video frames, the IIU integrates two streams:

  • Visual features from the video.
  • Pseudo action labels generated by the IDU.

By using the IDU's high-confidence detections as a secondary input, the IIU gains a "semantic" understanding of the scene, making it much more effective at forecasting future actions.

Datasets

The models were evaluated on two cornerstone datasets for action recognition:

  • TVSeries: A challenging dataset featuring 27 untrimmed videos from popular TV shows, containing 30 realistic, everyday actions with heavy occlusions and complex backgrounds.
  • THUMOS-14: A dataset focused on 20 sports-related actions (e.g., diving, high jump) across over 400 untrimmed videos.

Key Insights and Ideas

  • Active vs. Passive Memory: The "aha moment" here is that recurrent units shouldn't just be passive sponges. By giving the gates a reference point (the current frame), they become active filters.
  • The Power of Pseudo-Labels: For anticipation, knowing what is happening now is just as important as seeing how it looks. Using the output of the detection network as a "pseudo-label" for the anticipation network significantly boosts performance compared to using visual features alone.
  • Efficiency Matters: Despite the extra complexity, the IDU is actually more parameter-efficient than a standard GRU in certain configurations because the early embedding module reduces the dimensionality of the inputs before they hit the heavy weight matrices of the gates.

Key Results

The proposed Information Discrimination Network (IDN) set new records:

  • TVSeries: Achieved 86.1% mcAP, outperforming the previous best (TRN) by 2.4%.
  • THUMOS-14: Achieved 60.3% mAP, a massive 13.1% jump over previous state-of-the-art methods.

In action anticipation, the Information Integration Network (IIN) outperformed existing methods at almost all time horizons. For example, when predicting an action 2 seconds before it happens on THUMOS-14, the IIN achieved 37.1% mAP, significantly higher than its competitors.

Limitations and Future Work

While the model is excellent at identifying when an action starts, the authors noted that anticipating the exact end time of future actions remains a significant challenge. Additionally, because the anticipation network (IIN) relies on the detection network (IDN) for pseudo-labels, the total inference time is roughly doubled (196.5ms per chunk), which might be a bottleneck for ultra-low-latency robotics applications.

Future research could explore more efficient ways to generate these semantic cues or apply the IDU architecture to other streaming data tasks beyond video, such as financial forecasting or sensor fusion.

Summary updated

Abstract

Online action detection, which aims to identify an ongoing action from a streaming video, is an important subject in real-world applications. For this task, previous methods use recurrent neural networks for modeling temporal relations in an input sequence. However, these methods overlook the fact that the input image sequence includes not only the action of interest but background and irrelevant actions. This would induce recurrent units to accumulate unnecessary information for encoding features on the action of interest. To overcome this problem, we propose a novel recurrent unit, named Information Discrimination Unit (IDU), which explicitly discriminates the information relevancy between an ongoing action and others to decide whether to accumulate the input information. This enables learning more discriminative representations for identifying an ongoing action. In this paper, we further present a new recurrent unit, called Information Integration Unit (IIU), for action anticipation. Our IIU exploits the outputs from IDU as pseudo action labels as well as RGB frames to learn enriched features of observed actions effectively. In experiments on TVSeries and THUMOS-14, the proposed methods outperform state-of-the-art methods by a significant margin in online action detection and action anticipation. Moreover, we demonstrate the effectiveness of the proposed units by conducting comprehensive ablation studies.
BibTeX
@article{lee2023learning,
  author = {Sumin Lee and Hyunjun Eun and Jinyoung Moon and Seokeon Choi and Yoonhyung Kim and Chanho Jung and Changick Kim},
  title = {{Learning to Discriminate Information for Online Action Detection: Analysis and Application}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3204808},
}
Figure