TL;DR
Researchers have developed the Information Elevation Network (IEN), a high-speed architecture for detecting actions in streaming video as they happen. By introducing a novel "Information Elevation Unit" (IEU), the model selectively "lifts" relevant past information to the present, significantly outperforming previous methods while remaining efficient enough for real-world deployment by using only RGB frames.
The Challenge of Online Action Detection
Online Action Detection (OAD) is the task of identifying an action in a video stream at the very moment it occurs. Unlike "offline" detection, where a model can look at the entire video at once, OAD models only see the past and the current frame.
The industry standard for processing such time-series data is the Long Short-Term Memory (LSTM) network. However, standard LSTMs suffer from a critical flaw in OAD: the forget gate. This gate decides what information to discard based only on the previous state and the immediate input ($x_t$). Because it doesn't "know" what the current ongoing action is, it often accidentally discards past information that would have been vital for identifying the current action as it unfolds.
Furthermore, many high-performing models rely on "optical flow" (calculating pixel movement), which is computationally expensive and prevents true real-time performance on standard hardware.
Methodology: The Information Elevation Unit (IEU)
The core innovation of this paper is the Information Elevation Unit (IEU). The IEU is a modified LSTM cell designed to maintain a "goal-oriented" memory.
As shown in the figure above, the IEU introduces a new elevation gate ($e_t$). This gate takes the current frame's information ($x_0$) as an "anchor" input at every timestep. By feeding the current information back into the processing of past chunks, the model can determine which parts of the past hidden state are relevant to the current action and "elevate" them directly into the cell state. This effectively reinforces forgotten information that is suddenly relevant again.
To ensure the model is fast enough for practical use (like CCTV monitoring), the IEN discards optical flow entirely. Instead, it uses a Temporal Shift Module (TSM) as a feature extractor, which captures motion information directly from RGB frames with minimal overhead.
System Architecture
The IEN processes video chunks sequentially. Each chunk is passed through a feature extractor and then into a series of IEU blocks. The architecture ensures that the current frame ($x_0$) is routed specifically to the elevation, input, and output gates, while keeping the forget gate identical to a standard LSTM to maintain stability.
Key Results and Performance
The model was evaluated on two gold-standard OAD benchmarks: THUMOS-14 (sports actions) and TVSeries (realistic, complex environments).
- State-of-the-Art Accuracy: On THUMOS-14, the IEN achieved a mean Average Precision (mAP) of 60.4%. Remarkably, this RGB-only model outperformed existing "two-stream" models that use both RGB and expensive optical flow (which peaked at 55.7%). On TVSeries, it achieved a mean Calibrated Average Precision (mcAP) of 81.4%.
- Real-Time Speed: The feature extraction process is incredibly fast, reaching 266 FPS for RGB frames on a Titan XP GPU. In contrast, calculating optical flow typically crawls at around 12 FPS, making it the primary bottleneck in older systems.
- Efficiency Trade-off: Adopting the IEU instead of a standard LSTM resulted in a 5.78% performance boost with only a modest 12.5% decrease in processing speed—a trade-off that is highly beneficial for accuracy-critical applications.
Limitations and Future Work
While the IEN performs exceptionally well on distinct physical actions, it still struggles with subtle gestures (like "cricket shot" or "point") and actions that occur extremely quickly. Future work may involve refining the feature extractor to better capture these subtle visual cues without sacrificing the real-time speed that makes the IEN practical for real-world surveillance and monitoring.
Summary updated