LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition

Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, Larry S. Davis

Introduction

Video recognition is notoriously computationally expensive. Standard deep learning approaches typically apply the same heavy convolutional neural network (CNN) to every sampled frame in a video. However, not all frames are created equal: identifying a "desert" scene might only require a low-resolution glance, whereas distinguishing between "making a latte" and "making a cappuccino" requires zooming in on fine-grained details.

To address this "one-size-fits-all" inefficiency, researchers introduced LiteEval, a dynamic "look-twice" framework. LiteEval drastically reduces computational costs by skimming videos with a lightweight model and only invoking a heavy-duty model when necessary. By intelligently deciding which frames require high-resolution analysis, it achieves state-of-the-art accuracy while slashing computation by over 50%.

The LiteEval Architecture

LiteEval employs a hierarchical, coarse-to-fine architecture designed to dynamically allocate its computational budget. It consists of three main components:

  1. Coarse LSTM (cLSTM): This module runs on every frame. It takes features from a lightweight CNN (MobileNetV2) processing low-resolution ($112 \times 112$) images. It provides a "quick glimpse" and maintains a running history of the video.
  2. Conditional Gating Module: Acting as the "brain" of the system, this module looks at the coarse features and the previous state at each time step to decide if more detail is needed. It outputs a binary decision. To make this discrete "yes/no" choice trainable via standard backpropagation, the authors use the Gumbel-Softmax trick, which provides a differentiable approximation of a discrete distribution.
  3. Fine LSTM (fLSTM): This module only activates if the gate says "yes." It processes features from a powerful, expensive CNN (ResNet-101) using high-resolution ($224 \times 224$) images.
LiteEval Architecture

The "Sync" Mechanism: A critical innovation is how the two LSTMs stay in step. If the gate decides to skip the fine features for a frame, the hidden states from the coarse LSTM are synchronized (copied) into the fine LSTM. This ensures the fine-grained model doesn't "lose the thread" of the video's temporal progression even when it isn't actively processing high-res data.

Key Insights

  • Input Resolution is a Massive Lever: Evaluating a ResNet-101 on half-resolution images saves 74% of GFLOPs. LiteEval exploits this by using low-res as the default and high-res as the exception.
  • Difficulty-Aware Computation: The model naturally learns to spend more "effort" on difficult categories. For example, complex and varied "Marriage Proposal" videos triggered the fine-feature model much more frequently than visually consistent "Chorus" videos.
Fine feature usage across categories
  • Temporal Redundancy: Videos are highly redundant. By skipping fine-grained analysis on most frames, the model maintains accuracy while doing significantly less work.
  • Synchronization is Mandatory: Ablation studies showed that without the state synchronization between the coarse and fine LSTMs, accuracy plummeted (from 80% to 65.7% on FCVID). The fine model relies heavily on the context provided by the coarse model.

Datasets and Results

The researchers evaluated LiteEval on two massive video benchmarks: FCVID (~91k videos, generic topics) and ActivityNet (~20k videos, human activities).

  • Efficiency: LiteEval achieved a 51.8% reduction in computation (GFLOPs) on FCVID and a 51.3% reduction on ActivityNet compared to a strong uniform sampling baseline.
  • Accuracy: Despite massive computational savings, LiteEval maintained or improved accuracy. On FCVID, it matched the baseline at 80.0% mAP. On ActivityNet, it outperformed the baseline (72.7% vs 70.0% mAP).
  • Online Performance: The model is highly effective for "online" scenarios (where frames arrive sequentially), offering a superior accuracy-to-compute trade-off curve compared to standard models.

Limitations and Future Work

While highly effective, the framework has a few limitations that present opportunities for future research:

  • Recurrent Bottleneck: LSTMs were standard for temporal modeling at the time of publication, but the field has since shifted toward Transformers. Adapting this coarse-to-fine gating logic to a Transformer architecture is a logical next step.
  • Coarse Overhead: Even though the coarse model is lightweight, it still runs on every single frame. In extremely resource-constrained environments, this could remain a bottleneck.
  • Fixed Resolution: The model currently chooses between two fixed resolutions. A more flexible system might choose from a continuous spectrum of resolutions or even specific spatial "crops" of interest.

Summary updated

Abstract

This paper presents LiteEval, a simple yet effective coarse-to-fine framework for resource efficient video recognition, suitable for both online and offline scenarios. Exploiting decent yet computationally efficient features derived at a coarse scale with a lightweight CNN model, LiteEval dynamically decides on-the-fly whether to compute more powerful features for incoming video frames at a finer scale to obtain more details. This is achieved by a coarse LSTM and a fine LSTM operating cooperatively, as well as a conditional gating module to learn when to allocate more computation. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet, and the results demonstrate LiteEval requires substantially less computation while offering excellent classification accuracy for both online and offline predictions.
BibTeX
@inproceedings{wu2019liteeval,
  author = {Zuxuan Wu and Caiming Xiong and Yu-Gang Jiang and Larry S. Davis},
  title = {{LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=oSPp1rdiIMt},
}
Figure