Live Repetition Counting

Ofir Levy, Lior WolfICCV 2015 · OpenReview

Abstract

The task of counting the number of repetitions of approximately the same action in an input video sequence is addressed. The proposed method runs online and not on the complete pre-captured video. It analyzes sequentially blocks of 20 non-consecutive frames. The cycle length within each block is evaluated using a convolutional neural network and the information is then integrated over time. The entropy of the network's predictions is used in order to automatically start and stop the repetition counter and to select the appropriate time scale. Coupled with a region of interest detection mechanism, the method is robust enough to handle real world videos, even when the camera is moving. A unique property of our method is that it is shown to successfully train on entirely unrealistic data created by synthesizing moving random patches.

Introduction

Counting repetitions in video streams is a fundamental task for activity monitoring, sports analytics, and biological research. However, performing this "live" (online) on unconstrained video is notoriously difficult. Traditional methods are often "offline," requiring the entire video to be recorded first for frequency analysis. These approaches frequently fail when the camera moves, when action speeds vary, or when the video contains irrelevant segments. This paper presents a real-time system that overcomes these challenges by treating repetition counting as a temporal classification problem.

Methodology

The system processes video at 50 frames per second, making it suitable for live deployment. Its architecture consists of several key components designed for robustness and autonomy.

Region of Interest (ROI) Detection

To mitigate the effects of camera motion, the system first identifies the area of most significant motion in the frame and crops the view to this Region of Interest. This allows the subsequent neural network to focus on the action itself rather than background interference.

Overview of the system pipeline, showing input frames, ROI heatmaps, and temporal blocks for different actions.

3D Convolutional Neural Network

The core of the system is a 3D CNN that analyzes blocks of 20 non-consecutive frames. Instead of predicting a continuous repetition count, the network classifies each block into one of eight discrete "cycle length" categories. This classification approach is superior to regression because the temporal filters required to distinguish specific cycle lengths are well-defined and distinct.

A block of 20 non-consecutive frames from a squatting sequence, illustrating the temporal input analyzed by the network.

Training on Synthetic Data

A remarkable aspect of this work is that the CNN was trained entirely on unrealistic synthetic data. The training set consisted of 30,000 sequences of random patches moving in linear, circular, or oscillating patterns. The fact that this network generalizes to real-world human motion suggests that it has learned the fundamental temporal structure of periodicity rather than specific visual features of objects.

Entropy-Based Control

The system uses the entropy of the network's predictions as a confidence score. Low entropy indicates that the network is "sure" it sees a repetition. This signal is used to automatically start and stop the counter and to select the optimal temporal scale from three parallel detectors (sampling every 2nd, 5th, or 8th frame).

Datasets and Benchmarks

The authors validated the method using:

  • Synthetic Training Set: 30,000 sequences of moving random patches.
  • Existing Benchmarks: Six established datasets from prior literature.
  • The YouTube Benchmark: A new dataset of 100 challenging videos featuring diverse actions like cooking, exercising, and animal movements.
  • Live Interaction: An hour-long test where hundreds of users interacted with the system in a public setting.

Results

The system achieved an average error rate of approximately 7% on the YouTube benchmark, significantly outperforming previous state-of-the-art methods which often had error rates near 30%. Its robustness to camera motion and its ability to work even when videos are played in reverse demonstrate that it captures the underlying structure of repetitive cycles.

Conclusion

By combining ROI detection, a 3D CNN trained on synthetic data, and entropy-based control, this research provides a robust framework for live repetition counting. The ability to learn from abstract synthetic data highlights the power of temporal abstraction in computer vision tasks.

Summary updated

Abstract

The task of counting the number of repetitions of approximately the same action in an input video sequence is addressed. The proposed method runs online and not on the complete pre-captured video. It analyzes sequentially blocks of 20 non-consecutive frames. The cycle length within each block is evaluated using a convolutional neural network and the information is then integrated over time. The entropy of the network's predictions is used in order to automatically start and stop the repetition counter and to select the appropriate time scale. Coupled with a region of interest detection mechanism, the method is robust enough to handle real world videos, even when the camera is moving. A unique property of our method is that it is shown to successfully train on entirely unrealistic data created by synthesizing moving random patches.
BibTeX
@inproceedings{levy2015live,
  author = {Ofir Levy and Lior Wolf},
  title = {{Live Repetition Counting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=BJZ30eGd-r},
}
Figure