A Short Note on Evaluating RepNet for Temporal Repetition Counting in Videos

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, Andrew ZissermanarXiv 2024 · DOI · arXiv

Abstract

In the field of temporal repetition counting, RepNet (2020) has served as a foundational baseline. However, recent literature has reported surprisingly poor performance for this model on modern datasets. This note identifies a "baseline drift" where researchers inadvertently crippled RepNet by modifying its architecture to handle longer sequences. By re-evaluating the original RepNet using its intended multi-speed strategy, we show it remains superior or highly competitive to state-of-the-art models released as recently as 2024.

Introduction

Temporal repetition counting—estimating the number of cycles in periodic actions like exercise or industrial processes—is a specialized video understanding task. For years, RepNet has been the standard benchmark. Recently, papers such as TransRAC suggested that RepNet struggled with long sequences or actions with gaps.

Investigation reveals that these studies evaluated a "modified" version of RepNet. Specifically, researchers altered the final layer to accommodate more than 32 frames, which broke the model's learned priors. This "modified RepNet" became a straw man, with its poor results being cited across subsequent papers, creating a false narrative of progress in the field.

Evaluation Methodology

The authors emphasize that RepNet's original design handles long sequences through evaluation strategies rather than architectural changes:

  • Multi-speed Evaluation: Instead of increasing the input frame count, the video is sampled at multiple speeds (1× to 5×). At 5× speed, the 32-frame window covers 160 frames, allowing the model to capture longer periods without retraining.
  • Periodicity Thresholding: To handle non-continuous repetitions, a per-frame periodicity score ($p_i$) is used. Frames are only counted if their score exceeds a threshold ($\tau = 0.5$), effectively ignoring gaps in action.
  • Metric Standardization: The note clarifies discrepancies in Mean Absolute Error (MAE) reporting. Some papers use a smoothing parameter $\alpha=0.1$ while others use $\alpha=0$, leading to incomparable results.

Experimental Results

The re-evaluation spans three major benchmarks: Countix, UCFRep, and RepCount-A.

UCFRep Performance

On the UCFRep dataset, the "modified RepNet" reported in previous literature achieved a near-zero Off-by-one Accuracy (OBOA) of 0.009. In contrast, the correctly evaluated original RepNet achieves an OBOA of 0.7333, outperforming all 2024 models listed.

Table 1: Performance comparison on the UCFRep dataset. The original RepNet significantly outperforms the "modified" version and recent 2024 baselines.

Countix Performance

On the large-scale Countix dataset, RepNet achieves an MAE of 0.3083. This is substantially better than the 0.5278 MAE reported for RACNet (2024).

Table 2: Performance comparison on the Countix dataset. RepNet remains superior to newer models like RACNet.

Discussion and Conclusion

The primary insight is that a robust evaluation strategy can be more critical than architectural complexity. RepNet’s success at a low resolution ($112 \times 112$) compared to modern models requiring $224 \times 224$ underscores its efficiency.

This note serves as a cautionary tale regarding "inherited" results. Researchers must verify baseline implementations to ensure that reported improvements represent genuine algorithmic advances rather than comparisons against improperly configured predecessors. By providing corrected benchmarks and code, the authors aim to restore a rigorous foundation for future repetition counting research.

Summary updated

Abstract

We discuss some consistent issues on how RepNet has been evaluated in various papers. As a way to mitigate these issues, we report RepNet performance results on different datasets, and release evaluation code and the RepNet checkpoint to obtain these results. Code URL: https://github.com/google-research/google-research/blob/master/repnet/
BibTeX
@misc{dwibedi2024short,
  author = {Debidatta Dwibedi and Yusuf Aytar and Jonathan Tompson and Pierre Sermanet and Andrew Zisserman},
  title = {{A Short Note on Evaluating RepNet for Temporal Repetition Counting in Videos}},
  year = {2024},
  eprint = {2411.08878},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.08878},
}
Figure