Weakly-Supervised Temporal Localization via Occurrence Count Learning

Julien Schroeter, Kirill A. Sidorov, A. David MarshallICML 2019

Introduction

Temporal localization—identifying exactly when an event occurs in a data stream, such as a drum hit in audio or a gesture in video—traditionally requires massive datasets with precise manual timestamps. Labeling these "onset times" is incredibly tedious and prone to human error.

While "weakly-supervised" methods exist, they usually rely on simple presence/absence labels or complex attention mechanisms that do not always yield precise timing. However, it is much easier for a human annotator to state, "there are four drum hits in this 2-second clip," than to mark the exact millisecond each one starts. This paper introduces a novel weakly-supervised framework that learns to precisely localize events in time using only this total count of occurrences as a training label.

Methods: Event Process Estimation and LoCo-Loss

The authors propose framing the task as an Event Process Estimation (EPE) problem. Instead of predicting a single label for an entire clip, the model (typically an LSTM or GRU) outputs a probability $p(t)$ for every single time step, representing the likelihood of an event occurring at that exact moment.

LoCo-loss probability spikes aligning with events

The core innovation driving this framework is the LoCo-loss (Localization through Counting):

  1. Probabilistic Counting: The model treats the total count of events in a sequence as a sum of independent Bernoulli trials (the $p(t)$ values).
  2. Poisson-binomial Distribution: The distribution of this sum follows a Poisson-binomial distribution. The authors use a recursive formula to calculate the probability of the "true count" given the model's predicted sequence of $p(t)$ values.
  3. Implicit Constraint: Mathematically, to maximize the probability of a specific integer count (e.g., exactly 3 events), the model must drive its $p(t)$ values toward either 0 or 1. If the model is "unsure" and spreads small probabilities across many time steps, the variance of the count distribution increases, making the target count less likely.

Essentially, the model is forced to "pick a side" and create sharp spikes in probability to satisfy the counting requirement. These spikes naturally align with actual events in the data, meaning localization emerges purely from the loss function without any architectural hints or "localization priors."

Datasets and Key Results

The model was evaluated across three distinct domains, demonstrating remarkable versatility:

  • Drum Transcription: Using the IDMT-SMT-Drums and ENST Drums datasets, the task was to detect bass drum, snare, and hi-hat hits. The model achieved high precision, with localization errors often smaller than 20ms. On the D-DTD dataset, it achieved F1 scores (around 94-97%) comparable to state-of-the-art fully supervised models.
  • Piano Onset Detection: Using the MAPS database, the model tackled a significantly harder task involving 88 different pitches and complex polyphonic interactions. The LoCo model (F1: 71.99%) significantly outperformed older supervised baselines and approached modern state-of-the-art supervised systems (F1: 82.29%), despite never seeing a single timestamp during training.
  • Digit Detection (Images): Using a synthetic dataset based on MNIST, the authors turned 2D images into a 1D temporal sequence by sampling the images along a space-filling Hilbert curve.
MNIST digit localization using a Hilbert curve

In the MNIST experiment, the model achieved 99.12% recognition accuracy and successfully localized digits in 2D space by simply learning to count them along the 1D Hilbert path.

Key Insights and Limitations

The most striking insight from this work is that counting implies localization. Because the total count can only increase over time, the model is naturally prevented from "early triggering." If it predicts an event too early that doesn't happen, it cannot "undo" that count later, leading to a massive loss penalty (Mass Shift Irreversibility).

Limitations to consider: * Instantaneous Assumption: The current formulation assumes events are "instantaneous" (lasting one time step). While this works for drum hits or piano onsets, it might struggle with long, overlapping actions without modification. * Computational Complexity: The recursive loss calculation is $O(T^2)$, which can be slow for very long sequences. The authors suggest a "mass thresholding" trick to keep this manageable. * Rare Events: The model struggled slightly with very rare notes in the piano dataset, suggesting that a minimum frequency of occurrences is needed for the "counting" signal to become robust.

Ultimately, this framework offers a powerful new direction for weakly-supervised learning, potentially enabling the auto-labeling of massive datasets where only aggregate counts are known.

Summary updated

BibTeX
@inproceedings{schroeter2019weakly,
  author = {Julien Schroeter and Kirill A. Sidorov and A. David Marshall},
  title = {{Weakly-Supervised Temporal Localization via Occurrence Count Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
}
Figure