Duration-Aware Soft-Max Pooling for Weakly Labeled Bidirectional Sequence Models

Many sequence datasets come with only one label per recording, such as "this 10-second clip contains a dog bark," and no timestamps. A common recipe trains a bidirectional recurrent network, which reads the sequence both forwards and backwards so that each frame sees past and future context. The network predicts a probability for every frame, and a pooling function then collapses those frame probabilities into one clip-level score that can be compared with the label. The choice of pooling function decides which frames receive training signal. Max pooling rewards only the single loudest frame. Averaging drowns a half-second bark in nine seconds of traffic. Log-sum-exp (LSE) pooling and its softmax-weighted relatives sit in between. LSE is a smooth maximum whose "temperature" parameter slides it from averaging toward max. Every existing system fixes that temperature, or learns one value per class. Yet the right temperature depends on how much of the clip the event actually occupies, and that changes from clip to clip.

We propose to let the network set the pooling temperature for each clip and each class from its own bidirectional summary of the clip, with two constraints. The pooling weights must stay increasing in frame probability. During training, classes that are absent from a clip are pooled sharply. The second constraint closes the shortcut that has made learned attention pooling produce false alarms. The aim is to remove the long-versus-short "duration bias" of weakly supervised sound event detection with one principled operator, replacing the stack of hand-tuned subsampling, norm pooling and double thresholding that currently fixes it.

Core Claim

The softmax-weighted pooled score for class $c$ over frame probabilities $p_1,\dots,p_T$ is

$$ \hat y_c = \sum_{t=1}^{T} p_t \,\frac{e^{\beta_c p_t}}{\sum_{s} e^{\beta_c p_s}}, $$

where $\beta_c \ge 0$ is the temperature. With $\beta_c = 0$ this is the mean, and as $\beta_c \to \infty$ it becomes the max. A short calculation shows why one fixed $\beta_c$ cannot serve all durations. Suppose an event fills $k$ of $T$ frames at probability $p_{hi}$ against background $p_{lo}$. The pooled score approaches $p_{hi}$ only when

$$ \beta_c\,(p_{hi}-p_{lo}) \gg \log\frac{T-k}{k}. $$

This condition comes from requiring the event frames' total softmax weight to dominate the background's. The temperature an event needs therefore grows with the log of the inverse fraction of the clip it covers. A dog bark needs sharp pooling. A vacuum cleaner needs gentle pooling, so that gradient spreads over its whole extent and its detection does not fragment.

Our operator predicts $\beta_c(x) = \mathrm{softplus}(g_c(\bar h))$. Here $\bar h$ is the time-averaged concatenation of forward and backward hidden states, and $g_c$ is a small linear head. Because $\beta_c(x)$ is shared across all frames of the clip, the weights remain increasing in $p_t$ within that clip. This monotonic link between probability and weight is the property that A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling argued attention pooling lacks.

There is a remaining shortcut. On a clip where class $c$ is absent, the head could lower $\beta_c$ to average away a spurious spike instead of suppressing the spike. We block it with asymmetric training. For classes absent from the clip, the pooled score uses a fixed large temperature, close to max, so any false frame is penalised. The learned temperature applies only to classes present in the clip, and the temperature head receives gradient only from those classes. At inference the frame probabilities are thresholded for localization. The clip tag is computed with the predicted temperature, and we report this train/test asymmetry openly.

Research Program

We first build the operator into a convolutional-recurrent detector. The design follows TALNet from A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling: convolutional feature extraction, a bidirectional GRU, and per-frame class probabilities. We then swap the recurrent block for a bidirectional linear recurrent layer in the style of Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion, which runs forward and backward parallel scans at linear cost, to check that the operator does not depend on a particular backbone.

The prior work plays three roles. Duration robust sound event detection motivates the problem. It documents a 17-point F1 gap between long and short events in standard bidirectional detectors, and closes it with linear softmax pooling, intermediate $\alpha$-mean-max or $L^p$-norm subsampling, and double thresholding. We aim to match that with one learned operator and no pooling-specific post-processing. Adaptive pooling operators for weakly labeled sound event detection supplies the method we extend. Its auto-pool is exactly the equation above with a static per-class $\alpha$, and we make that dial depend on the clip. Geometric Scene Parsing with Hierarchical LSTM uses LSE between two recurrent stages over images, but with a fixed temperature and externally computed superpixels. It shows that LSE sits comfortably inside recurrent pipelines, though it does not address temporal duration.

The analysis side tests the log-inverse-fraction prediction directly. Using the strong labels available for evaluation, we regress the learned $\beta_c(x)$ against $\log((T-k)/k)$ per clip. A positive, roughly linear relation would show that the head learned a duration estimate rather than an arbitrary correction.

We are honest about the bidirectional ingredient. A causal network's final state has also seen the whole clip, so bidirectionality is not logically required for a clip-level temperature. Our hypothesis is narrower. Bidirectional frame probabilities have sharper onsets and offsets, as the sleep-staging transition results in Evaluating the Influence of Temporal Context on Automatic Mouse Sleep Staging through the Application of Human Models suggest, and so give a better duration signal. That is an empirical claim, and we test it by ablation.

Baselines

The naive reference is the same bidirectional detector with mean pooling and with max pooling. These two set the poles of the duration-bias problem. On URBAN-SED no number is reported for them in the file; the auto-pool paper only says auto-pool consistently beat them.

The strongest existing method on DCASE 2018 Task 4 is the duration-robust system of Duration robust sound event detection. It combines a bidirectional CRNN, linear softmax pooling, subsampling and double thresholding, and reports 32.5% event F1 on the evaluation set. In the same paper the DCASE 2018 winner reaches 23.32% F1 on short events and the $\alpha$-mean-max variant 29.66%, while the $L^p$-norm variant cuts the long–short gap to 4 points. We reimplement it on our backbone. Beating it means at least matching 32.5% overall F1 while reaching a long–short gap no larger than 4 points, using only standard thresholding.

The strongest adaptive-pooling competitor is auto-pool from Adaptive pooling operators for weakly labeled sound event detection. On URBAN-SED it reports frame-level F1 of 0.757, against 0.762 for a strongly supervised model. Since auto-pool already nearly closes the gap to strong supervision on URBAN-SED, gains there will be small. DCASE 2018, with real recordings and wider duration spread, is the main test.

Linear softmax pooling as in TALNet and unconstrained attention pooling complete the comparison. All methods share the backbone, features, training schedule and a common hyperparameter budget, with the same number of tuning trials per method and threshold selection on the same validation split.

Three ablations isolate the proposal's ingredients. The first keeps the temperature head but drops asymmetric training. The second feeds the same bidirectional summary into an unconstrained per-frame attention head, which isolates monotonicity. The third computes the temperature from a unidirectional final state, which isolates bidirectional context.

Why not just…?

Why not just learn one temperature per class, as auto-pool does? If a class's events have similar durations across clips, a per-class $\alpha$ captures nearly everything, and our head adds parameters for nothing. This is a real threat on URBAN-SED, whose synthetic soundscapes may have narrow per-class duration spread. The deciding experiment stratifies DCASE 2018 test events by duration within each class. Clip-conditioned temperature should help most where within-class duration variance is highest. If it gives no gain there, the conclusion is that duration bias is a class-level phenomenon, and we will report that.

Why not just use attention pooling, which already conditions weights on the hidden states? A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling found that attention pooling tags competitively but localizes poorly because of false positives, and lost to parameter-free linear softmax pooling. Our operator is a deliberately restricted form of conditioning: one nonnegative temperature per clip, monotonic weights, and sharp pooling for absent classes. The matched-input attention ablation tests whether the restriction, rather than the extra context, is what helps.

Why not just keep linear softmax pooling and tune the post-processing? Dinkel and Yu show that fixed median filtering erases short, confident events. Their gap reduction needed changes inside the network, namely subsampling and norm pooling, not thresholds alone. Linear softmax is also a fixed operator whose effective sharpness cannot adapt to duration. If our operator only equals their full pipeline, the contribution shrinks to simplicity. The comparison at an equal tuning budget will show whether that is the case.

Proposed Experiments

The primary experiment is DCASE 2018 Task 4. We report event-based F1 on the evaluation set, separate F1 for the long and short class groups defined by Dinkel and Yu, and the gap between them, with the duration-robust system, auto-pool, linear softmax, attention, mean and max as baselines. URBAN-SED serves as a secondary check against auto-pool's 0.757 F1. DCASE 2017 Task 4 tests localization error rate against linear softmax pooling; the file gives no reported number for TALNet's localization error, so we rerun it.

Three checks decide the claim. First, false-positive frames on clips where the class is absent must not exceed those of linear softmax pooling. If they rise, monotonicity plus asymmetry has failed to block the attention failure mode, and the operator is not worth adopting. Second, the learned temperature must correlate positively with the log inverse active fraction computed from strong labels. Without this the mechanism story is wrong even if F1 improves. Third, the gain must survive the swap to a bidirectional linear recurrent backbone, and its dependence on bidirectional versus causal input to the temperature head must be measured. This tells us whether bidirectionality matters for the temperature or only for the frame probabilities.

A transfer probe on THUMOS14 weakly supervised action localization swaps our pooling into a standard temporal class-activation pipeline. It asks whether the duration argument carries over to video. The bidirectional-consistency method of Weakly-Supervised Temporal Action Localization with Bidirectional Semantic Consistency Constraint, which reports 46.6% average mAP over IoU 0.1–0.7, is a reference point rather than a target.