Coupled low-rank front-ends for detecting short sounds from clip-level labels

Systems that learn to timestamp sounds from clip-level tags alone (labels saying only "a dog barks somewhere in these 10 seconds") are very good at long, steady sounds and very bad at short ones. In one DCASE 2018 baseline, cat, dog and speech each scored 0.0% F1 while vacuum cleaner reached 46.5% (Large-scale weakly labeled semi-supervised sound event detection in domestic environments). Almost every fix so far changes how frame scores are pooled into a clip score. The input spectrogram stays the same. That input already commits to one window length, and a window long enough to resolve pitch smears a bark or a click across time.

We propose a front-end that removes that commitment cheaply. It takes two spectrograms of the same clip: a long-window one that is sharp in frequency and a short-window one that is sharp in time. It explains both as blurred views of a single fine-grid map, written as a low-rank background (a steady hum is the same few spectral shapes repeated) plus a sparse event layer (a bark is brief and localized). The decomposition uses a fixed, small number of factorized gradient steps instead of a convex solver. This keeps its cost low and identical for every clip, which matters both for training on tens of thousands of clips and for keeping distributed workers in lockstep. The claim is that the sparse layer hands the detector short events that are already separated from the background, with timing from the short window and spectral identity from the long one. This should narrow the short-versus-long gap in a way that a two-channel input or better pooling does not.

Core Claim

The coupled "super-resolution spectrogram" idea already exists. A super-resolution spectrogram using coupled PLCA treats wideband and narrowband spectrograms as blurred copies of one latent map and recovers it by EM deconvolution. That work supplies our modeling assumption. It also constrains the design. It reports over a minute to process one second of speech, numerical degradation after about 150 iterations, and only qualitative results, with no link to detection.

Our proposal differs in three ways. First, the latent map gets a structural split into low-rank background and sparse events. Second, it is computed by a short unrolled Burer–Monteiro factorization, meaning the background is written directly as a product of two thin matrices rather than as a full matrix penalized by its nuclear norm. Third, it is trained end to end inside a weakly supervised detector.

Concretely, let $V^F$ and $V^T$ be the narrowband and wideband magnitude spectrograms. Let $A_F, A_T$ be known linear operators that blur and resample a fine time-frequency grid onto each of them, following the convolutional model of the coupled PLCA paper. We solve

$$\min_{U,W,S}\ \|V^F - A_F(UW^\top + S)\|^2 + \|V^T - A_T(UW^\top + S)\|^2 + \lambda \|S\|_1 + \mu \|U^\top U - W^\top W\|^2 .$$

Here $UW^\top$ is the rank-$r$ background on the fine grid, $S$ is the sparse event layer, $\lambda$ sets sparsity, and the last term is the balance regularizer from bi-factored gradient descent in Finding Low-rank Solutions to Matrix Problems, Efficiently and Provably. That paper supplies our method. It shows that factorized gradient descent avoids per-iteration SVDs, converges linearly inside a basin reached by spectral initialization, and runs 10x to 20x faster than SVD-based projection on matrix sensing.

We run $K$ fixed steps with alternating gradient and soft-threshold updates. The initialization uses a fixed number of power iterations, so it never calls a full SVD. Per-band $\lambda$ is learned through the unrolled steps. The detector receives $S$ and $UW^\top$ as channels and uses Linear Softmax pooling with double thresholding from Towards Duration Robust Weakly Supervised Sound Event Detection.

Research Program

The work has three parts, in order of importance.

Part 1: does the front-end reduce duration bias? This is the main question. We build the coupled front-end and plug it into a fixed CRNN detector, a network of convolutional layers followed by a recurrent layer. The fine output grid is chosen so that, after the backbone's temporal subsampling, frames stay well under the 200 ms evaluation collar. This respects the resolution limit reported by the Duration Robust paper, where subsampling by 16 gave 320 ms frames and hurt accuracy.

Within this part, we isolate what the coupling and the low-rank/sparse split each contribute by removing them one at a time. We also check sensitivity to the rank $r$ and to $K$. The open problem of rank selection in factorized methods is real here: a steady background is plausibly rank-few, but blenders and running water may not be.

Part 2: is unrolled factorization the right solver? We compare it with the convex alternative, which is the same objective with a nuclear-norm background solved by singular value thresholding. That solver needs an SVD of the fine-grid matrix at every iteration (Lafferty; Ngo et al.), and is typically run until a tolerance is met.

The benign-landscape results (Lee et al.; Rank Optimality for the Burer-Monteiro Factorization) describe the inner decomposition problem only. They say nothing about the outer weak-label loss composed with pooling. So we test the inner solution empirically: we compare our $K$-step output against the converged convex solution, and we measure variance across initialization seeds.

Part 3: is per-clip cost predictable enough for synchronous training? This is a constraint the solver must satisfy, and we treat it as one. Taming unbalanced training workloads in deep learning with partial collective operations identifies variable per-sample work as a source of stragglers, meaning slow workers that hold up the others. Its remedy is eager collectives that tolerate stale or missing gradients.

Our hypothesis is that the main source of per-sample variance in this pipeline is not clip length but solver iteration count. A convex solver with tolerance-based stopping creates stragglers. A fixed-$K$ factorization does not. If so, the right fix is to remove the variance at its source rather than to absorb it with stale gradients. Stale gradients carry the open risk, never measured, of blurring frame boundaries that are judged against a 200 ms collar.

Baselines

Naive reference. A single log-mel CRNN trained only on the 1,578 weakly labeled clips. Disentangled Feature for Weakly Supervised Multi-class Sound Event Detection reports 10.8% event-based macro F1 for the DCASE 2018 baseline. The semi-supervised study reports 7.51% (weak labels only) and 14.06% (with unlabeled data). We rerun this configuration ourselves so that every system shares the same feature pipeline.

Strongest existing methods. Class-wise attention pooling with disentangled features (cATP-DF) from the Disentangled Feature paper gives each class its own attention mask and its own slice of feature channels. It reports 34.0% macro F1 on DCASE 2018 Task 4. The Duration Robust system reports 32.52% macro F1 for its best fusion. Its $L^p$-norm pooling cut the short-versus-long gap to 4.02%, against 17.04% for the challenge winner.

These are the two strongest competitors because they attack exactly our target, short events under weak labels, through pooling and thresholding alone. Our system beats them only if it matches or exceeds 34.0% macro F1 and also lowers the duration gap below 4.02% under the same protocol. A gain on long events alone does not count.

Fair comparison. All systems use the same data splits, the same unlabeled-data regime, and the same detector backbone where architectures allow. They also get the same tuning budget (an equal number of hyperparameter trials) and the same GPU-hour cap.

Ablations. Each removes one piece of the proposal:

  • Two-channel input: both spectrograms resampled to a common grid and stacked as two input channels, in the spirit of the dual-resolution input of Wang et al.. Its compute matches ours.
  • No coupling: the low-rank/sparse front-end applied to the short-window spectrogram only.
  • No sparse layer: coupled reconstruction without the split.
  • Convex solver: singular value thresholding on the same objective.

We run coupled PLCA itself only on development clips, as a qualitative and runtime reference. Its reported cost rules it out at training scale.

Secondary check. On the modified DCASE 2017 mixtures, the joint separation-classification model of A joint separation-classification model for sound event detection of weakly labelled data produces full-resolution separation masks with rank-weighted pooling. It reports an equal error rate of 0.14 and 8.08 dB signal-to-distortion ratio with that pooling. These results tell us whether the sparse layer actually separates gunshots and glass breaks, rather than merely helping the classifier.

Why not just…?

Why not just stack the two spectrograms as input channels and let the CNN sort it out? This is the most serious threat. It is cheap, and a convolutional network can in principle learn background suppression and cross-resolution alignment on its own. Our argument is about sample efficiency. The Duration Robust paper found that learned convolutional subsampling underperformed with only 1,578 labeled clips, which suggests that structure learned from weak labels at this scale is hard to come by. Still, that is indirect evidence. The matched-compute two-channel ablation decides the question. If it equals our system on short-event F1 and on the duration gap, the factorized front-end adds nothing, and we would report that.

Why not just keep one spectrogram and improve pooling and thresholding? That route has been pushed hard. The Duration Robust paper reaches 32.52% macro F1, and cATP-DF reaches 34.0%. A residual gap remains, and pooling cannot recover timing that the analysis window has already smeared, which is the core point of the time-frequency trade-off (Wang et al.). We test this by running our front-end under each of these pooling heads. If the gains vanish once pooling is strong, the front-end is redundant.

Why not just use convex robust PCA, which has a unique global optimum? Its optimum is not the issue. Its cost and its variable running time are. Every iteration needs an SVD. Even on a small problem, MovieLens 1-bit completion, the convex TFOCS solver took 42.3 s against 25.4 s for factorized descent, as reported in the bi-factored gradient descent paper. The convex-solver ablation measures three things: wall-clock time per clip, the variance of that time across clips, and whether converging fully buys any F1 over $K$ fixed factorized steps.

Why not just bucket clips by length or train asynchronously with Eager-SGD? Length bucketing addresses variable clip length, but we expect that source to be small here, since clips are mostly of similar duration. We verify this. Bucketing does nothing about variable solver iterations. Eager-SGD gave 1.27x speedup on variable-length video in its own paper, but it was never tested on tasks scored by tight temporal collars. If Eager-SGD on the convex front-end matches our synchronous fixed-$K$ setup in both throughput and boundary precision, then our systems argument fails, and the solver choice must rest on F1 and compute alone.

Proposed Experiments

Main experiment. DCASE 2018 Task 4 with event-based macro F1 at the 200 ms collar, per-class F1 for cat, dog and speech, and the short-versus-long gap computed as in the Duration Robust paper. The decisive check is whether the full model improves short-event F1 over both cATP-DF and the two-channel ablation while holding long-event F1. We also need the gap below 4.02% and macro F1 at or above 34.0%, over at least three seeds.

Separation check. Modified DCASE 2017, reporting equal error rate and signal-to-distortion ratio of the sparse layer against the joint separation-classification results.

Systems check. Eight-worker data-parallel training. For each of the fixed-$K$ factorized, convex-tolerance, and convex-plus-Eager-SGD setups, we record per-step time distribution, worker idle fraction, and total time to a fixed F1. We also record the shift in predicted onsets relative to synchronous training.

Falsification conditions. The proposal is refuted if the two-channel ablation matches the full model, or if the convex solver's predictable-cost problem turns out to be negligible in practice. Either result would be informative. The first would say the duration gap under weak labels is not an input-resolution problem. The second would say solver choice in learned audio front-ends can rest on accuracy alone.