When a sound separator learns from tags such as "this 4-second clip contains a siren" instead of from clean recordings of the siren, it is usually trained by passing its output through a frozen sound classifier and rewarding outputs the classifier recognizes. The classifier only needs the few most telling time-frequency cues to say "siren." The separator therefore learns to keep those cues and discard the rest of the event, such as its harmonic tail, its lower-energy body and its onset and decay margins. The result is a thin, truncated estimate full of artifacts. This is a main reason weak supervision still trails full supervision. On dense UrbanSound8K mixtures, clip-level tags reach a 5.6 dB SI-SDR improvement versus 9.0 dB with clean targets (Finding Strength in Weakness).
We propose to attack truncation with the network's structure rather than with another loss on the final output. The separator is an encoder-decoder. Its coarsest decoder stage must produce a complete low-resolution mask that, on its own, convinces the classifier. Each finer stage may then only adjust that mask by a bounded residual. A mask that is 8× coarser in time and frequency cannot select isolated salient bins. To keep the class evidence, it has to cover the whole local region where the event lives. The bounded residuals then sharpen boundaries without hollowing the event out. Deep supervision here means applying the classifier loss at every decoder resolution. It is used as a scale-dependent restriction on what the separator is allowed to throw away, and gradient flow is not the motivation.
Core Claim
Truncation happens because a fine-resolution mask can satisfy a discriminative critic with very little energy. If the critic must first be satisfied by a mask that physically cannot be sparse at fine scale, and if fine-scale stages can only modify that mask within a fixed budget, then the separator keeps substantially more of each target event. We expect this to show up as higher recall of target energy in low-salience regions and higher SI-SDR. The training data stay the same clip-level labels. The testable bar is to match or exceed frame-level weak supervision (6.8 dB on UrbanSound8K in Finding Strength in Weakness) while using only clip-level tags.
Relation to Prior Work
The weak-supervision side supplies the training signal and the failure mode we target.
- Kong et al. (joint separation-classification model) showed that an un-downsampled mask network trained through a pooling classifier learns to separate. The choice of pooling mattered greatly: weighted rank pooling gave 8.08 dB SDR, while max pooling gave 0.03 dB. That contrast is direct evidence that how much of the event the critic "asks for" determines separation quality. Their network has a single output resolution and no intermediate losses.
- Pishdadian et al. (Finding Strength in Weakness) froze a pretrained classifier as critic and added a mixture reconstruction loss. That is the method we build on and the baseline we must beat. Their separator is a flat three-layer recurrent network. Their "multi-scale" comparison concerns label resolution, not architecture.
- Anchor mining (Universal Source Separation With Weakly Labelled Data) sidesteps the critic. It mines single-source segments with a tagging model and trains a deep ResUNet with an ordinary terminal loss. This only works where single-source windows exist. Its documented need to freeze the guiding networks against co-adaptation constrains us to keep the critic frozen too.
The deep-supervision side supplies the mechanism.
- Stacked hourglass networks (Newell et al.) and stacked deconvolutional networks (Fu et al.) show that making intermediate stages emit complete predictions in the output space, and then refining them, helps dense prediction. Both were trained with full labels.
- T-MLP (Yang et al.) supplies the specific design we borrow: early heads reconstruct coarse structure and later heads add residual detail.
- OCTAve (Chinkamol et al.) is the closest prior combination of weak labels and deep supervision. It enforces KL consistency between decoder attention maps and the final segmentation of retinal images. Its intermediate stages only have to agree with the output; ours must independently pass an external critic, under a hard limit on fine-scale change.
- Targeted activation penalties (Zhang et al.) motivate acting on intermediate representations rather than on input gradients. They report that input-gradient penalties let spurious cues re-emerge in deeper layers.
Research Program
The separator is a U-Net-style mask estimator on magnitude spectrograms, with $K$ decoder stages. Stage $k=1$ is the coarsest, at a factor $2^{K-1}$ below full resolution in time and frequency. Stage 1 predicts per-class mask logits $a_1$. Each finer stage $k$ predicts a residual $r_k$. The mask at stage $k$ is
$$ M_k = \sigma\Big(\mathrm{up}(a_1) + \sum_{j=2}^{k} \delta\,\tanh(r_j)\Big), $$
where $\sigma$ is the logistic function, $\mathrm{up}$ is nearest-neighbour upsampling to full resolution, and $\delta$ is a fixed per-stage budget on how far, in logit units, a finer stage may move the mask away from the coarse one. The full-resolution output is $M_K$.
Training uses only clip-level tags $y$, a frozen classifier $C$ pretrained on mixtures, and the mixture spectrogram $X$:
$$ \mathcal{L} = \sum_{k=1}^{K} w_k\, \mathrm{BCE}\big(C(M_k \odot X),\, y\big) + \lambda\, \Big\| \sum_{c} M_K^{(c)} \odot X - X \Big\|_1 . $$
Here $w_k$ weights the classifier loss at stage $k$, $\mathrm{BCE}$ is binary cross-entropy, $\odot$ is elementwise product, $M_K^{(c)}$ is the full-resolution mask for class $c$, and $\lambda$ weights mixture reconstruction. Every intermediate mask is upsampled to full resolution before the classifier sees it. The critic therefore always receives inputs of its training shape, which removes the dimension-mismatch problem the file flags for deep supervision.
The two knobs that matter are $\delta$ and the coarsest resolution. We study them as a sweep, not as tuned hyperparameters. With $\delta \to \infty$ the model reduces to ordinary deep supervision. With $\delta = 0$ it outputs only a blocky coarse mask.
Beyond SI-SDR, the analysis asks whether truncation actually decreased. We measure it with ground truth available only at evaluation time:
- Energy recall per salience bin. We split target time-frequency bins into those the critic's input gradient marks as salient and those it does not, and compute how much reference energy the mask retains in each.
- Signal-to-artifacts ratio. This captures the musical-noise side of truncation.
Baselines
Mixture as estimate. The do-nothing reference returns the mixture as the estimate. All improvements are reported relative to it, so it is 0 dB SI-SDR improvement by construction. Clean-target training of the identical U-Net gives the upper reference. Finding Strength in Weakness reports 9.0 dB SI-SDR improvement for strong supervision on UrbanSound8K.
Classifier-critic training with mixture reconstruction. This is the strongest competitor on exactly this question: a frozen classifier critic plus mixture reconstruction loss, trained from clip-level labels. Pishdadian et al. report 5.6 dB SI-SDR improvement with clip-level labels and 6.8 dB with frame-level labels on UrbanSound8K mixtures of 5 to 10 events, and 8.9 dB for vocals on MUSDB18.
- Reproduction. We will rerun their recurrent separator in our pipeline and check we reproduce roughly 5.6 dB.
- Fair comparison. We will also train their loss on our U-Net with only the terminal output. Architecture and objective are then separated.
- Shared conditions. All variants share the critic, data, epochs and an equal hyperparameter search budget.
- What counts as beating it. Exceeding the U-Net terminal-loss version by at least 1 dB, averaged over three seeds with non-overlapping confidence intervals, counts as beating it. Reaching 6.8 dB with clip labels is the stronger target.
Anchor-mining separation. This baseline uses the method of Universal Source Separation With Weakly Labelled Data, run with the same backbone and the same tagging model used to mine anchors. It is the strongest alternative that avoids the critic entirely. That paper reports 8.12 dB SDRi on MUSDB18 and 10.57 dB on FSDKaggle2018, but in a different training regime. No reported result was found on Lacuna for anchor mining on dense UrbanSound8K mixtures, so we run it ourselves.
Key-ingredient ablations. Two ablations isolate the bounded residual:
- Plain multi-scale deep supervision. The classifier loss is applied at every decoder stage, but the residuals are unbounded.
- Coarse-only masks. We set $\delta = 0$.
If plain deep supervision matches the full method, the gain comes from extra gradient paths, not from the truncation argument, and our core claim fails.
Why not just…?
Why not just train a bigger or deeper separator with the terminal loss? Truncation is an incentive problem, not a capacity problem. A larger network satisfies a sparse-cue critic more precisely, which can make truncation worse. The 60-layer ResUNet in Universal Source Separation With Weakly Labelled Data still averages only 5.57 dB SDRi across AudioSet classes. We include a depth-matched terminal-loss U-Net with equal or larger parameter count. If it closes the gap, the proposal is unnecessary.
Why not just add a leakage loss that penalizes the classifier for still detecting the target in the residual (mixture minus estimate)? This is the most natural fix, and we test it as a baseline. Our prediction is that it fails for the same reason truncation occurs. The discarded event body carries little discriminative evidence, so the critic does not fire on the residual and the penalty stays near zero. The salience-binned energy recall measurement settles this directly. Leakage loss should improve recall only in salient bins, while bounded coarse-first masks should improve it in non-salient bins. If leakage loss improves both, it is the simpler method and we would recommend it.
Why not just add a total-variation smoothness prior (a penalty on differences between neighbouring mask bins) to the terminal mask? This is a real threat, because smoothness also discourages isolated salient bins. The difference is that total variation penalizes all fine structure uniformly, including legitimate harmonic striations. Our method permits fine structure up to a bounded deviation from a mask the critic already accepted at coarse scale. We sweep the total-variation weight over the same budget as $\delta$ and compare the trade-off between signal-to-artifacts and signal-to-interference. If total variation dominates our method on that trade-off, our method offers nothing beyond a smoothness prior.
Why not just mine anchor segments and train as if supervised? In mixtures of 5 to 10 overlapping urban events, clean single-source windows are rare, so mined pseudo-targets are contaminated. The anchor-mining baseline above measures this directly on UrbanSound8K. On MUSDB18 it may well win, and we will report that honestly.
Proposed Experiments
Primary testbed. The primary testbed is the UrbanSound8K setup of 5 to 10 overlapping events with clip-level tags, following Finding Strength in Weakness. This is where truncation and the weak-strong gap are documented.
Secondary testbeds. MUSDB18 tests vocals and the bass-drum co-occurrence case. Its 8.9 dB clip-level vocal result is the reference. The modified DCASE 2017 synthetic set is checked against the 8.08 dB SDR of the joint separation-classification model.
Metrics. We report SI-SDR improvement, signal-to-artifacts ratio, signal-to-interference ratio, and salience-binned energy recall.
Decisive checks. There are three:
- Gap closure. With clip labels, the full method should approach the 6.8 dB frame-level result and clearly beat both the terminal-loss U-Net and plain deep supervision.
- Mechanism. The improvement should be concentrated in non-salient-bin energy recall and signal-to-artifacts ratio. Gains spread uniformly would point to an optimization effect, not reduced truncation.
- Interior optimum in the budget. The $\delta$ sweep should show an interior optimum. $\delta = 0$ should lose fine detail, and $\delta \to \infty$ should revert to truncated masks. A monotone curve in either direction would falsify the coarse-first account.
References
- Finding Strength in Weakness: Learning to Separate Sounds With Weak Supervision
- A Joint Separation-Classification Model for Sound Event Detection of Weakly Labelled Data
- Universal Source Separation With Weakly Labelled Data for Computational Auditory Scene Analysis
- Stacked Hourglass Networks for Human Pose Estimation
- Stacked Deconvolutional Network for Semantic Segmentation
- T-MLP: Tailed Multi-Layer Perceptron for Level-of-Detail Signal Representation
- OCTAve: 2D En Face Optical Coherence Tomography Angiography Vessel Segmentation in Weakly-Supervised Learning With Locality Augmentation
- Targeted Activation Penalties Help CNNs Ignore Spurious Signals