Audio Transformers read sound by cutting a spectrogram (a picture of sound energy over time and frequency) into small patches and treating each patch as a token. Uniform grids waste compute on silence and steady hum. A recent fix learns to squeeze uninformative time frames before the Transformer sees them. That fix was designed and tested for one task at a time. Once a single model serves several tasks, the squeezing step becomes a shared, lossy bottleneck. A frame that is redundant for detecting a dog bark may be exactly the steady background that identifies a train station or a speaker's voice. Anything the tokenizer throws away cannot be recovered by any later layer, adapter, or gradient trick.
This note proposes to measure that tokenizer-level conflict directly and to remove it. The tokenizer keeps a single shared token budget, but each task gets its own vote on which frames survive. The claim is simple. In multi-task spectrogram models with learned token reduction, most negative transfer happens at the tokenizer and not in the backbone. Letting each task vote on which frames to keep, under the same total budget, recovers single-task accuracy at a fraction of the cost of separate models.
Core Claim
Learning Temporal Resolution in Spectrogram for Audio Classification (DiffRes) introduces a differentiable module that scores each time frame and resamples the spectrogram so that low-information frames are merged. On FSD50K it improves mAP by 1.3% while cutting the frame rate from 100 to 25 frames per second. The research file notes that DiffRes has never been run with several objectives. Its guide loss deliberately squashes low-energy, steady-state frames, and those are precisely the frames that scene classification and speaker identification rely on.
We predict three things. First, a single DiffRes tokenizer trained on a summed multi-task loss will hurt the tasks that need stationary context, relative to their own single-task DiffRes models. Second, the gradients that different tasks send into the tokenizer's frame-scoring parameters will be clearly more opposed (lower cosine similarity) than the gradients they send into the shared Transformer. Third, a task-voted tokenizer will close most of the gap at equal token count.
This differs from the closest multi-task audio work. Fine-tuning Audio Spectrogram Transformer with Task-aware Adapters for Sound Event Detection places frame-level detection adapters in shallow layers and clip-level tagging adapters in deep layers. It keeps a fixed uniform patch grid, so it never faces the question of which frames exist at all. The pretraining work that mixes objectives, SSAST and ASiT, combines reconstruction and contrastive losses at the pretraining stage. It also uses a fixed grid and static loss averaging, and it fine-tunes one task at a time. Our proposal moves the multi-task question into the tokenizer itself.
Research Program
The backbone is a patch-based spectrogram Transformer initialized from a public SSAST checkpoint. It carries one head per task: a clip-level classifier for tagging, scenes, keywords and speakers, and a frame-level head where detection is used.
In front of it sits a lightweight convolutional frame scorer shared across tasks. The scorer emits $K$ importance curves $s_k(t)$, one per task $k$ over frame index $t$. These are combined into one allocation curve
$$ s(t) = \tau \log \sum_{k=1}^{K} \exp\big(s_k(t)/\tau\big), $$
a smooth maximum with temperature $\tau$. A frame is kept at high resolution if any task rates it highly. The combined curve drives the same differentiable resampling DiffRes uses, at a fixed budget $B$ of retained frames, for example 25% of the original. Each task's loss trains mainly its own curve $s_k$. Tasks therefore stop fighting over one shared importance score and instead compete only through the budget, which the smooth maximum arbitrates.
We will also build a partitioned variant. Here each task receives its own $B/K$ frames and the resulting token streams are concatenated. It spends the same budget but never shares a kept frame, and comparing it with the smooth-max version tells us whether sharing kept frames helps.
The analysis component is as important as the method. During training we will log per-task gradient cosine similarity separately for the frame-scorer parameters, the patch embedding, and each Transformer block. We will also log each task's importance curve on held-out clips. This lets us see whether scene and speaker tasks place weight on stationary regions that the event task discards. The log is the evidence for or against the claim that conflict concentrates at the tokenizer.
Baselines
The naive reference point is a uniform grid at the same reduced frame rate, 25 frames per second, feeding the same multi-task Transformer with a summed loss. It costs the same number of tokens and has no learned allocation. If the smarter tokenizers cannot beat plain downsampling at equal compute, the whole direction is moot.
The primary competitor on the multi-task question is a single shared DiffRes tokenizer (DiffRes) trained on the equally weighted sum of task losses. This is the obvious combination, and the one our claim says fails. The same system with gradient balancing is the strongest optimization-side competitor. We will apply the dual-balancing scheme from Dual-balancing for multi-task learning to the scorer, and RotoGrad-style representation rotation from RotoGrad to the backbone output. RotoGrad handles very wide representations by rotating a subspace; the paper reports strong results with a 1024-dimensional one.
An upper reference is $K$ separate single-task DiffRes models. They cost $K$ times the inference compute, and matching them within that budget is the target. For the reported state of the art on each task, we take figures from the file. AST reports 95.6% accuracy on ESC-50 and 98.1% on Speech Commands V2 with full-resolution uniform grids. ASiT reports 98.9% on Speech Commands V2 and 86.5% on VoxCeleb speaker identification. Kong et al.'s DCASE 2019 baselines report 70.3% on Task 1 acoustic scene classification with CNN9. These full-resolution single-task figures come from different training recipes. They are context for scale, not our fair comparison.
The key ablation replaces the per-task curves with one shared curve while keeping everything else identical. This isolates the task-voting ingredient. The partitioned-budget variant isolates the value of sharing kept frames.
Fairness rules apply to all rows. Every multi-task system uses the same SSAST initialization, the same balanced per-task batch sampling (following Yang et al.), the same token budget, the same number of training steps, and the same hyperparameter search budget of 12 trials per method, selected on held-out validation splits.
Beating the shared-DiffRes baseline means two things across three seeds. The stationary-context tasks must gain at least 1.5 points without the transient tasks losing more than 0.3. The macro-averaged relative gap to the separate single-task models must also shrink by at least half.
Why not just…?
Why not just downsample uniformly and let the Transformer sort it out? Uniform pooling treats a door slam and a second of room tone identically. DiffRes's FSD50K result shows that learned allocation beats fixed-rate input at equal frame rate for a single task. Whether that advantage survives several tasks is exactly the question here, and the uniform-25-frames-per-second row answers it. If that row matches ours on scene classification, we will report that stationary tasks do not need adaptive tokens.
Why not just add gradient balancing or RotoGrad to a shared tokenizer? This is the real threat. The file's own Gap 1 frames the problem that way, and RotoGrad shows that conflicting gradient directions cancel under plain loss sums. Our argument is that balancing still produces a single importance curve, and one curve cannot keep a frame for task A while dropping it for task B. The balanced-shared-DiffRes row tests this directly. If it matches task voting, the conflict was one of gradient scale rather than of which information gets kept, and the proposal reduces to "use balancing." We would report that outcome as the result.
Why not just use task-aware adapters in the backbone? Li et al. show that multi-task adapters beat single-task fine-tuning on DCASE 2022 Task 4. They report 64.85% event-based F1 and 0.5548 PSDS1, the Polyphonic Sound Detection Score, versus 59.60% and 0.5140 for their single-task AST-SED baseline. However, their adapters act after tokenization on a full uniform grid. We will run a variant with their shallow and deep adapter placement on top of shared DiffRes. If adapters close the gap, the conflict lives in the backbone and our tokenizer-bottleneck claim is wrong.
Why not just train separate models? Separate models are the accuracy ceiling. They multiply inference cost by $K$, and they discard the regularizing benefit of joint training reported for joint spoken-language models (Liu and Lane). We include them as the target, not as a competitor to beat.
Proposed Experiments
The main suite trains one model jointly on four tasks chosen to split along the stationary-versus-transient axis. Speech Commands V2 keywords and ESC-50 environmental events depend on transients. DCASE 2019 Task 1 acoustic scenes and VoxCeleb 1 speaker identification depend on sustained context. Metrics are accuracy on each and the mean relative gap to the single-task DiffRes models. A second setting adds DCASE 2022 Task 4, detection plus tagging, evaluated by event-based F1 and PSDS1. This tests whether frame-level detection, which needs temporal boundaries, pulls the allocation differently from clip-level tagging. No reported result found on Lacuna for DiffRes on any multi-task audio suite, so all multi-task numbers will be produced here under the matched protocol.
We rely on three decisive checks.
The first is the gradient-locality measurement. The claim predicts that tokenizer-parameter cosine similarity between scene and event tasks is negative under shared DiffRes, while backbone-block similarity is near zero. If instead conflict is largest in the backbone, we abandon the tokenizer framing.
The second is a counterfactual reinsertion test. For the shared-DiffRes model, we restore the frames it discarded at full resolution, with no retraining, and measure the per-task recovery. Large recovery for scene and speaker tasks but not for keywords shows that the lost accuracy is lost information rather than lost capacity.
The third sweeps the budget from 12% to 50% of frames. The gap between shared and task-voted tokenizers should shrink as the budget grows and vanish near full resolution, because conflict over which frames to keep only exists when frames are scarce. A gap that persists at high budget would point to an effect other than allocation. We would then have to explain it before claiming the mechanism.
References
- Learning Temporal Resolution in Spectrogram for Audio Classification
- Fine-tuning Audio Spectrogram Transformer with Task-aware Adapters for Sound Event Detection
- SSAST: Self-Supervised Audio Spectrogram Transformer
- ASiT: Audio Spectrogram vIsion Transformer for General Audio Representation
- Dual-balancing for multi-task learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- AST: Audio Spectrogram Transformer
- Cross-task learning for audio tagging, sound event detection and spatial localization: DCASE 2019 baseline systems
- Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
- Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling