Graph Attention Across Raw-Waveform Sub-Bands

Most audio classifiers never look at the raw waveform. The waveform arrives at 16,000 to 48,000 numbers per second, so it is first converted into a mel-spectrogram, a fixed picture of how much energy sits in each frequency band over time. A large image-style network then processes that picture. Front-ends that learn their own filters directly from the waveform tend to classify better, but they are either expensive or crude. The expensive kind uses long time-domain convolutions (Sainath et al.; Parcollet et al. / Zhang). The crude kind treats each frequency band in isolation.

This note proposes a middle path. First, split the waveform with a cheap learnable filterbank. Then sample each band only as often as its bandwidth requires, so low bands produce few time steps and high bands produce many. Finally, run graph attention across the resulting grid of (band, time) nodes. Graph attention here means a layer in which each node decides, from the content of its neighbors, how much to listen to each one. Because the neighbor weights depend on content, a harmonic sound can link its fundamental to its overtones even as the pitch moves, and a transient can link across bands at one instant. A fixed channel-mixing layer can do neither.

Core Claim

The strongest raw-waveform result in the file comes from the multi-head relevance weighting front-end of Parcollet et al.. It convolves the waveform with cosine-modulated Gaussian filters, splits the outputs into frequency "heads", and lets a tiny gating network rescale each head over time. With two heads it reaches 72.6% on DCASE2020 Task 1A and 63% on UrbanSound8K, against 52% for the mel baseline. The same paper reports that performance collapses when the number of heads grows to 80.

We read this collapse as the key evidence. Fine frequency resolution is useful only if narrow bands can exchange information, and independent gates give them no way to do so.

Our claim is that a sparse, content-dependent graph over multi-rate band-frame nodes does three things:

  1. It lets the front-end scale to many narrow bands without the collapse.
  2. It beats a static cross-band mixing layer with the same parameter count. The gain should be largest on pitched and harmonic material, where the useful cross-band links move from frame to frame.
  3. It does all this at lower total cost, counted end to end from the waveform, than a mel-spectrogram front-end followed by a 2D backbone at matched accuracy.

Existing graph audio models build graphs only after a heavy spectrogram encoder. ATGNN builds a k-nearest-neighbor graph over CNN feature-map patches. GraphAC runs top-k attention over pretrained-network frames. Neither touches the front-end cost.

Research Program

Building the graph nodes. The filterbank is the cosine-Gaussian design from Parcollet et al., reused as-is. We use K bands, with K swept from 16 to 128 and centers log-spaced. Each band is decimated in time to a hop inversely proportional to its bandwidth, roughly critically sampled, in the spirit of the length-halving wavelet split used by FreGrad. Each resulting (band, time) cell becomes a node. Its feature vector is small: log envelope, local energy slope, and a few pooled filter responses. With log-spaced bands the node count per second stays far below the sample rate. This avoids the obvious failure of attending over raw samples, where a 10-second clip would give 160,000 nodes and GAT-style edge cost of order nodes × neighbors × feature width would be out of reach.

Limiting who can connect. Each node gets a bounded candidate set of C neighbors, so cost is linear in node count. The candidates are its temporal neighbors within its own band, plus every node in other bands whose time span overlaps its own. This makes cross-band links, including harmonic ones, available by construction without any pairwise search.

Weighting the neighbors. Within the candidate set, edge weights are computed from node content as in Graph Attention Networks. The one change is that we normalize with sparsemax instead of softmax, which lets the number of active neighbors vary with the input:

$$\alpha_{ij} = \mathrm{sparsemax}_{j \in \mathcal{C}_i}\big(\mathbf{a}^\top[\mathbf{W}\mathbf{h}_i \,\|\, \mathbf{W}\mathbf{h}_j]\big).$$

Here $\mathcal{C}_i$ is node $i$'s candidate set, $\mathbf{h}$ are node features, $\mathbf{W}$ is a shared projection, and $\mathbf{a}$ is a learned scoring vector. Sparsemax sets low-scoring edges exactly to zero. As a result, silent regions can end up with almost no active edges while dense scenes keep many. This directly answers the fixed-sparsity limitation GraphAC reports for its $k=25$ top-k mask.

Guarding against smoothing. Two to four such layers are stacked with residual connections, so that repeated averaging does not wash out sharp onsets. The heterophily argument in Chikwendu et al. predicts exactly this smoothing failure for uniform averaging.

Back-end and cost accounting. The node grid is pooled per band into a sequence and fed to the same FCN or CNN-LSTM back-end that Parcollet et al. used, so the only thing that changes is the front-end. Cost is always reported from the raw waveform onward, counting multiply-accumulates, peak memory, and CPU latency.

We will not rely on claims that computing the STFT is itself the bottleneck (Sarthak et al.). An FFT is cheap. The real costs are the long time-domain filters and the 2D backbone (BirdSoundsDenoising; Liu and Cheng), and our accounting is designed to show where the cost actually goes.

Baselines

Naive reference. The do-nothing point is the fixed mel-spectrogram front-end with the same FCN back-end, as reported by Parcollet et al. (52% on UrbanSound8K). We re-run it in our pipeline with identical augmentation and training schedule, so that our own numbers are anchored to theirs.

Strongest raw-waveform competitor. This is Parcollet et al.'s cosine-Gaussian front-end with two relevance heads. They report 72.6% accuracy on DCASE2020 Task 1A and 63% on UrbanSound8K. We re-run it at 2, 16, 40, and 80 heads to reproduce the collapse, since no numeric value for the 80-head collapse is given in the file. Beating it means exceeding its best head count on both datasets at matched front-end compute, averaged over at least three seeds.

Strongest overall system. On FSD50K this is ATGNN, which reports 0.585 mAP with the pyramid-medium variant against 0.572 for AST, and 0.335 mAP on AudioSet-balanced. We do not expect to beat ATGNN's absolute mAP with a small back-end. Instead, we will attach our front-end to an ATGNN-sized back-end and compare accuracy against end-to-end compute.

Isolating the key ingredient: static versus dynamic weights. This ablation replaces the content-dependent weights $\alpha_{ij}$ with learned but input-independent weights on the same candidate graph. That is equivalent to a static cross-band plus temporal mixing layer with identical parameters and FLOPs. If this matches the dynamic version, our central claim fails.

Isolating the key ingredient: adaptive versus fixed sparsity. A second ablation swaps sparsemax for a fixed top-k mask in the style of GraphAC, with k tuned on validation data.

Fairness rules for all baselines. Every baseline receives the same data splits, the same number of hyperparameter trials (twenty), and a FLOP budget within 10% of ours.

Why not just…?

Why not just let the relevance heads share information through a static 1×1 convolution across bands? This is the most serious threat. Parcollet et al.'s collapse might be caused only by the lack of any mixing, rather than the lack of content-dependent mixing. The static-weight ablation above tests this directly. If static mixing closes the gap everywhere, we will report the front-end as a static multi-rate design and drop the dynamic claim. We expect the dynamic version to win specifically on pitched or harmonic classes and on inputs with changing pitch. The test is to stratify UrbanSound8K classes (sirens and music versus broadband noise) and to use a synthetic probe: tones with gliding pitch buried in noise, where the correct cross-band links shift over time.

Why not just use mel-spectrograms with AST or ATGNN? These are accurate: ATGNN reaches 0.585 mAP on FSD50K. But they spend most of their compute in a 2D backbone, which the file flags as too costly for constrained monitoring. Parcollet et al. also show learned raw-waveform front-ends outperforming mel on scene and urban sound tasks. Our experiments compare accuracy against end-to-end compute curves rather than peak accuracy alone. If mel plus a small CNN dominates our curve, the proposal fails on efficiency.

Why not just use dense self-attention or deep dilated convolutions over the band-frame grid? Dense attention costs grow with the square of node count. GraphAC also reports that unmasked attention over audio frames lets noise in. Dilated convolutions are linear in cost, but their receptive field is fixed, which ATGNN and GraphAC both identify as the limitation on non-local links. We will run both at matched FLOPs over the same multi-rate grid, so any difference comes from the aggregation rule and not from the node design.

Proposed Experiments

Main comparison. The core runs are on DCASE2020 Task 1A and UrbanSound8K, using official splits and the ten-fold protocol for UrbanSound8K. We report accuracy together with multiply-accumulates per second of audio, peak memory, and CPU real-time factor, all measured from the waveform.

Band-count sweep. The first decisive check sweeps K. We predict our front-end's accuracy rises or plateaus up to 80 to 128 bands, while two-head-style independent relevance gating degrades. We also predict the static-mixing ablation falls between the two.

Stratified and synthetic tests. The second decisive check is the stratified and synthetic-glide analysis described under the 1×1 convolution alternative. It tests whether the dynamic gain is concentrated where theory says it should be. We will also inspect the learned edges for harmonic ratios.

Sparsity behavior. The third check measures active-edge counts under sparsemax as a function of clip energy and event density. The test is whether sparsity actually adapts to content, which is the property GraphAC's fixed $k$ lacked, and whether it beats tuned top-k.

Scaling check. Finally, on FSD50K and AudioSet-balanced we attach the front-end to a larger back-end and plot mAP against end-to-end compute, alongside ATGNN and AST as reported.

What would count as success, and what would not. Success means matching two-head relevance accuracy with 64 or more bands, beating static mixing on harmonic subsets beyond seed variance, and sitting on or above the mel-plus-CNN accuracy-compute frontier. A null result on the static-versus-dynamic comparison would still be informative. It would show that the waveform front-end problem Parcollet et al. exposed is one of connectivity, not of content-dependent attention.