Choosing Which Moments of a Song to Keep: Sampling by Coverage of Discovered Concepts

Audio-language models answer questions about music, but they can only process a fixed number of audio tokens. For a full song they usually keep evenly spaced slices and throw away the rest. That works badly for music, because the time spent on something and its importance are unrelated. A two-bar guitar break, a single modulation, or a brief spoken intro can decide the answer to a question, while a minute of repeated chorus adds almost nothing. We propose a different way to spend the budget. First, discover the model's own vocabulary of musical concepts without labels, using a sparse autoencoder (a small network that rewrites each dense activation vector as a few active entries drawn from a large learned dictionary). Then choose the audio segments that together cover every concept present in the track, instead of segments spaced evenly in time.

The central bet is that sparse concept codes count presence rather than duration. A timbre or event that appears for three seconds becomes one dictionary entry that must be covered, exactly like a texture that lasts three minutes. We also sort the discovered concepts into tiers by how long each one stays active once it fires. This gives a data-driven hierarchy of short events, phrase-level patterns, and section-level states, with no fixed window lengths. The sampler uses the slow tier to divide the track into sections and the fast tier to decide what to keep inside each section. The contribution is a training-free, interpretable selector that plugs into a frozen music model. It comes with a controlled test of whether discovered concepts carry information that simpler signals, such as raw embedding change or query similarity, do not.

Core Claim

Under a fixed audio-token budget, choosing segments to maximize coverage of the sparse concepts active in a track will beat uniform sampling, random masking, and query-similarity selection on music questions that depend on brief or rare events. It will do so without losing accuracy on questions about global properties. The claim is specific to the sparse, discrete nature of the codes. The same coverage objective computed on dense embeddings should do worse, because distances between dense embeddings are dominated by whatever occupies most of the audio. If it does not do worse, the concept-discovery ingredient is unnecessary and the proposal reduces to diversity sampling. We treat that outcome as a real possibility and test it directly.

Where This Sits

Three lines of work motivate the proposal. Discovering and Steering Interpretable Concepts in Large Generative Music Models trains sparse autoencoders on MusicGen activations over 10-second clips. It recovers named and unnamed concepts ("Taiko Drums", "Electronic Beeps and Boops") and shows that some of them steer generation. It treats the features as a flat set within each layer and never uses them to decide what to process. We reuse its technique but apply it to a model that is doing understanding, and we put the codes to operational use. Concept-Guided Interpretability via Neural Chunking shows, mostly in text, that activations form discrete chunks that fire at structural boundaries. That supports our use of slow-tier concept switches as section boundaries, though it has not been tested on polyphonic audio. Discovering "Words" in Music finds recurring symbolic motifs by a shortest-coding-length principle and reaches 0.61 IoU against expert structure annotations. It shows that compression-style discovery recovers musical hierarchy, but only on clean scores. We move the idea to audio activations.

The sampling side supplies the competitors. Flexible Frame Selection for Efficient Video Reasoning learns a query-conditioned selector over a uniformly downsampled candidate pool. SlowFast-LLaVA combines sparse detailed frames with dense coarse frames. Both are query-driven or time-driven, and neither can say why a segment was kept. Learning Sparsity for Effective and Efficient Music Performance Question Answering randomly masks half of the audio and video patches and still reports 81.75% on MUSIC-AVQA.

This result partly constrains our proposal. If random dropping already helps, the redundancy in music is large, and the burden on any smart selector is to beat random at much tighter budgets than 50%, not at loose ones. On the hierarchy side, MART and Learning Multi-Level Representations for Hierarchical Music Structure Analysis learn multi-scale representations with fixed dyadic or four-window time scales. Our tiers come from how long each concept persists, not from preset windows. The Buisson et al. boundary model, which reaches L-recall of about 0.68–0.69 on SALAMI, serves as a boundary-based competitor.

The closest idea in the file is the "concept-guided frame selection" gap it lists itself. We found no paper that implements it. Our version differs from a generic "gate on concepts" selector in two ways: the objective covers the set of concepts rather than scoring each segment's salience, and the concept tiers are sorted by persistence.

Research Program

We freeze the audio encoder of an open audio-language model, with Qwen2-Audio as the primary host because the file reports its results on the target benchmarks. We collect its activations over a large unlabeled music corpus, using MusicSet-scale audio (about 160,000 clips) plus full-length tracks. We then train a top-k sparse autoencoder on one mid-depth layer, choosing the layer by reconstruction quality and by how interpretable its features are. For each learned concept $j$ we measure its persistence $\tau_j$: the median length of a contiguous run during which the concept stays active. We place concepts into fast, medium, and slow tiers by clustering $\log \tau_j$. We do not fix three tiers in advance. If the persistence distribution is not multimodal, we report that, because it would itself be evidence against a clean timescale hierarchy.

At inference, a track is cut into short segments $s$, and each segment gets its set of active concepts $A(s)$. Slow-tier concepts segment the track wherever the dominant slow concept changes. Within the budget $B$, the selector picks segments $S$ to maximize a weighted coverage

$$F(S) = \sum_{j} w_j \, \min\Big(1, \sum_{s \in S} a_{sj}\Big),$$

Here $a_{sj}$ is the activation of concept $j$ in segment $s$, and $w_j$ is a weight that rises with the concept's rarity in the track, so concepts that appear only briefly count as much as long-running ones. The selector also enforces a minimum of one segment per slow-tier section, so no section drops out entirely. Because $F$ is a coverage function, greedy selection comes within a constant factor of the best achievable coverage. Selection costs only one pass of the frozen encoder, which is far cheaper than the language model.

The kept segments go to the language model in temporal order, with their timestamps. Every kept segment can be explained by listing the concepts it was kept to cover. Following the SAE paper, we label those concepts by feeding each concept's top-activating audio segments to automatic taggers, knowing that CLAP scores abstract musical concepts poorly.

Baselines

The naive reference point is uniform sampling at the same token budget. This is the default in practice, and it is what FFS compares against; on NExT-QA video, FFS reports 62.2% against 61.6% for a 32-frame uniform baseline. Alongside it we run Sparsify-style random masking at matched budget, because Sparsify shows random dropping of audio patches is not harmful at 50%. A proposal that cannot beat random at 25% or 12.5% of frames has no case.

The strongest existing selection method is query-aware selection in the style of FFS and the training-free video–query–options similarity approach of A Training-Free Framework for Long Video Understanding via Video-Query-Options Similarity. We implement it by scoring each audio segment against the question and the answer options with an audio-text embedding and keeping the top segments. It is the strongest competitor because it sees the question, which our selector does not. We run it without the learned FFS gate, since we have no music training data for that gate. We report this honestly as a training-free adaptation. We found no reported result on Lacuna for any frame- or segment-selection method on music audio question answering. The nearest music reference numbers are Music Flamingo's 74.58% on MuChoMusic and Qwen-Audio's 51.4%, from Music Flamingo and MuChoMusic. These are full-input ceilings and reference points, not selection baselines.

We add a structure-boundary baseline. It uses the multi-level boundary detector of Buisson et al. (L-recall about 0.68–0.69 on SALAMI) and spends the budget evenly across the detected sections.

The ablation that isolates our key ingredient is dense-embedding diversity selection. It runs the same greedy procedure, but maximizes facility-location coverage over the encoder's dense activations rather than the sparse codes, at the same budget and the same segment size. We also run our method with the persistence tiers removed, which gives flat coverage, and with rarity weights set equal.

All methods share the frozen host model, the segment size, the budget grid, and the prompt format. Hyperparameters for every selector are tuned on the same held-out split with the same number of trials. Beating a baseline means a gain in accuracy at matched budget that holds across three seeds and bootstrap confidence intervals, measured at 25% and 12.5% of frames.

Why not just…?

Why not just use a longer context and uniform sampling? At generous budgets this may be fine, and Sparsify's result suggests music tolerates heavy random thinning. The case for selection rests on tight budgets and brief events, where SPORTU and UAL-Bench show that evenly spaced sampling misses short events in video. Our budget sweep reports where the curves cross. If uniform sampling matches us at 25%, the method is useful only in extreme compression regimes, and we will say so.

Why not just detect change points in the raw embeddings? This is the most serious threat. Novelty curves on embeddings find transitions cheaply, and the dense-diversity ablation is effectively this baseline plus coverage. Our argument is that change detection finds boundaries, whereas a rare event inside a stable section, such as a single vocal sample over a loop, produces a small dense shift that is easy to miss. The same event gets its own sparse concept that coverage must include. The inserted-event probe below tests exactly this. If dense diversity ties us there, we conclude the concept dictionary adds interpretability but not accuracy.

Why not just select by similarity to the question? Query-aware selection is powerful when the question names what to look for. It fails when answering needs context the question does not mention, such as the key established earlier, or which section came first. The file lists this "tunnel vision" as an open limitation of query-driven samplers. We split benchmark questions into those that name a local event and those that ask about order or global properties, and we report the two groups separately. We expect the query-aware baseline to win the first group. We also test the natural hybrid, where concept coverage reserves half the budget and query similarity fills the rest.

Proposed Experiments

The main evaluation runs on MuChoMusic, using its full-track questions, the MMAU music subset from MMAU, and the audio stream of MUSIC-AVQA. MMAU clips are at most 40 seconds, so we use it mainly at very tight budgets.

The decisive diagnostic is a controlled inserted-event probe built from full-length SALAMI and RWC tracks. We splice a short foreign event, between 1 and 4 seconds long, such as an instrument or sound absent from the rest of the track, into the middle of a section. We then ask whether it occurred, what it was, and in which section. This separates the hypotheses cleanly. Uniform sampling misses the event with a probability set by the budget. Change-point and dense diversity methods catch it only if it perturbs the embedding enough. Concept coverage should keep it whenever the autoencoder has a feature for it.

A second diagnostic uses SALAMI section labels to ask questions about section order. This tests whether keeping at least one segment from each slow-tier section preserves global structure.

To separate listening from guessing, we rerun every selector with the audio replaced by noise, following the MMAU and MuChoMusic protocol. Gains that survive this control are not real listening gains and will be discounted.

Beyond accuracy, we report three analyses. First, how often the kept segments contain the answer-bearing moment, using the known location of inserted events and SALAMI boundaries. Second, whether the persistence tiers match annotated upper and lower SALAMI boundaries better than fixed windows do. Third, a small human check of whether the concept listed as the reason for keeping a segment is audible in it.

The proposal succeeds if concept coverage beats both random masking and dense-embedding diversity on the inserted-event probe and on MuChoMusic at 12.5% of frames, and stays within noise of the query-aware baseline on local questions. It fails in an informative way if dense diversity matches it. That would show the benefit comes from coverage alone, and that unsupervised concept discovery earns its place here only as an explanation of the selection, not as a performance mechanism.