Search engines for text work because queries and documents share a vocabulary, which means a document can be stored in an inverted index (a table from each word to the documents containing it) and looked up instantly. Audio has no words. Today's audio search therefore maps sounds and sentences into a shared vector space and compares them by dot product. These vector systems have two documented problems. A geometric gap persists between where audio vectors and text vectors land. They also behave like a "bag of words" over sound events: they notice "glass breaking" and "siren" but ignore which came first. We propose to give each audio clip an explicit, sparse list of weighted English terms, the kind of list a text search engine can index directly. Audio and queries would then meet in one discrete vocabulary instead of across a geometric gap.
The mechanism has two steps. First, the clip is turned into a sequence of discrete "audio words": cluster labels of short sound frames, the classic bag-of-audio-words tokenization. Second, a bidirectional recurrent model reads that sequence forwards and backwards and decides which text terms each audio word stands for in context. The model also emits a small set of ordered event-pair terms such as "glass>siren". The bet is that an impulsive burst is ambiguous on its own (gunshot, door slam, firecracker), but what precedes and follows it usually settles the question. An offline index always has the whole clip, so the usual objection to bidirectional models, that they cannot stream, does not apply here.
Core Claim
A context-free "audio word to text term" translation table, the obvious lexical approach, fails for two reasons: acoustic clusters are polysemous, and orderless histograms cannot express sequence. Both failures can be repaired by one bidirectional linear recurrent encoder over the audio-word sequence. The result is a sparse text-vocabulary index for audio that should match strong dense retrievers on standard caption retrieval and beat them on order-sensitive queries. The vocabulary mismatch between queries and captions is handled on the audio side, by spreading weight across related terms, rather than by rewriting queries. What is new is the combination of three things: retrieval happens in the text vocabulary itself, audio words are disambiguated using both past and future context, and order is encoded as indexable discrete terms.
Relation to Prior Work
Two papers motivate the gap. Dissecting Temporal Understanding in Text-to-Audio Retrieval shows that dense audio-text retrievers score correctly ordered and reversed queries identically. It repairs this with a text-text contrastive loss against reversed paraphrases and a diagnostic set, SynCaps. We adopt its benchmark and its reversed-caption negatives. We differ in representing order as discrete pair terms in an index rather than as directions in a dense space. Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation shows that the audio-text modality gap persists under contrastive training, and narrows it with optimal-transport matching. We sidestep the gap by never comparing audio and text vectors at all; both sides become weights over the same word list.
Three lines of work supply the method. The bag-of-audio-words literature shows that unsupervised acoustic tokens are strong retrieval features. In Event-based Video Retrieval Using Audio, a 16,000-word MFCC K-means bag beat automatic speech recognition transcripts on event detection. A Text Retrieval Approach to Content-Based Audio Hashing applies inverted indices to audio tokens. Both papers discard order, which is the defect we fix. For the encoder, Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion introduces BLUR, a bidirectional linear recurrence trainable with parallel scans in linear time, which keeps indexing cheap for long clips. AudioCaps shows that a bidirectional LSTM encoder with word-to-segment alignment reduces caption hallucination. That model works on continuous frames and generates captions, whereas we use discrete tokens and produce an index. For context-dependent term weighting, Improved Neural Bag-of-Words recovers out-of-vocabulary names from speech by learning per-word importance against a context anchor. We generalize the idea from reweighting existing words to translating non-linguistic tokens into words.
Two papers constrain the design. UniCode unifies image tokens with an LLM vocabulary for generation. We borrow its premise that a shared discrete vocabulary can close a modality gap, but we target retrieval and use the existing text vocabulary rather than learning a new joint codebook. The generative retrieval work, MINDER and GLEN, reports identifier hallucination and collisions. That is our reason for choosing a sparse index over generating clip identifiers.
Research Program
Frames from a frozen audio encoder are clustered into $K$ audio words, and each clip becomes a run-length-collapsed token sequence. Each audio word $k$ first receives a context-free prior over text terms, which serves as a translation table. A BLUR-style bidirectional encoder then produces, at each position $t$, a fused state $h_t=[h^f_t;h^b_t]$ built from a forward pass $h^f_t$ and a backward pass $h^b_t$. It outputs non-negative corrections to that prior. A clip $a$ is scored against a query $q$ as
$$s(q,a)=\sum_{v\in V} w_q(v)\,\max_t \mathrm{softplus}\big(\pi_{k_t}(v)+u_v^\top h_t\big),$$
where $V$ is the text vocabulary extended with ordered pair terms, $w_q(v)$ is the query's weight on term $v$ (inverse document frequency, plus a light learned reweighting), $\pi_{k_t}(v)$ is the prior for the audio word at position $t$, and $u_v$ is a learned term vector. Pair terms are emitted when two segment-level top terms occur in order. Queries are mapped to pair terms by a small parser over connectives such as "followed by", "before", and "after". Training uses contrastive loss on AudioCaps and Clotho, the reversed-caption negatives from the temporal-dissection paper, and a penalty on total term mass so that the index stays sparse. At test time the term weights go into an ordinary inverted index.
The analysis has three parts. We will measure how often the bidirectional context changes an audio word's top term. We will check whether those changes concentrate on high-entropy clusters, which is the polysemy hypothesis. We will also test whether expansion terms cover query words absent from a clip's own caption, which is the vocabulary-mismatch hypothesis.
Baselines
The naive reference point is an orderless bag-of-audio-words with TF-IDF weighting, where each audio word is translated through the context-free prior table only. This system is our model with the bidirectional encoder removed. It is trained on the same data with the same codebook. Beating it isolates the value of context.
The strongest dense competitors are two. The first is m-LTM, which reports 39.10% text-to-audio R@1 on AudioCaps and 81.0% zero-shot accuracy on ESC-50 (m-LTM). The second is CED-Base with SONAR text encoders, which reports audio-to-text R@1 of 60.7 on AudioCaps and 30.9 on Clotho (Bridging Language Gaps). We will retrain a dense dual encoder on the same frozen audio frontend and the same training pairs as ours, giving it the same tuning budget. That controls for the frontend, so differences reflect the retrieval representation. On order, the strongest method is the dense model trained with the text-text loss from Dissecting Temporal Understanding. That paper reports R@1 on wrongly ordered SynCaps queries dropping from about 67% to 40%, meaning the model stops retrieving reversed matches. We will reproduce this baseline with identical negatives.
We count it as beating the baselines if we stay within two R@1 points of the retrained dense model on AudioCaps and Clotho, while showing a larger gap between correctly and wrongly ordered queries on SynCaps than the text-text-loss baseline.
Three ablations accompany these comparisons. A causal-only encoder tests whether future context matters. Continuous frames fed to the same encoder test whether quantization helps or merely costs accuracy. Removing pair terms tests whether order needs explicit discrete terms.
Why not just…?
Why not just caption each clip and index the captions with a text search engine? This is the most serious threat, because it also puts audio into text space. A captioner such as TopDown-AlignedAtt from AudioCaps reaches CIDEr 59.3 against a human score of 91.3. We found no reported result on Lacuna for caption-then-text-search retrieval on AudioCaps or Clotho. Our concern is that a single generated sentence commits to one phrasing, so the vocabulary mismatch reappears whenever the query uses different words. We will run this baseline with the same frozen frontend, using both one caption and five sampled captions per clip. If five sampled captions match our model, our claimed contribution shrinks to efficiency and order handling, and we will report it that way.
Why not just add audio-word bigrams to the bag? N-gram counts capture adjacent order at almost no cost, but run-length-collapsed bigrams of polysemous clusters inherit the clusters' ambiguity and cover only immediate neighbors. The "A before B" queries in SynCaps involve events separated by other sounds. A TF-IDF bag with bigrams and trigrams is included as a baseline. If it closes most of the SynCaps gap, then the gain we claim comes from the pair terms rather than from bidirectional disambiguation, and we will say so.
Why not just keep a dense retriever and expand queries with an LLM? Query rewriting does not change the audio-side representation, which is where the temporal blindness lives (Dissecting Temporal Understanding). We will nonetheless test LLM-expanded queries against the retrained dense model on a mismatch split. That split holds out clips whose test queries use content words absent from their training captions. Comparing it against our document-side expansion shows which side of the retrieval system the fix belongs on.
Proposed Experiments
The main evaluation covers text-to-audio and audio-to-text R@1, R@5 and R@10 on AudioCaps and Clotho, and mAP@10 on Clotho, where the hybrid model from WavText5K reports 27.12. It also covers zero-shot ESC-50 classification, in which class names are scored as one-term queries. For order, we use SynCaps correct and reversed queries and the reordered AudioCaps queries from Dissecting Temporal Understanding, split by whether the two events are adjacent or separated. For robustness, we follow the m-LTM protocol of 60% caption misalignment, where m-LTM reports 27.73% R@1 against 20.58% for contrastive loss.
The decisive checks are three. The bidirectional model must beat its causal ablation on SynCaps with separated events. Context must reduce error mainly on high-entropy audio words. And the sparse index must stay within the stated margin of the matched dense model on AudioCaps while using an inverted index. If the first check fails, bidirectionality is not earning its place. If the third fails badly, the lexical route costs too much accuracy to be worth it.
References
- Dissecting Temporal Understanding in Text-to-Audio Retrieval
- Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation
- Event-based Video Retrieval Using Audio
- A Text Retrieval Approach to Content-Based Audio Hashing
- Bidirectional linear recurrent models for sequence-level multisource fusion
- AudioCaps: Generating Captions for Audios in The Wild
- Improved Neural Bag-of-Words Model to Retrieve Out-of-Vocabulary Words in Speech Recognition
- UniCode: Learning a Unified Codebook for Multimodal Large Language Models
- Multiview Identifiers Enhanced Generative Retrieval
- GLEN: Generative Retrieval via Lexical Index Learning
- Bridging Language Gaps in Audio-Text Retrieval
- Audio Retrieval with WavText5K and CLAP Training