A common way to find where a short query, such as a spoken keyword, occurs inside a long recording is to draw a picture. The query's frames run along one axis and the recording's frames along the other, and each pixel holds how similar the two frames are. A true occurrence appears as a bright, roughly diagonal streak. Its slope bends when the speaker talks faster or slower, and it breaks where there is a pause or noise. Training a convolutional network (an image classifier built from small sliding filters) to spot these streaks beats classic dynamic time warping, the step-by-step alignment algorithm that has long been the default for this matching problem. The convolutional approach has one structural weakness: it needs the whole picture in advance. CNN Based Query by Example Spoken Term Detection resizes every matrix to a fixed 200×750 image. Temporal Alignment Prediction predicts an alignment from the full matrix. CB-Whisper runs a small detector over precomputed keyword-versus-speech matrices. When audio keeps arriving, the picture grows by one column per frame and never ends, and these models must re-scan windows of it.
We propose replacing the image classifier with a recurrent network, a model that keeps a running memory and updates it one step at a time. This network sweeps across the similarity matrix in the direction the stream arrives. Its memory is a small vector for every query position. Each vector can only receive information from positions that are earlier in both the query and the stream, so the model counts only in-order matches by construction. Learned gates decide when to hold a partial match through a pause, when to discard it as noise, and when to advance it. In effect, this is a learned, multi-channel, differentiable version of streaming dynamic time warping. It costs a fixed amount of computation per incoming frame, proportional to query length, and can hold a partial match open for an unbounded time. The central empirical question is whether it beats a well-engineered streaming convolutional network. That network is the obvious and strong alternative, and the experiments are designed around it.
Core Claim
Existing similarity-matrix detectors handle local noise and speed changes well. They do so with a fixed receptive field, the region of the image a filter stack can see at once, and they do not enforce chronological order. Existing recurrent networks keep unbounded state but read only one sequence, and they have been shown to model warping between sequences poorly. The multitask study Multitask Learning for Time Series Data with 2D Convolution argues that one-dimensional models carry a "locality bias" that prevents them from approximating recursive warping. The retrieval system An Efficient Content-based Time Series Retrieval System used LSTMs and GRUs only as sequence embedders, and they lost to its 2D template model.
Our claim is that both failures are about where the recurrence runs, not whether recurrence is useful. A recurrence placed on the two-dimensional alignment grid, with transitions restricted to the three monotone moves of dynamic time warping, should do two things. It should match the detection quality of 2D convolutional detectors on ordinary queries. It should also clearly beat them in two specific regimes: when a pause or slowdown stretches an occurrence beyond the convolution's receptive field, and when a distractor contains the query's pieces in the wrong order.
The Mechanism
Let $s_{i,t}$ be a small feature vector at query position $i$ and stream frame $t$. It is computed by a shallow convolution over the similarity values, and that convolution is causal: it looks only at stream frames up to $t$. Let $h_{i,t} \in \mathbb{R}^d$ be the scanner's state there. A softmax gate $a_{i,t} \in \Delta^3$, computed from $s_{i,t}$, mixes the three monotone predecessors into a single carried state:
$$m_{i,t} = a^{(1)}_{i,t} h_{i-1,t-1} + a^{(2)}_{i,t} h_{i,t-1} + a^{(3)}_{i,t} h_{i-1,t}.$$
The first term is a diagonal step, the second is a "stay" that absorbs a pause in the stream, and the third is a skip over a query frame. An update gate $g_{i,t} \in [0,1]^d$ then blends this carried state with fresh evidence:
$$h_{i,t} = g_{i,t} \odot m_{i,t} + (1-g_{i,t}) \odot \tanh(W s_{i,t} + U m_{i,t}).$$
The boundary row $h_{0,t}$ is a learned start vector at every $t$, so a match may begin anywhere in the stream. The detection score at frame $t$ is read from $h_{N,t}$, where $N$ is the query length.
Two properties follow directly from this construction. First, information never flows from a later query position to an earlier one, or from a later frame to an earlier one, so every detection corresponds to an in-order path. Second, when the stay weight and the update gate both saturate near one, the state passes through arbitrarily many pause frames without decaying. This borrows the intuition of the time gate in Phased LSTM, which keeps memory intact across long closed intervals, but here the gate is driven by the data rather than by a clock.
If the gates are replaced with a fixed soft-minimum over scalar costs, the model reduces exactly to soft subsequence dynamic time warping. That reduction gives us a clean ablation.
At inference, each new frame requires one pass over the $N$ query positions, and memory is constant in stream length. In training, cells on the same anti-diagonal are independent. The recurrence can therefore be computed as a wavefront with $N+M$ sequential steps rather than $N \cdot M$, where $M$ is the stream length. This addresses the parallelism concern that TAP raised against dynamic-programming alignment, although it does not fully remove it.
Relation to Prior Work
Three papers supply the case that similarity matrices plus learned local detectors beat hand-built alignment: the QbE-STD CNN, CB-Whisper, and TAP. We keep their input representation and replace their stateless detector with a stateful, order-respecting one.
TAP also motivates the monotonicity constraint. It predicts an alignment matrix with a 3-layer CNN, and the file notes that this prediction does not enforce chronological order. We enforce order architecturally. Unlike TAP, we accept that query-to-stream matching is asymmetric, since detection is naturally one-directional, and we do not claim a symmetric distance.
Two papers constrain the design. The first is DTWNet, which embeds exact dynamic programming as a network layer and includes a streaming subsequence variant. Its forward pass is still scalar dynamic programming, and it differentiates only along the single best path. We keep the monotone lattice but make the state multi-channel and every transition learned and differentiable. The second is Matrix-LSTM, the closest structural relative. It places a grid of LSTM cells over an event camera's pixel grid. Its axes are image coordinates, its cells share weights with translation invariance, and cells do not pass state to neighbours along alignment moves. Our grid's axes are the two time axes, and state flows only along the three alignment moves.
Finally, the 1D-locality-bias result from Multitask Learning for Time Series Data with 2D Convolution looks like it contradicts any recurrent proposal. We read it as evidence against recurrence along one sequence only, and our ablations test that reading directly.
Research Program
We will build the scanner on the input features used by the QbE-STD CNN, log dot products of phonetic posteriors. This makes the representation identical to the strongest published matrix detector, so any difference comes from the detector. We will also build it on Whisper-encoder cosine similarities, following CB-Whisper, for the keyword-biasing setting.
There are three analyses. The first asks whether the learned transition weights $a_{i,t}$ recover interpretable alignment paths. We will compare their arg-max path against subsequence dynamic time warping on clean matches. The second asks how the model behaves as pauses grow, measuring detection score as a function of inserted silence length. The third asks whether the monotone structure actually suppresses out-of-order distractors, or whether a convolutional detector trained with the same negatives learns order just as well.
Baselines
The naive reference point is streaming subsequence dynamic time warping on the same posterior features, in the spirit of the SPRING algorithm used by DTWNet, with a threshold on path cost. On SWS 2013 the file reports plain DTW at minCnxe 0.7181 and MTWV 0.3352 (minimum normalized cross-entropy, where lower is better, and maximum term-weighted value, where higher is better) CNN Based Query by Example Spoken Term Detection. Any learned method must clear this.
The strongest existing method is the VGG-style CNN from the same paper, which reports minCnxe 0.6483 and MTWV 0.3753 on SWS 2013. It is the right competitor because it uses exactly our input matrices and is the published evidence that image-style detection beats alignment. We will first reproduce it offline on full matrices. We will then run it in streaming form: causal in the stream axis, sliding over the stream with activation caching, and with receptive-field width chosen by validation. This way the comparison measures state against finite windows rather than streaming against offline. Both models get the same features, the same training negatives including out-of-order distractors, and the same hyperparameter search budget. We will report per-frame compute alongside accuracy.
Beating this baseline means one of two outcomes. The first is equal or better minCnxe and MTWV on the standard SWS 2013 streaming protocol at no greater per-frame compute. The second is a clear advantage on the pause-stretched and reordered stress sets while standard performance is within noise.
For keyword biasing on the Aishell hot-word subset, the competitor is CB-Whisper's CNN detector, which as part of the full system raised entity recall from 8.4% to 71.8% CB-Whisper. We swap only the detector and keep the Whisper prompting pipeline fixed. For QUESST 2014 there is no reported result found on Lacuna, so we will report our reproduced CNN numbers as the reference.
Four ablations isolate what matters. Fixing the gates to a scalar soft-minimum isolates learned multi-channel state. Removing the stay predecessor isolates pause handling. Adding a fourth, non-monotone predecessor $h_{i+1,t-1}$ isolates the value of the ordering constraint. Running an LSTM over the stream alone, reading the query through a pooled embedding, tests whether the recurrence must live on the 2D grid.
Why not just…?
Why not just run the 2D CNN causally with cached activations? This is the real threat. A causal convolution with caching is also constant-cost per frame, and for ordinary keywords its receptive field may cover every occurrence. If it matches the scanner on standard SWS 2013 and on the stress sets, the proposal is wrong about state mattering and we will say so. We expect the difference to appear only when an occurrence spans more frames than the receptive field, which the silence-insertion sweep measures directly. If it does appear there but nowhere natural, the honest conclusion is a niche win, and the paper will be framed that way.
Why not just use soft-DTW on learned features? The file reports that TAP runs about 10x faster than soft-DTW, and DTWNet describes soft-minimum relaxations as memory-hungry and numerically unstable. Soft-DTW is also exactly our fixed-gate ablation. If it ties the learned scanner, the learned gates add nothing, and the right system is soft subsequence DTW on good features.
Why not just put a recurrent network over one sequence, or use cross-attention? The retrieval study found LSTMs, GRUs and Transformers all below RN2Dw/T's 0.9336 NDCG@10 on the 128-dataset UCR archive An Efficient Content-based Time Series Retrieval System. CB-Whisper reports that feeding long keyword lists into Whisper's cross-attention caused hallucinated keywords. Our stream-only LSTM ablation re-checks the first finding under matched features.
Proposed Experiments and Decisive Checks
The main experiment is streaming query-by-example detection on SWS 2013 and QUESST 2014. We report minCnxe, MTWV, detection delay after the query's end, and floating-point operations and memory per frame. The second experiment is hot-word biasing on Aishell with the CB-Whisper pipeline, reporting entity recall and mixed error rate.
Two stress sets built from SWS 2013 test the claim directly. The first inserts silences or stretches of speech lasting 0.5 to 10 times the query length inside true occurrences. The second builds distractors by concatenating the query's halves or thirds in the wrong order.
As an offline sanity check on the claim that the monotone lattice learns real alignments, we will use the scanner's final-state score as a sequence distance on Spoken Arabic Digits. There TAP reports 0.961 MAP Temporal Alignment Prediction. We do not expect to exceed this, because TAP sees the whole sequence; we only need to show the scanner is competitive.
The decisive check is the gap between the scanner and the cached causal CNN as a function of inserted pause length, together with the reorder false-alarm rate against the non-monotone ablation. The idea is confirmed if the gap grows with pause length and the ordering constraint lowers reorder false alarms with standard accuracy intact. It is refuted if both curves are flat.
References
- CNN Based Query by Example Spoken Term Detection
- Temporal Alignment Prediction for Supervised Representation Learning and Few-Shot Sequence Classification
- CB-Whisper: Contextual Biasing Whisper Using Open-Vocabulary Keyword-Spotting
- Multitask Learning for Time Series Data with 2D Convolution
- An Efficient Content-based Time Series Retrieval System
- Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
- DTWNet: a Dynamic Time Warping Network
- A Differentiable Recurrent Surface for Asynchronous Event-Based Data