When we compare two sequences—whether they are audio signals, video frames, or financial time series—our traditional instinct is to align them step-by-step. Classic algorithms like Dynamic Time Warping (DTW) or recurrent neural networks attempt to track temporal states sequentially, calculating a mathematical alignment path from start to finish. However, these sequential approaches are often computationally expensive, sensitive to localized noise, and difficult to parallelize or differentiate.
A powerful alternative paradigm completely reframes sequence alignment: it transforms the 1D temporal problem into a 2D spatial pattern recognition task. By plotting the pairwise relationships between two sequences onto a coordinate grid, temporal alignment leaves a distinct geometric signature—a diagonal line. This allows us to bypass step-by-step tracking entirely and use computer vision architectures to "see" the match.
From Sequences to Surfaces
To understand why this works, imagine taking two sequences, $X$ and $Y$, and laying them out along the axes of a two-dimensional grid. The x-axis represents time steps in sequence $X$, and the y-axis represents time steps in sequence $Y$.
For every coordinate $(i, j)$ on this grid, we compute a scalar value representing how similar (or how far apart) element $X_i$ is from element $Y_j$. Typically, this is done using a simple dot product, cosine similarity, or Euclidean distance. The result is a 2D matrix that maps the global landscape of correspondences between the two signals.
Sequence Y
▲
│ ┌───┬───┬───┐
│ Y_3 │ │ │ ■ │ ◄── High similarity at (X_3, Y_3)
│ ├───┼───┼───┤
│ Y_2 │ │ ■ │ │ ◄── High similarity at (X_2, Y_2)
│ ├───┼───┼───┤
│ Y_1 │ ■ │ │ │ ◄── High similarity at (X_1, Y_1)
│ └───┴───┴───┘
│ X_1 X_2 X_3
└────────────────────────► Sequence X
If sequence $X$ contains the exact same content as sequence $Y$, then $X_1$ matches $Y_1$, $X_2$ matches $Y_2$, and so on. On our 2D grid, these high-similarity points form a perfect diagonal line running from the bottom-left to the top-right.
If sequence $X$ is a sped-up version of sequence $Y$, the line remains continuous but changes its slope. If there are pauses, insertions, or deletions, the line shifts or bends. In every case, a meaningful temporal alignment manifests as a continuous, quasi-diagonal path through the matrix, while random noise appears as scattered, unstructured dots.
Why Convolutions Outperform Step-by-Step Alignment
Instead of using a rigid, hand-crafted path-finding algorithm to trace this line step-by-step, we can treat the similarity matrix as a single-channel image. Convolutional Neural Networks (CNNs) are inherently designed to detect spatial structures like lines, edges, and textures.
When a 2D CNN processes a similarity matrix, its convolutional filters act as learnable alignment detectors:
- Robustness to Noise: A localized drop in signal quality might break a strict frame-by-frame matching path. However, a 2D convolutional filter looks at a broader spatial context, allowing it to bridge small gaps and recognize the global texture of a line despite local imperfections.
- Handling Temporal Warping: Because CNNs learn hierarchical representations, they can recognize warped, curved, or shifted diagonals without requiring explicit mathematical constraints on how fast a signal can accelerate or decelerate.
- Efficiency and Differentiability: Computing a pairwise matrix and passing it through a feedforward CNN avoids the slow, iterative loops of traditional alignment algorithms. This makes the entire pipeline fully differentiable and highly parallelizable on modern hardware.
Applications Across Modalities
This spatial perspective has proven highly effective across a wide variety of domains, replacing complex sequence modeling with lightweight visual classification.
Speech and Keyword Spotting
In spoken term detection, traditional systems struggle to match a short spoken query to a long audio file due to variations in speaker pace and background noise. Miculicich et al. demonstrated in CNN Based Query by Example Spoken Term Detection and subsequent work (Miculicich et al.) that treating audio similarity as a 2D image allows a CNN to classify matches based on global textures rather than step-by-step path calculations.
This geometric approach enables highly efficient open-vocabulary keyword spotting. For instance, Li et al. in CB-Whisper and Li et al. in KWS-Whisper utilized lightweight CNNs to detect diagonal patterns between audio features and target keyword signatures generated by text-to-speech engines, boosting entity recall without retraining large foundation models. The mechanism even crosses modalities: Momeni et al. applied it to zero-shot audio-visual keyword spotting in Seeing wake words, using a CNN to find diagonal patterns that match phonetic text sequences directly to video frames of lip movements.
Video Retrieval and Cross-Modal Matching
Video analysis presents unique alignment challenges, such as varying frame rates, partial overlaps, and edited sequences. To capture fine-grained spatio-temporal relations, Kordopatis-Zilos et al. introduced ViSiL (Kordopatis-Zilos et al.), a model that feeds a frame-to-frame similarity matrix into a CNN to identify flexible temporal alignment patterns. The authors later adapted this concept to audio tracks in Audio-based Near-Duplicate Video Retrieval with Audio Similarity Learning, demonstrating that CNNs can recognize structural sequences even when the audio has undergone significant speed transformations.
For complex cross-modal tasks like video background music recommendation, Li et al. showed in Video Background Music Recommendation Based on Multi-Level Fusion Features that treating a cross-modal dot-product similarity matrix as a 2D image outperforms traditional rigid mathematical projections by preserving fine-grained regional details. Similarly, in music generation, Huang et al. used frame-by-frame cosine similarity matrices in Generating Music Medleys via Playing Music Puzzle Games to capture how internal musical structures evolve and repeat across different segments.
General Time Series Retrieval
In multi-domain time series retrieval, learning the warping mechanism natively is crucial. Yeh et al. (Temporal Treasure Hunt) and Chen et al. (An Efficient Content-based Time Series Retrieval System) modeled time series similarity by constructing 2D pairwise distance matrices, transforming sequential comparison into a spatial pattern recognition problem solved via 2D Residual Networks (ResNets).
By treating these distance matrices as images, Fan et al. demonstrated in Multitask Learning for Time Series Data with 2D Convolution that a 2D convolutional architecture can effectively learn to approximate complex functions like DTW. Taking this a step further, Su et al. developed Temporal Alignment Prediction (Su et al.), which constructs multi-channel matrices capturing both feature distances and temporal offsets. This allows a feedforward convolutional network to directly predict optimal correspondences, enabling fast, end-to-end representation learning.
Conclusion
By mapping 1D sequences into a 2D space, similarity matrix pattern recognition bypasses the limitations of step-by-step sequence tracking. It leverages the highly optimized spatial processing capabilities of convolutional networks to detect the geometric signatures of alignment. Whether dealing with the rhythm of a song, the cadence of a spoken word, or the fluctuations of a time series, treating time as space provides a robust, efficient, and elegant solution to sequence matching.
Key Foundational Work
CNN Based Query by Example Spoken Term Detection (2018) — contributed 3 concept elements to this direction.
ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning (2019) — contributed 2 concept elements to this direction.
Generating Music Medleys via Playing Music Puzzle Games (2018) — contributed 1 concept elements to this direction.