Abstract: We present softdtw-cuda-torch, an open-source PyTorch library for computing Soft Dynamic Time Warping (SoftDTW) on GPUs. Our implementation addresses three key limitations of existing GPU implementations of SoftDTW: a hard sequence-length cap of 1024, numerical instability in the backward pass for small smoothing parameters, and excessive GPU memory consumption from materializing pairwise distance tensors. We introduce (1) tiled anti-diagonal kernel execution that removes the sequence-length constraint, (2) a log-space back-ward pass that prevents floating-point overflow, and (3) a fused distance-computation mode that eliminates the O(BN M ) intermediate distance tensor, achieving up to 98% memory reduction compared to prior work. The library supports arbitrary sequence lengths, full PyTorch autograd integration, and Soft-DTW Barycenter computation. Code is available at https://github.com/BGU-CS-VIL/sdtw-cuda-torch.
Introduction
Soft Dynamic Time Warping (SoftDTW) is a differentiable variant of the classic Dynamic Time Warping (DTW) algorithm, widely used in time-series analysis to measure similarity between sequences that may be shifted or stretched. While SoftDTW is a powerful loss function for deep learning, its adoption has been hindered by significant computational bottlenecks on GPUs, particularly regarding sequence length limits and memory overhead.
Overcoming GPU Limitations
Existing GPU implementations of SoftDTW often suffer from three critical issues: a 1024-step sequence length cap due to CUDA thread management, numerical instability (NaN gradients) during the backward pass when using small smoothing parameters ($\gamma$), and massive memory consumption caused by materializing a large 3D tensor of pairwise distances ($Batch \times Length \times Length$).
To address these, softdtw-cuda-torch introduces three technical innovations:
- Tiled Anti-Diagonal Execution: By launching separate GPU kernels for each anti-diagonal of the dynamic programming table, the library removes the 1024-step sequence length constraint, allowing for much longer time-series processing.
- Log-Space Backward Pass: Gradient calculations are reformulated in log-space using the
logsumexptrick, ensuring numerical stability and preventing the floating-point overflows that typically crash training. - Fused Distance Computation: In its "fused" mode, the library recomputes pairwise distances on-the-fly within the CUDA kernel. This eliminates the need to store the intermediate distance tensor, drastically reducing the memory footprint.
Performance and Scaling
Benchmarks against prior state-of-the-art implementations (e.g., Maghoumi's pytorch-softdtw-cuda) demonstrate significant improvements in both scalability and efficiency. While the fused mode introduces a runtime trade-off due to recomputation, it enables the processing of large-scale models and long sequences that were previously impossible to fit on standard GPU hardware.
As shown in the benchmarks, the fused mode achieves a peak GPU memory reduction of up to 98%. For example, with a batch size of 32 and sequence length of 512, memory usage drops from approximately 8 GB to just 161 MB.
Application: SoftDTW Barycenters
The library's efficiency and stability make it ideal for computing SoftDTW Barycenters—the "average" sequence of a given set. Unlike standard Euclidean means, which blur features when sequences are misaligned, SoftDTW Barycenters preserve the structural characteristics of the input data.
Conclusion
softdtw-cuda-torch provides a robust, memory-efficient, and scalable solution for integrating SoftDTW into PyTorch workflows. By breaking the 1024-step barrier and solving numerical stability issues, it opens new possibilities for long-form time-series modeling and complex sequence alignment tasks in deep learning.
Summary updated