Pruning Feature Channels by What They Do to the Warping Path

Many systems compare two sequences, such as two videos of someone pouring water or an audio track and lip movements, by first passing each frame through a deep network and then running dynamic time warping (DTW). DTW is a dynamic program that finds the best monotonic matching between the time steps of the two sequences. The expensive part is usually not the dynamic program. It is building the $T \times T$ table of distances between every pair of frame embeddings, which costs $O(T^2 D)$ for embeddings with $D$ channels, and then backpropagating through that table. Compression methods for the encoder judge channels by how well they reconstruct the next layer's activations or the classifier's input. The quantity that matters here is different: whether removing a channel changes which frames get matched.

We propose a channel-pruning criterion for encoders that feed an alignment layer. It scores each channel by how much the soft alignment moves when that channel is removed. The scores are then propagated backward to prune filters in earlier layers. The key fact is that squared-Euclidean alignment costs add up over channels. A channel matters for alignment only if it separates frames that should be matched from frames that should not. A channel that adds roughly the same cost everywhere does not matter, however large its variance. One common source of such channels is static appearance, such as the background or the identity of the person. Variance-based compression (PCA, weight norms) keeps these channels. An alignment-aware criterion can discard them. This shrinks the $D$ factor of the quadratic cost and the encoder's own compute in one step.

Core Claim

Write $x_t, y_s \in \mathbb{R}^D$ for the embeddings of two sequences. The pairwise cost matrix splits exactly into one matrix per channel, $C = \sum_{d=1}^{D} C^{(d)}$ with $C^{(d)}_{ts} = (x_{t,d} - y_{s,d})^2$. Soft-DTW is a smoothed version of DTW that replaces the hard minimum over warping paths with a soft minimum, so it can be differentiated. Its gradient with respect to $C$ is the expected alignment matrix $E(C)$: entry $(t,s)$ is the probability that frame $t$ is matched to frame $s$ (Soft-DTW: a Differentiable Loss Function for Time-Series). We define the alignment importance of channel $d$ as

$$ s_d = \mathbb{E}_{(x,y)} \left\| E(C) - E\!\left(C - C^{(d)}\right) \right\|_1 , $$

averaged over training pairs. A channel whose $C^{(d)}$ is nearly constant across the cost matrix has $s_d \approx 0$, because adding a constant to every cell shifts every path's cost by the same amount relative to the others. The importance therefore measures contrast along versus off the path, not magnitude.

Computing $s_d$ exactly takes one extra soft-DTW pass per channel on a calibration batch. This is affordable offline for $D$ in the hundreds. A first-order alternative uses one Hessian-vector product of soft-DTW in the direction $C^{(d)}$. We will check that the cheap version ranks channels the same way as the exact one.

Final-layer scores are pushed back to earlier convolutional or Transformer filters using the backward score propagation of NISP. NISP propagates importance from the layer just before the classifier through absolute weight magnitudes. We keep its propagation step but replace its starting signal: instead of feature-selection scores on a classification layer, we start from $s_d$. After pruning, the surviving filters are refit by least squares as in ThiNet. Unlike ThiNet, the fit is weighted by $E$, so that reconstruction error counts only on frame pairs the alignment actually uses. A short fine-tuning pass with the original alignment loss finishes the procedure.

Relation to Prior Work

The pruning side supplies methods and constraints. Several methods already judge a channel by how it is consumed downstream. ThiNet minimizes next-layer reconstruction error and reports 3.31× fewer FLOPs on VGG-16 for a 0.52% top-5 drop. Where to Prune scores the consuming kernels in the next layer. CPMC scores a filter together with the kernels that consume its output. NISP propagates importance from the final feature layer. MINT scores channels by mutual information between adjacent layers. All of them define "downstream" as Euclidean reconstruction or classification. They therefore treat two embeddings that differ in Euclidean distance as different, even when the two produce the same warping. Our change is the target of the dependency, not the propagation machinery.

MINT also constrains the design. It found that concentrating information in fewer channels hurts adversarial robustness. We therefore test robustness to temporal jitter explicitly.

Channel Matters is the nearest channel-level work on sequences. It prunes raw input sensors using gradient-based influence for forecasting. It does not prune hidden channels, and its setting involves no alignment.

The alignment side motivates the work. LRProp trains video Transformers with DTW teacher paths plus soft-DTW, and names DTW's quadratic cost as what keeps it on short clips. Deep Canonical Time Warping aligns the latent spaces of dense, uncompressed encoders with standard DTW. Neural Edit Operations reports that its differentiable alignment layer runs more than 10× slower than convolution.

Three prior approaches reduce alignment cost along a different axis. Banding methods, such as Fast Global Alignment Kernels, restrict the search to a diagonal window. Linear-time 1D transport, as in OTTC, replaces the dynamic program. Anchor embeddings, as in Random Warping Series, compare each sequence to random templates instead of to each other. All three attack the $T^2$ factor or avoid alignment altogether.

One limitation should be stated plainly. Channel pruning does not shrink the $T \times T$ dynamic-programming table. It removes the $D$ factor in building the cost matrix and in backpropagating through it, and it shrinks the encoder. It can therefore be combined with banding.

Research Program

The work has four stages.

The first stage is analytic. We characterize which channels have $s_d \approx 0$: those whose per-channel cost is close to separable, of the form $a_t + b_s$, along the band of plausible paths. We also show that additive offsets of this kind leave the DTW path unchanged. This gives a testable prediction: channels with high variance between videos but low variance within a video are the ones PCA keeps and our criterion drops.

The second stage builds the scorer, both the exact and the Hessian-vector-product versions. It then adds NISP-style propagation and $E$-weighted least-squares refitting. Residual connections are handled by sharing scores across tied channels, avoiding the restriction ThiNet needed of pruning only inside residual blocks.

The third stage runs the pruning on LRProp-style video encoders and on DCTW-style two-modality encoders, sweeping the fraction of channels kept from 50% down to 5%.

The fourth stage measures the cost of the full pipeline, encoder plus cost matrix plus soft-DTW forward and backward. We report wall-clock time and peak GPU memory as sequence length grows. We then test whether the saved memory allows training on longer clips, which is the limitation LRProp names.

Baselines

The naive reference is the unpruned encoder with full soft-DTW. It sets the accuracy ceiling and the cost floor. The next simplest reference prunes each layer to the same width by $L_1$ weight norm, or at random, and then fine-tunes. Neither uses downstream information. ThiNet reports that both fall behind reconstruction-based pruning at high compression on classification.

The strongest existing pruning competitors are ThiNet and NISP, run as published. ThiNet minimizes next-layer reconstruction. NISP propagates scores from the final embedding layer with its original feature-selection scores. These represent the best inter-layer dependency methods available, and they differ from ours only in what "downstream" means.

The strongest non-pruning competitor for the $D$ factor is a learned or PCA projection of the final embedding to the same reduced width $D'$, followed by fine-tuning. It cuts cost-matrix compute by exactly the same factor.

The key ablation keeps everything in our pipeline but replaces $s_d$ with NISP's original score. A second ablation drops the $E$-weighting from the refit. Together these isolate whether alignment-awareness, rather than propagation or refitting, explains any gain.

Every method gets the same calibration data, the same fine-tuning epochs, and the same tuning budget for its hyperparameters. All are compared at matched FLOPs and matched $D'$.

The reported numbers to beat are as follows. On Pouring, LRProp reports 93.88% phase classification accuracy with 25% labels. On XRMB, DCTW reports an alignment error of 7.19. For pruning methods applied to alignment tasks, no reported result was found on Lacuna. For Kendall's tau on PennAction, no reported number was found on Lacuna either; LRProp states only "significant improvements". We count the method as successful if it stays within one point of the unpruned Pouring accuracy, and within a comparable margin of the unpruned XRMB error, at 25% of channels. It must also beat ThiNet, NISP and PCA by a clear margin at 10% of channels.

Why not just…?

Why not just project the final embedding with PCA or a learned linear layer to a smaller dimension? This is the most serious threat, because it buys the same $O(T^2 D')$ saving with a trivial change. Our argument is that PCA keeps directions of high variance, and in cross-sequence alignment much of that variance is appearance that is constant within a video and adds a nearly flat offset to the cost matrix. We test this by measuring, for each top PCA direction, how separable its cost is and what $s_d$ it receives.

If PCA matches us in alignment quality at equal $D'$, we concede the $D$-factor claim. In that case the contribution narrows to encoder compression, which a final-layer projection cannot provide, and we would say so.

Why not just train a narrower encoder from scratch, or distill into one? A small model trained with the soft-DTW loss might find a good embedding on its own. We include a narrow model distilled to match the teacher's alignment matrices, trained with the same total compute as pruning plus fine-tuning. If it matches us, the scoring adds nothing beyond a training recipe.

Why not just use a Sakoe-Chiba band or 1D optimal transport and leave the network alone? Banding cuts the $T^2$ factor, not $D$, so the two approaches combine rather than compete. We report banded variants of every method. Fast Global Alignment Kernels warns that bands clip paths when sequence lengths differ by more than about a factor of two, so the banded comparison also tells us where banding alone fails. OTTC pays a measurable accuracy cost: LibriSpeech word error rate rises from 2.20% with CTC to 2.52%. We include it as a cost-matched reference, not a replacement.

Proposed Experiments

The main experiments use Pouring and PennAction, with an LRProp-style Transformer encoder. Metrics are phase classification accuracy, Kendall's tau of the alignment, and average precision.

The two-modality experiments use XRMB and CUAVE with DCTW-style encoders, scored by alignment error. On CUAVE, no alignment number was found on Lacuna, so we run the unpruned model ourselves as the reference.

Cost is measured as encoder FLOPs, cost-matrix FLOPs, peak backward memory, and wall-clock time. Lengths are swept from 128 up to the longest $T$ that fits unpruned on one GPU.

Three decisive checks would settle the claims. The first is the PCA comparison at matched $D'$, which tests whether alignment-aware selection beats variance-based compression. The second is a correlation test: across all channels, $s_d$ should predict the measured change in Kendall's tau after removing that channel better than $L_1$ norm, ThiNet reconstruction error, or PCA loading do. The third is a robustness test. Following MINT's warning, we perturb inputs with random temporal stretching and dropped frames, and check whether pruned models lose more alignment accuracy than unpruned ones.

A final experiment uses the memory freed by pruning to train on clips 2–4× longer. It asks whether this measurably improves Kendall's tau over the short-clip regime that LRProp is limited to.