Learning Fixed Length Representations for Variable Length Sequences

132 concepts126 papers1 proposal

Machine learning algorithms—from classical Support Vector Machines (SVMs) and $k$-means clustering to standard Convolutional Neural Networks (CNNs)—fundamentally expect data to live in a static, fixed-dimensional vector space. However, real-world sequential data is messy and inherently variable in duration. A spoken word can take half a second or two seconds; an online student's learning trajectory can span five steps or fifty; a genomic sequence or a network traffic log has no fixed boundary.

As noted by Smyth et al. and Linardi, forcing these dynamic sequences into rigid, uniform boxes usually requires destructive preprocessing. Naive heuristics like zero-padding, truncation, or downsampling distort the underlying temporal dynamics, introduce artificial noise, or discard critical moment-to-moment nuances (Shen). To apply powerful, general-purpose machine learning tools to raw sequence data, we need a principled way to bridge this gap: transforming variable-length sequences into fixed-length representations without losing their structural information.

The Generative-Discriminative Bridge

How do we turn an arbitrary sequence of tokens or coordinates into a single, fixed-size vector? One of the earliest and most elegant insights was to use generative models as a mathematical bridge.

Consider a generative model, such as a Hidden Markov Model (HMM) or a Gaussian Mixture Model (GMM), trained on a collection of sequences. This model learns the underlying statistical regularities of the data. In their seminal work, Jaakkola et al. introduced the Fisher Kernel. Instead of feeding a raw sequence directly to a classifier, they fed it to a generative model and computed the gradient of the log-likelihood with respect to the model's parameters.

Why does this work? The gradient vector (the Fisher score) has a fixed size determined entirely by the number of parameters in the generative model, regardless of how long the input sequence is. Intuitively, this vector captures how the parameters of the model would need to adjust to better fit this specific sequence. It transforms a variable-length history into a fixed-length signature of architectural adjustments, which can then be fed into powerful discriminative classifiers like SVMs. This strategy of using probabilistic models as static feature extractors has been widely adapted, from speaker verification using GMMs (Lee et al.) to trajectory clustering via HMM parameter spaces (Porikli et al.).

Sequence-Aware Kernels and Random Projections

Another approach avoids explicit vector transformation by modifying the similarity metric itself. If a kernel function can natively compare two sequences of different lengths, distance-based algorithms can operate directly on raw sequences.

For instance, Shimodaira et al. introduced the Dynamic Time-Alignment Kernel (DTAK), which embeds non-linear time-warping directly into a kernel function, allowing SVMs to classify variable-length speech. Similarly, Cortes developed Rational Kernels using weighted transducers to compare variable-length sequences in linguistics and biology. More recently, Alemohammad et al. proposed the Recurrent Neural Tangent Kernel (RNTK), providing a mathematically rigorous framework that naturally accommodates variable-length temporal data without zero-padding.

However, computing pairwise sequence alignments (like Dynamic Time Warping) is computationally expensive, often scaling quadratically with sequence length. To scale these methods, researchers use Random Features. The core intuition behind the Random Warping Series (RWS) framework is to project a variable-length time series against a set of randomly generated sequences. By measuring how a sequence aligns to these stable random anchors, RWS extracts a fixed-length vector of alignment scores in linear time, bypassing the need for expensive pairwise comparisons across the entire dataset.

Deep Representation Learning and Adaptive Pooling

Modern deep learning solves the fixed-length constraint by training neural networks to compress arbitrary durations into static embedding spaces.

A classic architecture for this is the sequence-to-sequence autoencoder. As demonstrated by Chung et al. with Audio Word2Vec, an encoder RNN processes an audio segment of any duration step-by-step, updating its internal hidden state. The final hidden state acts as a fixed-length bottleneck vector that encapsulates the entire sequence's phonetic structure, which a decoder then attempts to reconstruct.

Alternatively, convolutional architectures can be adapted for variable lengths using specialized pooling layers. Standard CNNs require fixed inputs because their fully connected layers expect a predetermined number of features. Sawada et al. overcame this by introducing Adaptive Multi-scale Pooling (AMSpool). Instead of using a fixed pooling window, AMSpool dynamically adjusts its window size based on the input length, collapsing the temporal dimension into a consistent, fixed-size representation regardless of whether the input sequence is short or long.

By combining these architectural innovations with similarity-preserving losses, frameworks like SPIRAL (Lei et al.) can map variable-length time series into standard feature vectors while explicitly preserving complex pairwise temporal similarities, unlocking the use of highly optimized, general-purpose machine learning tools on raw sequential data.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_length_input_constraint_d0712822,
  title = {fixed-length input constraint},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-length-input-constraint/rdr_d0712822fa544ba3846b5710a622aec3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_fixed_length_input_constraint_d0712822_v1,
  title = {fixed-length input constraint},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-length-input-constraint/rdr_d0712822fa544ba3846b5710a622aec3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake