Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations

Sawyer Birnbaum, Volodymyr Kuleshov, S. Zayd Enam, Pang Wei Koh, Stefano Ermon

TL;DR

Temporal FiLM (TFiLM) is a novel architectural layer that combines the computational efficiency of convolutional neural networks (CNNs) with the long-range dependency tracking of recurrent neural networks (RNNs). By using a small RNN to modulate the activations of a CNN, TFiLM significantly boosts performance on sequence tasks like audio super-resolution and text classification while remaining highly computationally efficient.

The Problem: The Sequence Modeling Trade-Off

Deep learning researchers often face a difficult trade-off when processing sequential data like audio, text, or genomic sequences:

  1. Recurrent Neural Networks (RNNs) are naturally designed for sequences but are notoriously slow to train. They process data step-by-step and often struggle with "vanishing gradients," making it difficult for them to remember information from the distant past.
  2. Convolutional Neural Networks (CNNs) are fast, parallelizable, and easy to train. However, they have a "finite receptive field," meaning they only look at a small, localized window of data at a time. Capturing long-range dependencies—like a word at the start of a paragraph affecting the meaning at the end—usually requires very deep stacks of layers or complex dilated convolutions, which are memory-intensive.

The authors seek a "best of both worlds" approach: a model that trains as fast as a CNN but possesses the global contextual awareness of an RNN.

Methods: Introducing TFiLM

The core innovation of this paper is Temporal Feature-Wise Linear Modulation (TFiLM). This builds on the existing "FiLM" concept, which was originally used to let one type of data (like a text question) influence the processing of another (like an image). In TFiLM, the model uses its own history to modulate itself over time.

TFiLM Architecture

As shown in the architecture diagram above, the process works in several steps:

  1. Block Splitting: The activations from a 1D convolutional layer are split into temporal blocks (e.g., chunks of an audio waveform).
  2. Pooling: Each block is summarized into a single representation using max pooling.
  3. Recurrent Modulation: These summaries are fed into a small RNN (specifically, a BiLSTM). Because the RNN only processes one summary per block rather than every single timestep, it runs incredibly fast.
  4. Affine Transformation: The RNN outputs two parameters for each block: a scaling factor and a shift factor.
  5. Application: These parameters are used to element-wise "re-scale" the original CNN activations.

Essentially, the RNN acts as a "global conductor," telling the local convolutional filters how to adjust their behavior based on what has happened elsewhere in the sequence.

Key Insights

  • Self-Conditioning: Unlike previous FiLM applications that used an external input to modulate a network, TFiLM uses the sequence's own history to modulate its future.
  • Efficiency through Sparsity: By running the RNN on pooled blocks rather than individual samples, the computational overhead is drastically reduced. For a 5-second audio clip at 16kHz, a standard RNN would need 80,000 steps; TFiLM only needs about 50.
  • Receptive Field Expansion: TFiLM effectively gives a shallow CNN an "infinite" receptive field, allowing it to account for dependencies that are thousands of timesteps apart.

Key Results

The authors evaluated TFiLM across three distinct domains: text classification, audio super-resolution, and genomics.

Text Classification

Adding TFiLM to a lightweight "SmallCNN" yielded massive improvements on the Yelp sentiment analysis datasets.

Text Classification Results

As seen in the results table, the baseline SmallCNN achieved 78.1% accuracy on Yelp-2. Adding TFiLM skyrocketed this accuracy to 95.6%. This brought the lightweight model remarkably close to the performance of massive transformer models like BERT and XLNet, while utilizing fewer than 1.5 million parameters.

Audio Super-Resolution

For the task of reconstructing high-frequency signals from low-resolution inputs, TFiLM improved the Signal-to-Noise Ratio (SNR) by an average of 1.0 dB over standard CNN baselines. It significantly outperformed traditional spline interpolation and dense neural networks on both speech (VCTK dataset) and music (Piano dataset).

Genomics

In ChIP-seq reconstruction—where the goal is to reconstruct high-quality experimental measurements from noisy, low-subsample versions—TFiLM achieved a Pearson correlation of 0.90 for certain histone marks, compared to 0.72 for the previous state-of-the-art CNN baseline. This suggests TFiLM could make expensive genomic experiments much more affordable by requiring less raw data.

Limitations and Future Work

While powerful, the authors noted that a model trained on speech doesn't perform well on piano music (and vice versa), highlighting a need for more diverse training data to create truly "universal" super-resolution models. Additionally, the paper leaves room to explore how the choice of block size affects different types of data (e.g., phonemes in speech vs. words in text). Future directions could see TFiLM applied to text-to-speech generation and other complex generative tasks where long-term structure is critical.

Summary updated

Abstract

Learning representations that accurately capture long-range dependencies in sequential inputs --- including text, audio, and genomic data --- is a key problem in deep learning. Feed-forward convolutional models capture only feature interactions within finite receptive fields while recurrent architectures can be slow and difficult to train due to vanishing gradients. Here, we propose Temporal Feature-Wise Linear Modulation (TFiLM) --- a novel architectural component inspired by adaptive batch normalization and its extensions --- that uses a recurrent neural network to alter the activations of a convolutional model. This approach expands the receptive field of convolutional sequence models with minimal computational overhead. Empirically, we find that TFiLM significantly improves the learning speed and accuracy of feed-forward neural networks on a range of generative and discriminative learning tasks, including text classification and audio super-resolution.
BibTeX
@inproceedings{birnbaum2019temporal,
  author = {Sawyer Birnbaum and Volodymyr Kuleshov and S. Zayd Enam and Pang Wei Koh and Stefano Ermon},
  title = {{Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=LrYjTxUvf_y},
}
Figure