Conditioning Sequence-to-sequence Networks with Learned Activations

Alberto Gil Couto Pimentel Ramos, Abhinav Mehrotra, Nicholas Donald Lane, Sourav BhattacharyaICLR 2022 · OpenReview

Overview

Researchers have developed a novel approach to "personalize" neural networks by dynamically learning activation functions based on a conditioning vector, such as a user's unique voice profile. This method achieves state-of-the-art performance in speech enhancement and recognition tasks while using up to 95% fewer additional parameters than standard conditioning methods. By shifting the conditioning mechanism from the linear to the non-linear components of the network, this technique emerges as a highly efficient candidate for on-device AI.

The Challenge of Parameter-Hungry Conditioning

In many modern AI tasks, models must adapt their behavior based on specific contexts—a process known as conditioning. For instance, in Personalized Sound Enhancement (PSE), a model must filter out background noise while preserving the specific voice of a "target" speaker.

The standard approaches to conditioning are typically concatenation (appending a user-profile vector directly to the input) or modulation (scaling and shifting intermediate features, similar to FiLM). While effective, these methods are notoriously "parameter-hungry." They require adding a massive number of new weights that scale linearly with the size of the network's internal layers. This parameter bloat makes deploying personalized models on resource-constrained devices, like smartphones or smartwatches, highly impractical.

Proposed Method: Learned Activations (LA)

To solve this efficiency bottleneck, the authors introduce Learned Activation (LA) functions. Instead of relying on a fixed, static activation function like ReLU or Swish throughout the network, the model dynamically constructs a custom activation function for every user.

Comparison of conditioning methods: (a) Concatenation, (b) Modulation, and (c) the proposed Learned Activation.

The process works as follows:

  1. Weighted Combination: The system defines a dictionary of "basic" activation functions (e.g., ReLU, Tanh, Sigmoid, Swish).
  2. Dynamic Selection: A small, lightweight sub-network processes the conditioning vector (the user's voice profile) and outputs a set of weights.
  3. The Activation: The final activation function applied in a given layer is the weighted sum of those basic functions.

Mathematically, if $z$ is the user profile, the activation for a hidden value $h$ becomes: $$LA(h | z) = \sum s_i(z) \cdot A_i(h)$$

This architectural shift is clever because the number of extra parameters depends only on the size of the user profile and the number of basic functions—not the number of channels in the neural network.

Datasets and Evaluation

The researchers evaluated their method on primary speech tasks using established benchmarks:

  • LibriSpeech (English): Used for both sound enhancement and personalized speech recognition.
  • VoxForge (Spanish): Used to test the generalizability of the approach across different languages.
  • DEMAND: A dataset of real-world ambient noise (cafes, offices, etc.) used to stress-test the sound enhancement models.

Key Insights and Ideas

  • Efficiency Through Non-Linearity: The core insight is that you do not need to modify every single neuron's weight to achieve personalization. By simply changing the "shape" of the non-linearity (the activation function), the model can effectively gate or emphasize information relevant to a specific user. Conditioning the non-linear component is just as powerful as linear transformations, but vastly cheaper.
  • Visualizing Personalization: Because the activations are learned combinations, they can be plotted. The authors demonstrate that the model learns distinct "shapes" for different users and across different layers, providing a rare, interpretable window into how the model adapts to individual profiles.
Visualizations of learned activation functions for different users across various network layers, showing distinct personalized shapes.
  • Regularization Matters: The authors found that using entropy regularization (encouraging the model to confidently pick a "best" activation) or t-SNE regularization (ensuring similar voices get similar activations) further improved overall performance.

Key Results

  • Massive Size Reduction: Compared to standard concatenation, the LA approach reduced the number of additional parameters required for conditioning by 10% to 95%, depending on the specific network architecture.
  • Competitive Quality: In Personalized Sound Enhancement, LA-based models achieved Signal-to-Distortion Ratio (SDRi) improvements comparable to much larger, traditional models.
  • ASR Gains: For personalized speech recognition (PASR), the method achieved an average 19% relative improvement in Word Error Rate (WER). Notably, some individual users saw up to a 40% improvement.
  • Speed and Latency: LA is computationally faster than concatenation and offers a superior quality-to-latency trade-off compared to modulation techniques on both CPUs and GPUs.

Limitations and Future Work

While highly effective for audio tasks, the paper focuses exclusively on speech enhancement and recognition. The authors hypothesize that this logic applies to computer vision or generative modeling, but those domains remain untested. Furthermore, the experiments assumed "clean" enrollment data (a clear 2-second clip of the user). In real-world scenarios, enrollment might happen in noisy environments, requiring further study on the robustness of LA to "dirty" conditioning vectors. Finally, computational constraints limited the hyperparameter search, suggesting that even better results might be achievable with more extensive tuning of the basic activations and regularization weights.

Summary updated

Abstract

Conditional neural networks play an important role in a number of sequence-to-sequence modeling tasks, including personalized sound enhancement (PSE), speaker dependent automatic speech recognition...
BibTeX
@inproceedings{ramos2022conditioning,
  author = {Alberto Gil Couto Pimentel Ramos and Abhinav Mehrotra and Nicholas Donald Lane and Sourav Bhattacharya},
  title = {{Conditioning Sequence-to-sequence Networks with Learned Activations}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=yzkvPB932P2},
}
Figure