Gaussian Adaptive Attention is All You Need: Robust Contextual Representations Across Multiple Modalities

Georgios Ioannides, Aman Chadha, Aaron ElkinsarXiv 2024 · OpenReview

Introduction

The Transformer architecture's reliance on scaled dot-product attention has defined modern deep learning. However, this mechanism often suffers from low entropy bias and struggles with non-stationary data, particularly in modalities like speech where signal statistics fluctuate rapidly. Furthermore, fine-tuning large pre-trained models (PTMs) remains computationally prohibitive. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have gained popularity, they still require significant parameter overhead to maintain performance.

This paper introduces the Multi-Head Density Adaptive Attention Mechanism (DAAM), a probabilistic framework that replaces standard attention with learnable Gaussian distributions. DAAM achieves state-of-the-art results across speech, text, and vision tasks while utilizing up to 24x fewer parameters than LoRA.

The DAAM Framework

The core of DAAM is a shift from dot-product similarity to a density-based approach. By modeling feature significance through learnable means and variances, the model can dynamically adjust its focus and the "spread" of its attention.

Algorithm 1: The Density Adaptive Attention Mechanism (DAAM) provides a step-by-step procedure for normalizing input tensors and applying Gaussian transformations to recalibrate feature significance.

As detailed in Algorithm 1, the mechanism normalizes the input $x$ using a learnable mean offset and variance scaling. The attention-modified tensor is then computed using a Gaussian function: $$f(x) = \exp\left(-\frac{y_{norm}^2}{2c^2}\right)$$ where $c$ is a learnable parameter controlling the distribution's spread. By employing multiple heads, DAAM acts as a Mixture of Gaussians, allowing it to approximate complex, non-Gaussian data distributions.

Experimental Framework

To evaluate the robustness of DAAM, the researchers utilized a frozen encoder architecture across three distinct modalities:

  1. Speech: WavLM-Large evaluated on IEMOCAP (emotion recognition).
  2. Text: Llama2-13B evaluated on AG News (classification).
  3. Vision: BEiT-Large evaluated on CIFAR100 (image classification).
The overall architecture consists of a frozen pre-trained encoder, a DAAM-based decoder network featuring an attention module and convolutional layers, and a postprocessing stage.

The decoder network, shown in the figure above, integrates the DAAM module followed by two blocks of Conv2D and ReLU layers, concluding with a linear layer. This setup ensures that the heavy lifting of feature extraction is handled by the frozen PTM, while DAAM performs efficient task-specific modulation.

Results and Discussion

DAAM demonstrates exceptional performance and parameter efficiency compared to standard Multi-Head Attention (MHA) and LoRA.

Modality-Specific Insights

The study reveals that different modalities benefit from different Gaussian adjustments:

  • Speech: Capturing emotional nuances requires high variability in both the mean (focus) and variance (width) of the attention distribution.
  • Text: Semantic shifts are primarily tracked through adjustments in the mean.
  • Vision: Visual features are relatively stable, allowing for simpler, more consistent attention distributions.

Performance Benchmarks

  • Speech (IEMOCAP): DAAMv1 achieved 67.4% accuracy, outperforming MHA (62.3%) and LoRA (53.3%).
  • Vision (CIFAR100): DAAM achieved ~80% accuracy, whereas LoRA (rank 8) reached only ~40%, highlighting a significant breakthrough in visual feature adaptation.
  • Efficiency: DAAM requires only 0.016M to 0.082M parameters—approximately 24x fewer than a rank-4 LoRA implementation.

Conclusion

The Multi-Head Density Adaptive Attention Mechanism offers a robust, explainable, and highly efficient alternative for fine-tuning large-scale models. By leveraging the probabilistic nature of Gaussian distributions, DAAM overcomes the entropy limitations of standard attention. Future work will explore integrating DAAM directly into the pre-training phase and utilizing the Importance Factor (IF) for advanced model pruning.

Summary updated

Abstract

We propose the Multi-Head Density Adaptive Attention Mechanism (DAAM), a novel probabilistic attention framework that can be used for Parameter-Efficient Fine-tuning (PEFT), and the Density Adaptive Transformer (DAT), designed to enhance information aggregation across multiple modalities, including Speech, Text, and Vision. DAAM integrates learnable mean and variance into its attention mechanism, implemented in a multi-head framework, enabling it to collectively model any probability distribution for dynamic recalibration of feature significance. This method demonstrates significant improvements, especially with highly non-stationary data, surpassing the state-of-the-art attention techniques in model performance, up to approximately +20% (abs.) in accuracy. Empirically, DAAM exhibits superior adaptability and efficacy across a diverse range of tasks, including emotion recognition in speech, image classification, and text classification, thereby establishing its robustness and versatility in handling data across multiple modalities. Furthermore, we introduce the Importance Factor, a new learning-based metric that enhances the explainability of models trained with DAAM-based methods.
BibTeX
@misc{ioannides2024gaussian,
  author = {Georgios Ioannides and Aman Chadha and Aaron Elkins},
  title = {{Gaussian Adaptive Attention is All You Need: Robust Contextual Representations Across Multiple Modalities}},
  year = {2024},
  eprint = {2401.11143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.11143},
}
Figure