A Fractal Approach to Characterize Emotions in Audio and Visual Domain: A Study on Cross-Modal Interaction

Sayan Nag, Uddalok Sarkar, Shankha Sanyal, Archi Banerjee, Souparno Roy, Samir Karmakar, Ranjan Sengupta, Dipak GhosharXiv 2021 · arXiv

Introduction

The intersection of human emotion and sensory perception has long been a subject of psychological inquiry, typically relying on subjective self-reporting. However, recent advancements in non-linear dynamics suggest that emotional stimuli—whether auditory or visual—possess inherent structural complexities that can be quantified. This study explores a novel mathematical framework using fractal analysis to objectively classify emotions in music and art, specifically focusing on the cross-modal interaction between these two domains.

Methodology

The research utilizes a suite of non-linear data analysis tools to extract mathematical signatures from emotional stimuli. The primary metric employed is Detrended Fluctuation Analysis (DFA). For 1D audio signals, DFA calculates the scaling exponent (Hurst exponent, $\alpha$), which quantifies long-range temporal correlations. To analyze the visual domain, the authors propose a 2D-DFA algorithm that processes paintings by decomposing them into Red, Green, and Blue (RGB) components to calculate spatial scaling exponents.

Additionally, Detrended Cross-Correlation Analysis (DCCA) was used to measure the structural similarity between different signals within and across emotional categories. The study utilized 12 validated stimuli: six audio clips from the International Affective Digitized Sounds (IADS) database and six famous paintings (e.g., works by Van Gogh and Picasso), evenly split between "Happy" (positive valence) and "Sad" (negative valence) categories.

Results and Discussion

Fractal Signatures of Emotion

A central finding of the study is that "sad" stimuli exhibit significantly higher long-range correlations than "happy" ones. In both the audio and visual domains, a DFA scaling exponent greater than 1.5 consistently identified stimuli belonging to the sad category.

DFA scaling exponents for audio clips (left) and visual images (right) across RGB channels.

As shown in the figure above, audio clips 4-6 (Sad) show higher exponents (~1.8-1.9) compared to clips 1-3 (Happy, ~1.5-1.6). In the visual domain, the scaling exponents for sad paintings were particularly pronounced in the green and blue color channels, providing a mathematical basis for the artistic intuition that "cool" colors evoke melancholy.

Cross-Modal Consistency and Correlation

The study revealed a remarkable consistency in how emotions are mathematically structured across different senses. The Pearson correlation coefficient between the DFA values of happy audio and happy images reached 0.987, while sad stimuli showed a correlation of 0.96. This suggests that the "mathematics of feeling" may be universal, transcending the specific sensory modality.

Detrended Cross-Correlation Analysis (DCCA) coefficients for audio clips, showing stronger internal correlation for sad stimuli.

DCCA results further supported these findings. Cross-correlation was strongest among stimuli of the same valence, particularly within the sad category. Conversely, inter-valence comparisons (comparing a happy stimulus to a sad one) often resulted in coefficients near zero, indicating a lack of shared structural complexity between opposing emotional states.

Conclusion

By moving beyond subjective surveys toward computational classification, this research demonstrates that fractal geometry can serve as a robust proxy for emotional valence. The discovery of a consistent mathematical signature across audio and visual domains opens new avenues for automated emotion recognition systems, adaptive media, and therapeutic interventions. Future work will aim to expand the dataset and explore whether these fractal tools can differentiate between more nuanced emotional states beyond the binary of happy and sad.

Summary updated

Abstract

It is already known that both auditory and visual stimulus is able to convey emotions in human mind to different extent. The strength or intensity of the emotional arousal vary depending on the type of stimulus chosen. In this study, we try to investigate the emotional arousal in a cross-modal scenario involving both auditory and visual stimulus while studying their source characteristics. A robust fractal analytic technique called Detrended Fluctuation Analysis (DFA) and its 2D analogue has been used to characterize three (3) standardized audio and video signals quantifying their scaling exponent corresponding to positive and negative valence. It was found that there is significant difference in scaling exponents corresponding to the two different modalities. Detrended Cross Correlation Analysis (DCCA) has also been applied to decipher degree of cross-correlation among the individual audio and visual stimulus. This is the first of its kind study which proposes a novel algorithm with which emotional arousal can be classified in cross-modal scenario using only the source audio and visual signals while also attempting a correlation between them.
BibTeX
@misc{nag2021fractal,
  author = {Sayan Nag and Uddalok Sarkar and Shankha Sanyal and Archi Banerjee and Souparno Roy and Samir Karmakar and Ranjan Sengupta and Dipak Ghosh},
  title = {{A Fractal Approach to Characterize Emotions in Audio and Visual Domain: A Study on Cross-Modal Interaction}},
  year = {2021},
  eprint = {2102.06038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.06038},
}
Figure