A Fuzzy-based Approach to Predict Human Interaction by Functional Near-Infrared Spectroscopy

Xiaowei Jiang, Liang Ou, Yanan Chen, Na Ao, Yu‐Cheng Chang, Thomas Do, Chin‐Teng Lin

TL;DR

This paper introduces a novel "Fuzzy Attention Layer" that enhances the interpretability and performance of Transformer models for analyzing brain activity (fNIRS) during social interactions. By integrating fuzzy logic, the model can identify specific neural patterns contributing to predictions, offering deeper insights into human social behavior and bridging the gap between AI and psychological theories.

What Problem Does This Paper Address?

Understanding human social interaction is a complex challenge, especially when trying to link neural activity to behavior. Traditional methods for analyzing brain signals like fNIRS often lack transparency, making it difficult to understand why a model makes a certain prediction. This "black box" problem is particularly acute with powerful models like Transformers, which can be hard to interpret. Researchers need ways to not only predict behavior from neural data but also to understand the underlying neural mechanisms, thus validating psychological theories with machine learning. This paper aims to bridge this gap by creating a more interpretable AI model for analyzing fNIRS data during social interactions.

Methods

The core innovation is the Fuzzy Attention Layer, which replaces the standard self-attention mechanism in a Transformer Encoder. Instead of relying solely on dot-product similarity, this layer uses fuzzy logic principles to calculate attention weights.

Here's a breakdown of the key components:

  • Fuzzy Logic Integration: The layer leverages fuzzy set theory and fuzzy neural networks. It models input sequences as fuzzy sets and outputs "firing strengths" that indicate how strongly a particular rule is activated. This allows the model to learn interpretable fuzzy rules.
  • Fuzzy Attention Mechanism: The Fuzzy Attention Layer calculates attention weights based on a fuzzy membership function, which considers both the "center" (mean) and "width" (standard deviation) of fuzzy sets. This is contrasted with standard self-attention, which primarily relies on cosine similarity. The paper argues this fuzzy approach is more adept at capturing complex patterns in sparse data like fNIRS.
  • Transformer Encoder Architecture: The Fuzzy Attention Layer is integrated into a standard Transformer Encoder. This allows the model to process sequential fNIRS data and capture inter-channel interactions.
  • Paired-Sample Framework: The model is designed to analyze data from pairs of participants, enabling the study of within- and between-subject variations in neural activity during interaction.
  • Explainable AI (xAI): The fuzzy rules learned by the attention layer provide a mechanism for interpretability, allowing researchers to understand which neural patterns are most influential for a given prediction.

Datasets

The study utilized two newly implemented datasets for training and evaluation:

  • Picture Recognition Dataset: Participants (friends and strangers) performed an emotional image recognition task under handholding and non-handholding conditions.
  • Picture Rating Dataset: Similar to the recognition dataset, but participants rated the emotional valence and arousal of images while in handholding or non-handholding conditions.

Both datasets involved pairs of participants seated back-to-back, with fNIRS data collected from the prefrontal cortex (PFC).

Key Insights and Ideas

  • Interpretability through Fuzzy Logic: The Fuzzy Attention Layer provides a direct link between learned fuzzy rules and neural activity, offering a more transparent view into the model's decision-making process compared to standard attention mechanisms.
  • Enhanced Pattern Recognition: The fuzzy approach is shown to be more effective than standard dot-product attention for capturing complex patterns in fNIRS data, which can be sparse and noisy.
  • Bridging AI and Psychology: The model's ability to learn interpretable neural patterns allows for the validation of psychological theories about social interaction and touch using machine learning.
  • Importance of Input Structure: The study highlights that the performance of the model can be influenced by how the fNIRS data is structured (e.g., channel-first vs. time-first), suggesting that optimal data representation is task-dependent.
  • Inter-Brain Synchrony (IBS) and Social Touch: The research found significantly higher IBS during handholding compared to non-handholding conditions, supporting the idea that physical touch enhances emotional connection and empathy.

Key Results

  • Performance Improvement: The Fuzzy Attention Layer-based Transformer model consistently outperformed baseline models (vanilla Transformer and LSTM) across various metrics (accuracy, recall, precision, F1, ROC AUC, PR AUC) on both datasets.
  • Top Performance in PR AUC: The Fuzzy Attention model achieved the highest PR AUC scores in Image Recognition (channel-first) and Image Rating (channel- and time-first) experiments, indicating its strength in classifying imbalanced datasets.
  • Interpretable Neural Patterns: The learned fuzzy rules were able to identify significant neural patterns associated with handholding, particularly in channels related to the prefrontal cortex (PFC). For instance, rules highlighted specific channels and time points that were crucial for distinguishing between handholding and non-handholding conditions.
  • IBS Enhancement: Correlation and cosine similarity metrics showed significantly higher IBS in the handholding condition, supporting the role of physical touch in fostering emotional connection.

Limitations and Future Work

Limitations:

  • Computational Speed: The reliance on the Transformer architecture leads to low computational speed. The authors suggest reducing model size as a potential remedy.
  • Rule Redundancy: The presence of similar fuzzy rules can lead to redundancies, impacting model efficiency. Optimizing for rule diversity is suggested.

Future Work:

  • Optimization of Fuzzy Centers and Rules: Further refinement of fuzzy center designs and automating the selection of the optimal number of fuzzy rules.
  • Improved Training Speed: Enhancing the speed of training to increase practical applicability.
  • Real-time Applications: Adapting the model for real-time social neuroscience studies and human-machine interaction.
  • Longitudinal Studies: Conducting studies to evaluate the long-term effects of social interactions on neural dynamics.
  • Broader Applications: Extending the Fuzzy Attention Layer to other neural data types like EEG and fMRI for enhanced decoding and interpretation.

Summary updated

Abstract

The paper introduces a Fuzzy-based Attention (Fuzzy Attention Layer) mechanism, a novel computational approach to enhance the interpretability and efficacy of neural models in psychological research. The proposed Fuzzy Attention Layer mechanism is integrated as a neural network layer within the Transformer Encoder model to facilitate the analysis of complex psychological phenomena through neural signals, such as those captured by functional Near-Infrared Spectroscopy (fNIRS). By leveraging fuzzy logic, the Fuzzy Attention Layer is capable of learning and identifying interpretable patterns of neural activity. This capability addresses a significant challenge when using Transformer: the lack of transparency in determining which specific brain activities most contribute to particular predictions. Our experimental results demonstrated on fNIRS data from subjects engaged in social interactions involving handholding reveal that the Fuzzy Attention Layer not only learns interpretable patterns of neural activity but also enhances model performance. Additionally, the learned patterns provide deeper insights into the neural correlates of interpersonal touch and emotional exchange. The application of our model shows promising potential in deciphering the subtle complexities of human social behaviors, thereby contributing significantly to the fields of social neuroscience and psychological AI.
BibTeX
@article{jiang2025fuzzy,
  author = {Xiaowei Jiang and Liang Ou and Yanan Chen and Na Ao and Yu‐Cheng Chang and Thomas Do and Chin‐Teng Lin},
  title = {{A Fuzzy-based Approach to Predict Human Interaction by Functional Near-Infrared Spectroscopy}},
  journal = {IEEE Transactions on Fuzzy Systems},
  year = {2025},
  doi = {10.1109/tfuzz.2025.3528376},
}
Figure