Active Inference Through Energy Minimization in Multimodal Affective Human–Robot Interaction

Takato Horii, Yukie NagaiFrontiers in Robotics and AI 2021 · DOI

TL;DR

Robots interacting with humans often have limited "attention" and cannot process every sensory cue (face, voice, gestures) simultaneously. This paper proposes a method based on Active Inference that allows a robot to strategically select the most informative modality by minimizing the "energy" of a multimodal neural network. The result is a system that recognizes human emotions faster and more accurately than traditional methods, especially when sensory signals are highly correlated.

What Problem Does This Paper Address?

In Human-Robot Interaction (HRI), estimating a partner's emotional state is critical. Humans express emotions through a cocktail of modalities: a furrowed brow, a sharp tone of voice, or a sudden gesture. While a high-end server might process all these streams at once, a mobile robot has limited computational resources and sensor bandwidth. It must decide: Where should I look or listen next to be sure of what this person is feeling?

Existing "active perception" strategies often assume that different modalities (like touch vs. sight) are independent. However, human emotional expressions are deeply interrelated—if someone is shouting, their mouth is likely open. The authors argue that failing to account for these correlations leads to inefficient "attention" shifts. They aim to bridge the gap between robotic perception and the biological principle of Active Inference, which suggests the brain acts specifically to reduce uncertainty about the world.

Methods

The researchers implemented a Multimodal Deep Belief Network (MDBN). Unlike standard feed-forward networks, an MDBN is an energy-based model. In this framework, every possible state of the network has an "energy" value; states that are more likely to occur in the real world have lower energy.

The core of their approach is a four-step active perception loop:

  1. Initial Estimate: The robot receives one initial signal (e.g., voice) and estimates the emotion.
  2. Prediction: The MDBN predicts what the unobserved modalities (e.g., facial expression, hand gestures) should look like based on the current estimate.
  3. Energy Minimization: The robot calculates which unobserved modality, if measured, would most effectively minimize the overall "energy" (uncertainty) of the system.
  4. Update: The robot samples that modality and updates its emotional estimate.

Mathematically, the authors demonstrate that this energy minimization is equivalent to maximizing Information Gain while also accounting for the likelihood of the predicted signals—a key component of the "Free Energy Principle" in neuroscience.

Datasets

The study utilized the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset.

  • Content: 12 hours of audiovisual data from 10 actors performing emotional interactions.
  • Features: The authors extracted 7 distinct modalities: movements of the right hand, left hand, mouth, cheek, and eyebrows, plus two audio streams (pitch/intensity and MFCCs).
  • Labels: 10 emotional categories, including happiness, anger, sadness, and "ambiguous."

Key Insights and Ideas

  • Correlations are a Feature, Not a Bug: By using an energy-based model, the robot learns the "joint probability" of modalities. It understands that if it hears a certain vocal pitch, it can already guess the eyebrow position, allowing it to spend its "attention budget" on a different, more surprising modality.
  • The Second Term Advantage: Traditional methods often focus solely on Information Gain. The authors show that their energy-based approach includes a second mathematical term representing the "negative entropy" of the prediction. This makes the robot much more efficient when dealing with mutually correlated data.
  • Emotion-Specific Difficulty: The model revealed that "Happy" states are often identifiable with very few modalities, whereas "Anger" and "Neutral" states are much more complex and require more sensory "steps" to confirm.

Key Results

The authors compared their Energy Minimization (E.min) method against Information Gain Maximization (IG.max) and Random Selection.

  • Speed of Accuracy: In the first step of active perception, the E.min method achieved significantly higher accuracy (~26%) compared to IG.max (~24%) and Random (~21%).
  • Efficiency: The proposed method reached its peak accuracy using fewer modalities than the other approaches.
  • Robustness: The advantage was most pronounced in scenarios where the robot had plenty of training data to learn the correlations between modalities (the "10% test case"). In "novel person" scenarios where correlations were unknown, it performed on par with standard information-gain methods.

Limitations and Future Work

  • The "Static" Assumption: The current model assumes the human's emotion stays the same while the robot is sampling different modalities. In reality, emotions are fluid and change during an interaction.
  • The Cost of Action: The study doesn't factor in the physical "cost" of moving a camera or switching sensors. Future work could incorporate these costs into the energy function.
  • Accuracy Ceiling: The overall emotion recognition accuracy topped out at around 40%. The authors attribute this to the fact that the MDBN (representation) and the FFNN (classifier) were trained separately. They suggest that an end-to-end energy-based model could significantly boost these numbers.
  • Real-World Testing: The next step is moving from the IEMOCAP dataset to real-time interactions with physical robots.

Summary updated

Abstract

During communication, humans express their emotional states using various modalities (e.g., facial expressions and gestures), and they estimate the emotional states of others by paying attention to multimodal signals. To ensure that a communication robot with limited resources can pay attention to such multimodal signals, the main challenge involves selecting the most effective modalities among those expressed. In this study, we propose an active perception method that involves selecting the most informative modalities using a criterion based on energy minimization. This energy-based model can learn the probability of the network state using energy values, whereby a lower energy value represents a higher probability of the state. A multimodal deep belief network, which is an energy-based model, was employed to represent the relationships between the emotional states and multimodal sensory signals. Compared to other active perception methods, the proposed approach demonstrated improved accuracy using limited information in several contexts associated with affective human-robot interaction. We present the differences and advantages of our method compared to other methods through mathematical formulations using, for example, information gain as a criterion. Further, we evaluate performance of our method, as pertains to active inference, which is based on the free energy principle. Consequently, we establish that our method demonstrated superior performance in tasks associated with mutually correlated multimodal information.
BibTeX
@inproceedings{horii2021active,
  author = {Takato Horii and Yukie Nagai},
  title = {{Active Inference Through Energy Minimization in Multimodal Affective Human–Robot Interaction}},
  booktitle = {Frontiers in Robotics and AI},
  year = {2021},
  doi = {10.3389/frobt.2021.684401},
}
Figure