Abstract
Emotion recognition (ER) is an important part of building an intelligent human-computer interaction system and plays an important role in human-computer interaction. Often, people express their feelings through a variety of symbols, such as words and facial expressions. A business ecosystem is an economic community based on interacting organizations and individuals. Over time, they develop their capabilities and roles together and tend to develop themselves in the direction of one or more central enterprises. This paper aims to study a multimodal ER method based on attention mechanism. It analyzes the current emotional state of consumers and the development direction of enterprises through multi-modal ER of human emotions and analysis of market trends, so as to provide the most appropriate response or plan. This paper firstly describes the related methods of multimodal ER and deep learning in detail, and briefly outlines the meaning of enterprise strategy in the business ecosystem. Then, two datasets, CMU-MOSI and CMU-MOSEI, are selected to design the scheme for multimodal ER based on self-attention mechanism. Through the comparative analysis of the accuracy of single-modal and multi-modal ER, the self-attention mechanism is applied in the experiment. The experimental results show that the average recognition accuracy of happy under multimodal ER reaches 91.5%.
Introduction
In the modern "business ecosystem"—a decentralized, interconnected network of organizations and individuals—traditional strategic management often fails because it lacks a pulse on the human element. Specifically, enterprises struggle to accurately gauge consumer sentiment through a single lens (e.g., just reading a review or just looking at a facial expression).
Single-modal ER often suffers from low accuracy and a lack of context. For example, a sarcastic text might be read as positive by a text-only model, but the tone of voice (audio) or a rolling of the eyes (video) would tell a different story. The challenge lies in multimodal fusion: how do we combine these disparate data streams—text, sound, and image—without losing the nuanced relationships between them?
Methodology
The authors explore three levels of data fusion: Data Layer Fusion, Feature Layer Fusion, and Decision Layer Fusion. Feature layer fusion, in particular, involves extracting specific features (like facial landmarks or word embeddings) before merging them into a single representation for classification.
The core of the proposed model is the Self-Attention Mechanism integrated with Long Short-Term Memory (LSTM) and Convolutional Neural Networks (CNNs). CNNs are used to extract high-level features from images and audio, while LSTMs handle the temporal dependencies—how emotions change over time.
Self-Attention is the "secret sauce." Unlike standard RNNs that can "forget" earlier parts of a sequence, the self-attention mechanism assigns weights to different parts of the input data, ensuring that the most emotionally significant moments (a specific keyword or a sudden smile) are prioritized regardless of when they occur in the video clip.
The study utilizes two gold-standard benchmarks:
- CMU-MOSI: 2,199 video clips annotated for sentiment intensity from 93 speakers.
- CMU-MOSEI: A larger dataset featuring 3,228 videos and 1,000 different speakers across 250 topics.
Key Insights and Results
The researchers found that text remains the most informative modality. When text data is combined with audio and video, the accuracy jumps significantly, but text-only models still consistently outperform audio-only or video-only models. Traditional Recurrent Neural Networks (RNNs) struggle with long-term dependencies; however, self-attention effectively "compensates for the weak memory of RNNs" by allowing the model to look at the entire sequence simultaneously.
Experimental results demonstrate that the multimodal approach significantly outperforms single-modal benchmarks:
- Multimodal Superiority: On the CMU-MOSI dataset, multimodal fusion (Text+Video+Audio) achieved an accuracy of 81.5%, compared to 74.1% for text alone.
- High Precision: On the CMU-MOSEI dataset, the model reached an average recognition accuracy of 91.5% for "happy" emotions and 82.1% for "anger."
- Model Comparison: The proposed self-attention-based model consistently outperformed "Simple LSTM" and "Hierarchical RNN" architectures, proving that attention mechanisms are superior at capturing contextual information.
Conclusion
This paper frames ER not just as a technical feat, but as a management tool. In a business ecosystem, understanding the "like and love" of consumers allows for more agile, non-centralized decision-making. While the results are promising, limitations include modality imbalance (prioritizing text) and high computational costs. Future work should focus on "in-the-wild" case studies within actual corporate ecosystems to see how these ER insights translate into specific strategic pivots.
Summary updated