Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)

Nathaniel Blanchard, Daniel Moreira, Aparna Bharati, Walter J. Scheirer, Jennifer Williams, Steven Kleinegesse, Ramona Comanescu, Oana Radu, Saurav Sahay, Shachi Kumar, Rui Xia, Jonathan Huang, Lama Nachman, Yong Woo, Ye Choi, Chan Song, Imran Sheikh, Harsha Dumpala, Rupayan Chakraborty, Sunil Kumar, Leimin Tian, Catherine Lai, Johanna D. Moore, Noé Tits, Kevin El Haddad, Thierry Dutoit, Hai Pham, Thomas Manzini, Paul Pu Liang, Barnabás Póczos, Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Cheng Wei Ming, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Björn W. Schuller, Stefan Steidl, Anton Batliner, Mohammad Soleymani, David García, Brendan Jou, Bjrn Schuller, Shih-Fu Chang, Maja Pantic, Andreas Stolcke, Noah Coccaro, Rebecca Bates, Paul Taylor, Carol Van Ess-Dykema, Klaus Ries, Chan Lee, Kyu Song, Ji-Hoon Jeong, Woo Ri Choi, Sunil Kumar Kopparapu, Yong, Sri Dumpala, Harsha, Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool, Michael Brady, Andrew Olney, Marci Glaus, Xiaoyi Sun, Martin Nystrand, Borhan Samei, Sean Kelly, Sidney D' Mello, Patrick Donnelly, Brooke Ward, Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette Chang, Sungbok Lee, Shrikanth Narayanan, Minghai Chen, Sen WangOpenalex Unknown 2018 · DOI · OpenAlex

Abstract

In the last decade, video blogs (vlogs) have become an extremely popular method through which people express sentiment. The ubiquitousness of these videos has increased the importance of multimodal fusion models, which incorporate video and audio features with traditional text features for automatic sentiment detection. Multimodal fusion offers a unique opportunity to build models that learn from the full depth of expression available to human viewers. In the detection of sentiment in these videos, acoustic and video features provide clarity to otherwise ambiguous transcripts. In this paper, we present a multimodal fusion model that exclusively uses high-level video and audio features to analyze spoken sentences for sentiment. We discard traditional transcription features in order to minimize human intervention and to maximize the deployability of our model on at-scale real-world data. We select high-level features for our model that have been successful in nonaffect domains in order to test their generalizability in the sentiment detection domain. We train and test our model on the newly released CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset, obtaining an F1 score of 0.8049 on the validation set and an F1 score of 0.6325 on the held-out challenge test set.

Introduction

The proliferation of video blogs (vlogs) has transformed how sentiment is expressed and captured online. While traditional sentiment analysis has focused heavily on text, the multimodal nature of vlogs—incorporating facial expressions, gestures, and vocal tone—presents a richer data source. However, most multimodal models remain tethered to human-generated transcripts, creating a significant bottleneck for real-world, at-scale applications.

In the lead paper for the 2018 Grand Challenge on Human Multimodal Language, "Getting the subtext without the text," Blanchard et al. propose a paradigm shift: a sentiment classification pipeline that completely ignores linguistic content in favor of visual and acoustic "subtext."

The Problem: The Transcription Bottleneck

Traditional multimodal research often treats text as the primary modality, with video and audio acting as supplementary features. This approach relies on manual transcription, which is prohibitively expensive and slow for massive datasets. While Automatic Speech Recognition (ASR) offers an alternative, it often fails to capture the nuances of "in-the-wild" social media content.

Furthermore, text alone is frequently ambiguous. A transcript cannot easily distinguish between a sincere "Great" and a sarcastic one. By focusing on how something is said (prosody) and the speaker's physical delivery (gestures and expressions), the authors aim to build a model that is both scalable and capable of resolving linguistic ambiguity without human intervention.

Methodology: A Bag-of-Features Approach

The researchers implemented a three-level Bag-of-Features (BoF) pipeline, adapting techniques typically reserved for computer vision to the domain of sentiment analysis.

1. Feature Extraction

  • Visual Modality: The model utilizes Temporal Robust Features (TRoF). Originally designed for detecting motion-heavy content like violence, TRoF isolates spatiotemporal segments of video to describe movement. This study tests whether these features can generalize to the subtle movements associated with human affect.
  • Acoustic Modality: Using the openSMILE toolkit, the team extracted prosodic features including pitch, loudness, and voicing probability. Crucially, the model ignores the phonemes and words, focusing entirely on the non-verbal acoustic signal.

2. Mid-level Representation

To bridge the gap between raw signals and classification, the authors used Gaussian Mixture Models (GMMs) to create "codebooks." These codebooks serve as a vocabulary of visual and acoustic patterns, effectively quantizing the data into a "Bag-of-Visual-Words" and "Bag-of-Audio-Words."

3. Classification and Fusion

Separate Linear Support Vector Machines (SVMs) were trained for each modality. The authors then explored two fusion strategies:

  • Score-level fusion: Averaging the confidence scores from both the audio and video models.
  • Output-level fusion: Binning and thresholding the raw scores to produce a final sentiment prediction.

Dataset: CMU-MOSEI

The model was evaluated using the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset. At the time of the study, this was a significant upgrade over previous datasets, containing over 23,500 sentence-level videos (65+ hours) from more than 1,000 diverse YouTube speakers covering a wide array of topics.

Results and Discussion

The experimental results highlight a clear trade-off between scalability and peak accuracy.

  • Performance: The output-level fusion model achieved the best results on the CMU-MOSEI validation set with an F1 score of 0.8049 and a Mean Absolute Error (MAE) of 0.7760. On the more rigorous Grand Challenge held-out test set, it reached an F1 score of 0.6325.
  • Modality Strength: A key finding was the disparity between modalities. The video-only model (F1: 0.75) significantly outperformed the audio-only model (F1: 0.58). This suggests that while prosody is valuable, visual cues captured by TRoF are more robust indicators of sentiment when linguistic context is absent.
  • Generalizability: The success of TRoF features—originally intended for non-affect domains—demonstrates that high-level motion features are surprisingly effective at capturing human sentiment.

Conclusion and Future Work

Blanchard et al. demonstrate that a "text-free" approach is viable for large-scale sentiment detection, prioritizing deployability. While the exclusion of text results in lower accuracy compared to transcript-reliant models, the BoF approach offers a path toward analyzing the vast quantities of unlabelled video data generated daily.

Future work may look to incorporate "noisy" text from ASR to bridge the performance gap or improve the mapping of SVM distances to sentiment intensity. For now, the study serves as a benchmark for how much "subtext" can be extracted when the words themselves are stripped away.

Summary updated

Abstract

In the last decade, video blogs (vlogs) have become an extremely popular method through which people express sentiment. The ubiquitousness of these videos has increased the importance of multimodal fusion models, which incorporate video and audio features with traditional text features for automatic sentiment detection. Multimodal fusion offers a unique opportunity to build models that learn from the full depth of expression available to human viewers. In the detection of sentiment in these videos, acoustic and video features provide clarity to otherwise ambiguous transcripts. In this paper, we present a multimodal fusion model that exclusively uses high-level video and audio features to analyze spoken sentences for sentiment. We discard traditional transcription features in order to minimize human intervention and to maximize the deployability of our model on at-scale real-world data. We select high-level features for our model that have been successful in nonaffect domains in order to test their generalizability in the sentiment detection domain. We train and test our model on the newly released CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset, obtaining an F 1 score of 0.8049 on the validation set and an F 1 score of 0.6325 on the held-out challenge test set.
BibTeX
@inproceedings{blanchard2018proceedings,
  author = {Nathaniel Blanchard and Daniel Moreira and Aparna Bharati and Walter J. Scheirer and Jennifer Williams and Steven Kleinegesse and Ramona Comanescu and Oana Radu and Saurav Sahay and Shachi Kumar and Rui Xia and Jonathan Huang and Lama Nachman and Yong Woo and Ye Choi and Chan Song and Imran Sheikh and Harsha Dumpala and Rupayan Chakraborty and Sunil Kumar and Leimin Tian and Catherine Lai and Johanna D. Moore and Noé Tits and Kevin El Haddad and Thierry Dutoit and Hai Pham and Thomas Manzini and Paul Pu Liang and Barnabás Póczos and Ashwin Ram and Rohit Prasad and Chandra Khatri and Anu Venkatesh and Raefer Gabriel and Qing Liu and Jeff Nunn and Behnam Hedayatnia and Cheng Wei Ming and Ashish Nagar and Eric King and Kate Bland and Amanda Wartick and Björn W. Schuller and Stefan Steidl and Anton Batliner and Mohammad Soleymani and David García and Brendan Jou and Bjrn Schuller and Shih-Fu Chang and Maja Pantic and Andreas Stolcke and Noah Coccaro and Rebecca Bates and Paul Taylor and Carol Van Ess-Dykema and Klaus Ries and Chan Lee and Kyu Song and Ji-Hoon Jeong and Woo Ri Choi and Sunil Kumar Kopparapu and Sri Harsha Dumpala and Yong and Sri Dumpala and Harsha and Herbert Bay and Andreas Ess and Tinne Tuytelaars and Luc Van Gool and Nathaniel Blanchard and Michael Brady and Andrew Olney and Marci Glaus and Xiaoyi Sun and Martin Nystrand and Borhan Samei and Sean Kelly and Sidney D' Mello and Nathaniel Blanchard and Patrick Donnelly and Andrew Olney and Borhan Samei and Brooke Ward and Xiaoyi Sun and Sean Kelly and Martin Nystrand and Sidney Mello and Carlos Busso and Murtaza Bulut and Chi-Chun Lee and Abe Kazemzadeh and Emily Mower and Samuel Kim and Jeannette Chang and Sungbok Lee and Shrikanth Narayanan and Minghai Chen and Sen Wang},
  title = {{Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML)}},
  booktitle = {Openalex Unknown},
  year = {2018},
  doi = {10.18653/v1/w18-33},
}
Figure