TL;DR
This paper introduces a multimodal machine learning approach to predict subjective user experience in videoconferencing, specifically identifying moments of low conversational fluidity and enjoyment. By combining audio, facial, and body motion data, their models achieve impressive accuracy, paving the way for automated analysis and potential real-time interventions to improve online interactions.
What Problem Does This Paper Address?
Videoconferencing has become ubiquitous, yet it often falls short of the natural flow and engagement of in-person conversations, leading to phenomena like "Zoom fatigue." Identifying the specific moments where conversations become awkward, disengaging, or frustrating is crucial for understanding and improving this technology. However, these "negative experience" moments are rare and subjective, making them difficult to detect and analyze using traditional methods that require extensive manual annotation. This paper aims to bridge this gap by using machine learning to automatically predict these high-level subjective outcomes from multimodal signals.
Methods
The core of this work lies in a multimodal machine learning pipeline. They extract features from three domains:
- Audio: Using pre-trained deep neural networks (VGGish, YAMNet, Wav2Vec2) to generate embeddings that capture acoustic characteristics.
- Facial Action: Employing OpenFace to extract facial action unit intensities, representing subtle facial movements.
- Body Motion: Calculating Granger Causality (GC) coupling strength of body motion time series between participants, a measure of coordination.
These features are then fed into a logistic regression model trained using stochastic gradient descent. The models are optimized using Bayesian optimization for hyperparameter tuning and evaluated using cross-validation to ensure generalizability. They explore the predictive power of different feature combinations and time windows (e.g., 7-second clips vs. 3-second pre-event features).
Datasets
The study utilizes the RoomReader corpus, a newly introduced benchmark containing 30 Zoom videoconference sessions totaling nearly 9 hours and involving 118 participants. To focus on challenging moments, they specifically sampled clips where participants were either silent or speaking simultaneously. These clips were then manually rated by over 500 participants on a 5-point Likert scale for fluidity and enjoyment, and to classify conversational events (gaps, backchanneling, interruptions).
Key Insights and Ideas
- Multimodality is Key: Combining audio, facial, and body motion signals significantly outperforms using any single modality for predicting conversational outcomes.
- Audio Dominance: Domain-general audio features (like VGGish) are surprisingly critical, often outperforming speech-specific models and facial/body motion features alone. This suggests that the acoustic nuances of conversation carry substantial information about user experience.
- Pre-event vs. During-event: While audio features capture dynamic shifts in conversation, facial features seem to reflect more stable, underlying states of engagement (or disengagement).
- Fluidity and Enjoyment Link: Low fluidity is a strong predictor of low enjoyment, but the reverse is not as pronounced, suggesting that awkward silences or interruptions are more detrimental to enjoyment than a lack of enjoyment necessarily leading to a lack of fluidity.
- Conversational Event Signatures: Different conversational events (gaps, interruptions, backchanneling) have distinct multimodal "signatures," with backchanneling being a positive indicator and gaps being particularly detrimental to enjoyment.
Key Results
- The best performing models achieved an ROC-AUC of up to 0.87 on hold-out videoconference sessions for predicting fluidity and enjoyment.
- Models combining VGGish audio embeddings, facial actions, and body motion GC demonstrated the highest performance.
- For predicting conversational events, a 7-second model using VGGish, facial, and body motion features achieved an ROC-AUC of 0.867.
- Even with only 3-second pre-event features, models could still predict outcomes with reasonable accuracy, with facial features showing surprising resilience.
Limitations and Future Work
The authors acknowledge that the RoomReader corpus, while designed to elicit naturalistic behavior, might be "overly self-consistent," potentially limiting the generalizability of their models to broader, more diverse videoconferencing contexts.
Future work could involve:
- Testing the models on a wider range of videoconferencing scenarios and user groups.
- Developing real-time versions of these models to enable in-the-moment interventions and feedback for users.
- Exploring how to integrate these predictions into videoconferencing platforms to proactively improve user experience.
Summary updated