Improving Meeting Inclusiveness using Speech Interruption Analysis

Szu-Wei Fu, Yaran Fan, Yasaman Hosseinkashi, Jayant Gupchup, Ross CutlerACM Multimedia 2022 · DOI

TL;DR

This paper tackles the challenge of making remote meetings more inclusive by focusing on speech interruptions. They introduce a novel system that detects "failed interruptions" – instances where someone tries to speak but doesn't get the floor – and propose using this detection to nudge users to raise their virtual hand, thereby improving participation and inclusivity.

What Problem Does This Paper Address?

In the era of remote work, virtual meetings have become ubiquitous. However, ensuring that all participants, especially those joining remotely, have an equal opportunity to contribute is a significant challenge. A recent study highlighted that the ability for remote participants to interrupt and gain the floor is a key factor in meeting inclusiveness. While tools like the "virtual raise hand" (VRH) feature exist, their adoption is low (less than 1% of meetings). This paper addresses the gap in understanding and actively improving the dynamics of speech interruptions in remote meetings, specifically focusing on instances where participants attempt to speak but are unsuccessful in gaining the floor.

Methods

The core of the proposed system is a machine learning model designed to detect "failed interruptions." This involves several key steps:

  • Speech Overlap Categorization: The authors define and categorize different types of speech overlaps, including backchannels, failed interruptions, successful interruptions, and laughter. This forms the basis for their labeling efforts.
  • Feature Extraction: They leverage state-of-the-art self-supervised learning (SSL) representations, specifically WavLM, to extract rich audio embeddings. These embeddings are then processed using an attention pooling mechanism to create utterance-level representations.
  • Classifier Model: A feed-forward neural network (DNN) is trained on these representations to classify speech overlaps. The model takes the last 5 seconds of audio from both the interrupter's channel and a mixed channel of all other participants.
  • Propensity Score Stratification: To analyze the impact of VRH usage on meeting inclusiveness without a controlled experiment, they employ propensity score stratification. This method helps to create comparable groups of meetings with and without VRH engagement by accounting for confounding factors like meeting size and duration.

Datasets

The paper introduces a newly created labeled dataset for speech interruption analysis.

  • Public Datasets: AMI Meeting Corpus and ICSI Meeting Corpus were used to augment training data. These datasets, however, have limitations as they were collected in face-to-face settings and may include cross-talk.
  • Original Data Source: To address the limitations of public datasets and create a more representative dataset for remote meetings, the authors collected 90 hours of new data from 148 remote meetings with 3-4 participants.
  • Labeled Dataset: A total of 40,000 audio clips were created and labeled. This dataset includes categories like backchannel, failed interruption, and successful interruption. The labeling process involved both crowdsourcing and expert labeling, with a target accuracy of 95% for the test set.

Key Insights and Ideas

  • "Failed Interruption" as a Key Indicator: The paper identifies "failed interruptions" as a critical signal of potential disengagement or exclusion in remote meetings.
  • SSL Embeddings are Crucial: The use of advanced SSL models like WavLM significantly outperforms traditional acoustic features (MFCC, spectrogram) in detecting failed interruptions, highlighting the importance of semantic information in speech.
  • Bidirectional Audio Information is Beneficial: Including audio from both the interrupter and the mixed audio of other participants (two channels) leads to better performance than using only the interrupter's audio (one channel). This suggests that understanding the broader conversational context is vital.
  • VRH Usage Correlates with Inclusiveness: Even without a controlled experiment, the analysis suggests that meetings where the VRH feature is used tend to be more inclusive.

Key Results

  • High Accuracy Detector: The developed failed speech interruption detector achieves an Area Under the Curve (AUC) of 0.95, with a True Positive Rate (TPR) of over 50% at a False Positive Rate (FPR) of less than 1%. This performance is deemed sufficient for production use.
  • VRH Impact: Propensity score stratification indicates that using the VRH feature can lead to an absolute increase of 3.4% in the predicted inclusive score for meetings with more than two participants.
  • Dataset Quality: The newly created labeled dataset achieved a labeling accuracy of 95% (Fleiss' Kappa).

Limitations and Future Work

  • Dataset Expansion: The authors acknowledge that the current dataset is limited to English and needs expansion to support multiple languages and diverse scenarios (varying network latency, device conditions, and acoustic environments).
  • Hosting a Challenge: They plan to host a challenge and release a baseline model to stimulate further research in this area.
  • Integration into Teams: The model is being integrated into Microsoft Teams, and AB tests are planned to measure its impact on inclusiveness and overall meeting effectiveness.
  • Acoustic Echo Canceller: Development of a full-duplex machine learning-based acoustic echo canceller is underway, which is expected to further aid remote participants in interrupting and participating.

Summary updated

BibTeX
@inproceedings{fu2022improving,
  author = {Szu-Wei Fu and Yaran Fan and Yasaman Hosseinkashi and Jayant Gupchup and Ross Cutler},
  title = {{Improving Meeting Inclusiveness using Speech Interruption Analysis}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3548379},
}
Figure