TL;DR
This paper introduces DisCaaS, a system that uses a single 360-degree camera to automatically analyze micro-behaviors like speaking and nodding during meetings. By focusing on non-verbal cues, DisCaaS offers a privacy-preserving and reproducible method for understanding meeting dynamics, with potential applications for both in-person and virtual discussions.
What Problem Does This Paper Address?
Meetings are crucial for collaboration and decision-making, but they are also time-consuming and costly for organizations. While much research has focused on analyzing meeting content (verbal communication) or using complex sensor setups, this paper addresses the need for a simpler, more privacy-conscious approach to understanding meeting dynamics. Specifically, it tackles the challenge of automatically recognizing subtle, non-verbal "micro-behaviors" like speaking and nodding, which are often overlooked but can reveal a lot about participant engagement and interaction. The authors aim to develop a system that is contactless, reproducible, and doesn't require access to sensitive meeting content.
Methods
The core of DisCaaS is its reliance on a camera as a "smart sensor." The system employs a 360-degree camera to capture all participants simultaneously. For analyzing micro-behaviors, the researchers leverage the OpenFace toolkit to extract facial landmarks and head pose information (specifically, head rotations).
The key micro-behaviors targeted are speaking and nodding.
- Speaking is detected by analyzing the distance between facial points 62 and 66 (representing the lips).
- Nodding is identified by analyzing the
pose_Rxcomponent of head rotation.
These extracted features are then processed using a sliding window approach (1.06 seconds with 50% overlap) to generate time-series data. Finally, a Random Forest classifier is used to distinguish between "speaking," "nodding," and "other" (non-active) states for each window. The choice of Random Forest is motivated by its robustness to a large number of features and its ability to maintain accuracy without complex hyperparameter tuning.
Datasets
The study utilizes three datasets:
- Offline Meeting Dataset A: Collected from 22 unique participants in 16 five-minute recordings. This dataset was previously collected and is publicly available.
- Offline Meeting Dataset B: A newly collected dataset consisting of 12 unique participants in seven ten-minute recordings.
- Online Meeting Dataset: Data collected from online meetings (using Google Meet) involving six participants in seventeen five-minute sessions. This dataset was created to explore the applicability of the method to virtual environments.
Key Insights and Ideas
- Camera as a Ubiquitous Sensor: The paper strongly advocates for using readily available cameras (especially 360-degree ones) as powerful, non-intrusive sensors for capturing rich behavioral data in meetings.
- Non-Verbal Cues are Informative: The research demonstrates that simple non-verbal cues like speaking and nodding can be reliably detected and are indicative of participant engagement, even without analyzing the meeting's content.
- Speaking is Easier to Detect than Nodding: The results suggest that speaking is generally easier to recognize with higher accuracy than nodding. This is attributed to the more consistent and universal physical manifestation of speaking (mouth movements) compared to the varied and sometimes subtle nature of nodding.
- Offline vs. Online Differences: Feature importance shifts between offline and online meetings. Lip-related features are more crucial for offline analysis, while head rotation becomes more dominant for online settings, likely due to different camera perspectives and participant positioning.
Key Results
- Offline Meetings:
- A macro average F1-score of 67.9% was achieved for the 10-fold random split cross-validation across all offline datasets.
- The leave-one-participant-out cross-validation yielded a macro average F1-score of 62.5%.
- Dataset B (longer individual meeting durations) showed the highest performance, suggesting that more extensive individual behavioral data improves accuracy.
- Online Meetings:
- The 10-fold random split cross-validation achieved a macro average F1-score of 55.3%.
- The leave-one-participant-out cross-validation resulted in a lower macro average F1-score of 31.1%. This significant drop highlights the challenges of applying the method to online settings, particularly with varying camera angles and potential tracking issues.
- Feature Importance: For offline meetings, features related to lip movements (distance between facial points) were most important. For online meetings, head rotation features (
pose_Rx) were more significant.
Limitations and Future Work
- Simultaneous Action Recognition: The current model predicts only one micro-behavior at a time. Future work could explore recognizing simultaneous actions (e.g., speaking while nodding).
- Context of Nodding: The system detects the physical movement of nodding but cannot distinguish between genuine agreement, nervous tics, or other contextual meanings. Future research could aim to incorporate context into nodding recognition.
- Meeting Size and Duration: The experiments were conducted with relatively small meetings (up to four participants) and short durations (10 minutes). Scaling to larger meetings or longer durations, especially with clear face visibility for all participants, remains a challenge.
- Online Meeting Challenges: Tracking faces in online meetings is more difficult due to varying camera angles, screen sharing, and potential distractions. The authors suggest developing custom applications or plugins for online meeting tools to improve robustness.
- Processing Time: The current work does not consider processing time, which would be crucial for real-time micro-behavior recognition.
- Dataset Expansion: The authors emphasize that increasing the size and diversity of datasets, particularly for online meetings, is essential for improving model accuracy.
Summary updated