When Text and Speech are Not Enough: A Multimodal Dataset of Collaboration in a Situated Task

Ibrahim Khebour, Richard Brutti, Indrani Dey, Rachel Dickler, Kelsey Sikes, Kenneth Lai, Mariah Bradford, Brittany Cates, Paige Hansen, Changsoo Jung, Brett Wisniewski, Corbyn Terpstra, Leanne Hirshfield, Sadhana Puntambekar, Nathaniel Blanchard, James Pustejovsky, Nikhil KrishnaswamyJournal of Open Humanities Data 2024 · DOI

Abstract

To adequately model information exchanged in real human-human interactions, considering speech or text alone leaves out many critical modalities. The channels contributing to the “making of sense” in human-human interactions include but are not limited to gesture, speech, user-interaction modeling, gaze, joint attention, and involvement/engagement, all of which need to be adequately modeled to automatically extract correct and meaningful information. In this paper, we present a multimodal dataset of a novel situated and shared collaborative task, with the above channels annotated to encode these different aspects of the situated and embodied involvement of the participants in the joint activity.
BibTeX
@article{khebour2024when,
  author = {Ibrahim Khebour and Richard Brutti and Indrani Dey and Rachel Dickler and Kelsey Sikes and Kenneth Lai and Mariah Bradford and Brittany Cates and Paige Hansen and Changsoo Jung and Brett Wisniewski and Corbyn Terpstra and Leanne Hirshfield and Sadhana Puntambekar and Nathaniel Blanchard and James Pustejovsky and Nikhil Krishnaswamy},
  title = {{When Text and Speech are Not Enough: A Multimodal Dataset of Collaboration in a Situated Task}},
  journal = {Journal of Open Humanities Data},
  year = {2024},
  doi = {10.5334/johd.168},
}
Figure