Narrative Datasets through the Lenses of NLP and HCI

Sharifa Sultana, Renwen Zhang, Hajin Lim, Maria AntoniakNAACL 2022

TL;DR

This paper bridges the gap between Natural Language Processing (NLP) and Human-Computer Interaction (HCI) by analyzing how each field approaches narrative datasets. It highlights key tensions and opportunities for collaboration, suggesting mixed-methods approaches can lead to richer datasets and deeper understandings of human stories.

What Problem Does This Paper Address?

The paper addresses the disconnect between how Natural Language Processing (NLP) and Human-Computer Interaction (HCI) research communities collect and utilize narrative data. NLP often focuses on large-scale, decontextualized datasets for tasks like story generation or understanding, sometimes leading to biases and a lack of depth. Conversely, HCI emphasizes rich, contextualized data from smaller samples, often gathered through qualitative methods like interviews and ethnography, which can be time-consuming and difficult to scale. This divergence creates a gap in understanding the full spectrum of narrative phenomena and their real-world implications. The authors aim to illuminate these differences and propose ways to bridge them for more comprehensive narrative research.

Methods

The paper doesn't introduce new computational methods or algorithms. Instead, its primary "method" is a comparative analysis of existing approaches in NLP and HCI for narrative dataset collection and study. It draws upon established research within both fields to:

  • Outline dominant framings of narrative: How NLP and HCI conceptualize and operationalize "narrative."
  • Identify key tensions: Differences in dataset size, contextuality, positionality, and dataset design.
  • Propose a mixed-methods approach: Suggesting how to combine NLP's scalability with HCI's depth to create more robust narrative research.

The authors advocate for a synthesis of qualitative HCI methods (like interviews, ethnography, and open coding) with NLP techniques (like topic modeling and computational analysis) to achieve a more holistic understanding of narratives.

Datasets

The paper discusses various types of datasets used in NLP and HCI research without introducing new ones.

NLP Datasets: * Artificial Story Datasets: Often created for specific "shared tasks" (e.g., Story Cloze task) and labeled by crowdworkers, these are typically short, decontextualized scenarios. Examples include ROCStories. * Curated Datasets: These are gathered from existing sources like novels, news stories, biographies, or personal stories shared online. They are often large but collected after the fact and may lack explicit context. * Naturalistic NLP Datasets: A smaller category that includes datasets collected with some grounding in personal experiences or social contexts, such as autobiographical stories collected from crowdworkers (e.g., Sap et al., 2020). * Massive, Pretrained Datasets: Datasets like "The Pile" used for training large language models, which are too vast for detailed documentation and can encode biases.

HCI Datasets: * HCI research often relies on data collected through qualitative methods like interviews, focus groups, and ethnography, which are inherently contextual and often smaller in scale. * HCI also utilizes curated social media data for analysis, similar to NLP, but with a stronger emphasis on understanding the context of use and user experiences.

Key Insights and Ideas

  • Narrative is situated: Unlike many NLP datasets that strip away context, HCI emphasizes that narratives are deeply embedded in specific social and personal contexts. This situatedness is crucial for understanding their meaning and impact.
  • Tension between volume and depth: NLP excels at analyzing vast quantities of data, offering breadth, while HCI's qualitative methods provide deep, nuanced insights from smaller samples. Bridging this requires finding ways to combine both.
  • Contextuality as a bridge: The inherent situatedness of narratives can serve as a key concept for integrating NLP and HCI. Researchers can "zoom in" on specific stories with HCI methods and "zoom out" to analyze broader patterns with NLP.
  • Positionality matters: Both NLP and HCI researchers need to be aware of their own biases and the positions of narrative authors. HCI's emphasis on reflexivity can inform NLP practices, especially when dealing with personal narratives.
  • Openness to discovery: HCI's open-ended research goals and open coding processes can inspire NLP to move beyond task-specific, pre-defined labels towards more discovery-driven approaches, even allowing for disagreements among annotators.
  • Mixed methods for richer understanding: Combining NLP's ability to analyze large-scale patterns with HCI's deep contextual understanding can lead to more comprehensive and valid insights into narrative phenomena and their social implications.

Key Results

This paper is primarily a conceptual and analytical work, not an empirical study with quantitative results. Therefore, there are no specific numerical findings to report. The "results" are the insights and arguments presented regarding the differences and potential synergies between NLP and HCI approaches to narrative datasets.

Limitations and Future Work

Limitations: * The paper acknowledges that it does not aim to provide solutions to all challenges but rather to highlight points of difference and windows for collaboration. * The discussion is constrained to NLP and qualitative HCI, recognizing that other fields (like literary studies) also engage with narratives.

Future Work: * The authors advocate for the development of mixed-methods approaches that draw from both NLP and HCI to create richer narrative datasets and achieve more holistic understandings. * They suggest that by combining these disciplines, new questions can be raised and new directions for narrative research can be opened up. * Specific future directions include exploring how platform design, moderation, and other contextual features shape online narratives, and how to incorporate narrative authors into dataset design.

Summary updated

BibTeX
@inproceedings{sultana2022narrative,
  author = {Sharifa Sultana and Renwen Zhang and Hajin Lim and Maria Antoniak},
  title = {{Narrative Datasets through the Lenses of NLP and HCI}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure