Human-Centric Research for NLP: Towards a Definition and Guiding Questions

Bhushan Kotnis, Kiril Gashteovski, Julia Gastinger, Giuseppe Serra, Francesco Alesiani, Timo Sztyler, Ammar Shaker, Na Gong, Carolin Lawrence, Zhao XuarXiv 2022 · OpenReview

Abstract

With Human-Centric Research (HCR) we can steer research activities so that the research outcome is beneficial for human stakeholders, such as end users. But what exactly makes research human-centric? We address this question by providing a working definition and define how a research pipeline can be split into different stages in which human-centric components can be added. Additionally, we discuss existing NLP with HCR components and define a series of guiding questions, which can serve as starting points for researchers interested in exploring human-centric research approaches. We hope that this work would inspire researchers to refine the proposed definition and to pose other questions that might be meaningful for achieving HCR.

Introduction

In the current Natural Language Processing (NLP) landscape, research is often driven by researcher intuition and the pursuit of "State-of-the-Art" (SOTA) scores on public leaderboards. While this drives technical progress, it creates a gap between academic success and real-world utility. Many NLP projects are not truly human-centric because the research questions, datasets, and evaluation metrics are designed entirely by researchers, often leading to a "solution looking for a problem."

Researchers at NEC Laboratories Europe propose a formal framework to move NLP research away from "ivory tower" curiosity and toward solving real-world problems by involving stakeholders throughout the entire research lifecycle. They define Human-Centric Research (HCR) as any project where external stakeholders—like end-users or domain experts—actively participate alongside researchers.

The HCR Framework

The proposed methodological framework breaks down the research process into four distinct stages. Unlike traditional pipelines, HCR requires the active participation of stakeholders in each phase to ensure the research provides tangible benefits to the people it's meant to serve.

The Human-Centric Research (HCR) lifecycle and its key stakeholders.

The framework identifies several key stakeholders beyond just the researcher, including end users, domain experts, data collectors, and developers. By involving these groups, researchers can move beyond standard crowdsourcing and researcher-led constraints toward a more collaborative and impactful model.

Guiding Questions for HCR

To help practitioners implement this framework, the authors provide a taxonomy of "Guiding Questions" across the four research stages:

  1. Problem Definition: Identify the specific "pain points" of target stakeholders rather than selecting a task based on popularity.
  2. Data Collection: Involve stakeholders in the planning and creative execution of data gathering, ensuring the data reflects real-world needs.
  3. Model Building: Utilize "Human-in-the-loop" (HITL) designs where users can interact with, correct, or provide feedback to the model during training or inference.
  4. Task Evaluation: Shift from averaged metrics (like Accuracy or F1) to evaluations based on stakeholder requirements, moving up an "evaluation ladder" from automatic metrics to real-world stakeholder testing.
Example guideline questions for the four research stages in the context of text summarization.

As shown in the example for summarization, these questions force researchers to consider the utility of the output: "What do the end users need in a summary?" or "Can we improve end users' decision speed or accuracy by giving them a summary?"

Key Insights and Discussion

A core tenet of this framework is the Participation Requirement. Research is only human-centric if stakeholders actively participate; simply using a dataset of human speech is insufficient. This shift critiques the use of common datasets like CNN/Daily Mail or benchmarks like GLUE, which are often chosen for convenience rather than their ability to meet specific human needs.

By providing a structured taxonomy, the authors aim to help the community align on what HCR actually means and provide a checklist for researchers to evaluate the human-centricity of their own work.

Limitations and Future Work

As a young research field, this framework is a first draft intended to spark discussion. The authors acknowledge that HCR is significantly more resource-intensive than traditional research, requiring more time and funding to engage external stakeholders. Furthermore, human-centricity is inherently subjective; what benefits one group may not benefit another. Future work will focus on lowering these barriers and refining these questions for diverse research settings.

Summary updated

Abstract

With Human-Centric Research (HCR) we can steer research activities so that the research outcome is beneficial for human stakeholders, such as end users. But what exactly makes research human-centric? We address this question by providing a working definition and define how a research pipeline can be split into different stages in which human-centric components can be added. Additionally, we discuss existing NLP with HCR components and define a series of guiding questions, which can serve as starting points for researchers interested in exploring human-centric research approaches. We hope that this work would inspire researchers to refine the proposed definition and to pose other questions that might be meaningful for achieving HCR.
BibTeX
@misc{kotnis2022human,
  author = {Bhushan Kotnis and Kiril Gashteovski and Julia Gastinger and Giuseppe Serra and Francesco Alesiani and Timo Sztyler and Ammar Shaker and Na Gong and Carolin Lawrence and Zhao Xu},
  title = {{Human-Centric Research for NLP: Towards a Definition and Guiding Questions}},
  year = {2022},
  eprint = {2207.04447},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.04447},
}
Figure