Guiding Principles for Participatory Design-inspired Natural Language Processing

Tommaso Caselli, Roberto Cibin, Costanza Conforti, Enrique Encinas, Maurizio TeliACL 2021

TL;DR

This paper introduces nine guiding principles for integrating Participatory Design (PD) into Natural Language Processing (NLP) development. By bridging PD and NLP, the authors aim to create more democratic, fairer, and less-biased language technologies, addressing critical issues in data collection, annotation, and deployment.

What Problem Does This Paper Address?

The development of Natural Language Processing (NLP) systems often faces challenges related to bias, fairness, and democratic representation. While NLP has made significant strides in processing and generating human language, these advancements can inadvertently perpetuate societal biases present in the data they are trained on. Furthermore, the design and deployment of these powerful technologies often occur without meaningful input from the very communities they are intended to serve. This can lead to systems that are not only biased but also misaligned with user needs and cultural contexts. Participatory Design (PD), a field focused on involving end-users in the design process, offers a potential solution. However, the integration of PD principles into the complex and data-driven world of NLP is not straightforward. This paper seeks to bridge this gap by proposing concrete guiding principles for a PD-inspired approach to NLP development.

Methods

The paper's methodology is unique, structured around nine guiding principles, each presented as a distinct section. The authors draw upon their collective experience and ongoing dialogue between designers and NLP experts.

  • Principles 1-3: These principles lay the foundational assumptions and methods of community-based PD, emphasizing mutual learning, a variety of research and design methods (workshops, observations, cards, games, scenarios, prototypes), and the understanding that communities are dynamic and can form through the design process itself.
  • Principles 4-6: These principles focus on the impact of PD on NLP system design, specifically addressing:
  • Data Collection & Annotation: Shifting the paradigm from "language as data" to "language as people," recognizing that language data is produced by human speakers and that communities should be involved in the NLP pipeline.
  • Deployment & Evaluation: Advocating for "use-before-use" (envisioning tool use before implementation) and "design-after-design" (acknowledging that design continues after delivery through user appropriation and feedback). This includes considering the context-specific nature of participation and the need for transparent communication about data usage.
  • Principles 7-9: These principles guide a new reflexivity for NLP research, emphasizing:
  • Text as a Means, Not an End: NLP outputs should serve human needs rather than merely imitating language production, promoting a problem-solving perspective over pure automation.
  • Consent and Intrusion: Navigating the delicate balance of community involvement, ensuring informed consent and avoiding predatory practices.
  • Combining Goals: The need to integrate research goals, funding requirements, and the concrete social and political dynamics of the communities involved, positioning the researcher/designer as an intermediary.

The authors also utilize two fictional design scenarios to elicit the identification of principles 4-6, grounding the abstract principles in practical considerations.

Datasets

Not discussed in the paper. The paper focuses on principles and methodologies rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • "Language as People" over "Language as Data": This is a fundamental shift. Instead of treating language data as an inert resource to be mined, the paper argues for recognizing that language is produced by people, and their agency and context are paramount.
  • Communities are Not Static: PD acknowledges that communities are not pre-defined entities but can emerge and evolve through the design process itself, challenging traditional NLP sampling methods.
  • Design is an Ongoing, Messy Process: The concepts of "use-before-use" and "design-after-design" highlight that the design and use of NLP tools are iterative and continuous, extending beyond initial deployment.
  • Ethical Engagement is Crucial: Building trust with communities through respect, equity, and transparency is essential, especially when dealing with sensitive data and marginalized groups.
  • NLP Outputs Should Be Actionable, Not Mimetic: The goal of NLP should be to facilitate human understanding and action, not just to replicate human language patterns. This implies designing systems that provide interpretable and useful outputs for users.
  • The Researcher as Intermediary: In PD-inspired NLP, researchers and designers play a vital role in navigating the complex interplay of community needs, research objectives, and funding agency interests.

Key Results

This paper does not present quantitative results from experiments or evaluations. Its contribution lies in the conceptual framework of nine guiding principles for integrating Participatory Design into NLP. The "results" are the principles themselves, which aim to guide future development practices.

Limitations and Future Work

  • Context-Specificity: The authors acknowledge that the applicability of these principles is context-specific and requires situated appropriation and transformation. While they aim for broad applicability, the nuances of different cultural and social contexts will necessitate adaptation.
  • Funding and Industrial Pressures: Principle 9 highlights the challenge of balancing community needs with external funding requirements and industrial pressures, which can be a significant hurdle in implementing truly participatory approaches.
  • Future Work: The paper implicitly suggests future work in developing concrete methodologies and case studies that demonstrate the successful application of these principles in real-world NLP projects. It also points to the need for further research into how to effectively translate community visions into technical alternatives and how to foster genuine collaboration that avoids "participation washing."

Summary updated

BibTeX
@inproceedings{caselli2021guiding,
  author = {Tommaso Caselli and Roberto Cibin and Costanza Conforti and Enrique Encinas and Maurizio Teli},
  title = {{Guiding Principles for Participatory Design-inspired Natural Language Processing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure