Feasibility, Opportunities, and Challenges of Utilizing AI for Collaborative Qualitative Analysis

Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka-Wei Lee, Simon T. Perrault

Abstract

While AI-assisted individual qualitative analysis has been substantially studied, AI-assisted collaborative qualitative analysis (CQA)—a process that involves multiple researchers working together to interpret data—remains relatively unexplored. After identifying CQA practices and design opportunities through formative interviews, we designed and implemented CoAIcoder, a tool leveraging AI to enhance human-to-human collaboration within CQA through four distinct collaboration methods. With a between-subject design, we evaluated CoAIcoder with 32 pairs of CQA-trained participants across common CQA phases under each collaboration method. Our findings suggest that while using a shared AI model as a mediator among coders could improve CQA efficiency and foster agreement more quickly in the early coding stage, it might affect the final code diversity. We also emphasize the need to consider the independence level when using AI to assist human-to-human collaboration in various CQA scenarios. Lastly, we suggest design implications for future AI-assisted CQA systems.

Introduction

Qualitative Analysis (QA) is a fundamental yet labor-intensive process of labeling text to identify recurring themes. When researchers collaborate (Collaborative Qualitative Analysis or CQA), the complexity increases as they must reconcile differing interpretations to develop a unified codebook. While AI has successfully assisted individual coders, its potential to facilitate collaboration between humans remains under-explored. Current collaborative environments, such as shared documents, often suffer from "coding bias," where real-time visibility of a partner’s work can stifle independent thought. This research investigates whether an AI can act as a silent mediator, offering suggestions based on collective history without the pressure of direct peer observation.

Methods: The CoAIcoder System

We developed CoAIcoder, a web-based platform integrating a collaborative text editor with a Natural Language Understanding (NLU) backend using the Rasa NLU pipeline. As users code, the model is fine-tuned every 10–20 seconds based on their specific coding history.

Overview of the CoAIcoder interface and the four collaboration methods evaluated in the study.

The system supports four collaboration methods, as illustrated in the figure above:

  • Condition A: Traditional coding (No AI, Asynchronous).
  • Condition B: Individual AI (AI-assisted, but models are not shared).
  • Condition C: Shared AI, Asynchronous (Partners code at different times).
  • Condition D: Shared AI, Synchronous (Partners code at the same time).

In "Shared Model" conditions (C and D), the AI suggests codes based on the combined inputs of both partners, acting as an indirect bridge between their conceptual frameworks.

Participants and Data

The study involved 32 pairs of CQA-trained participants (64 individuals) with varying levels of QA experience and backgrounds in Human-Computer Interaction (HCI).

Demographics of the study participants, showing a range of expertise and software familiarity.

Participants coded mock interview transcripts (approx. 1,000 words each) covering general topics like leadership and personal challenges, selected for their accessibility to the diverse participant pool.

Results

The evaluation revealed a significant Efficiency-Diversity Trade-off.

Efficiency and Agreement

Shared AI models significantly improved coding speed and initial agreement. Condition C (Shared AI, Asynchronous) was the fastest, completing the initial phase in 16.06 minutes compared to 19.00 minutes for the no-AI baseline. Furthermore, Condition D (Shared AI, Synchronous) achieved the highest initial Inter-Rater Reliability (Cohen’s Kappa of 0.31), nearly double the 0.16 observed in the baseline and individual-AI groups.

Impact on Code Diversity

However, this efficiency came at the cost of analytical breadth. The baseline group (No AI) produced an average of 14.88 unique codes, whereas Shared AI groups dropped to as low as 9.88. The AI's tendency to nudge users toward existing labels reduced the likelihood of participants inventing new, nuanced categories.

Discussion and Design Implications

The AI acted as a "soft bridge," allowing for terminology alignment before formal discussion. Participants noted that the AI was particularly helpful for "getting started," acting as a scaffold for novices.

We distinguish between efficiency-oriented tasks (where speed is prioritized) and creativity-oriented tasks (where deep, independent interpretation is vital). AI mediation is highly beneficial for the former but poses risks for the latter. Future systems should consider:

  • LLM Integration: Moving beyond intent-classification to Large Language Models (LLMs) for more semantic suggestions.
  • Explainability: Indicating why a code is suggested (e.g., "Your partner used this label") to help users maintain independence.
  • Latency: Reducing the 10-20 second training cycle to prevent disruptions in user flow.

Conclusion

CoAIcoder demonstrates that AI can effectively mediate human collaboration in qualitative analysis. While shared models foster efficiency and early alignment, designers must balance these gains against the potential loss of interpretive diversity.

Summary updated

Abstract

While AI-assisted individual qualitative analysis has been substantially studied, AI-assisted collaborative qualitative analysis (CQA)-a process that involves multiple researchers working together to interpret data-remains relatively unexplored. After identifying CQA practices and design opportunities through formative interviews, we designed and implemented CoAIcoder, a tool leveraging AI to enhance human-to-human collaboration within CQA through four distinct collaboration methods. With a between-subject design, we evaluated CoAIcoder with 32 pairs of CQA-trained participants across common CQA phases under each collaboration method. Our findings suggest that while using a shared AI model as a mediator among coders could improve CQA efficiency and foster agreement more quickly in the early coding stage, it might affect the final code diversity. We also emphasize the need to consider the independence level when using AI to assist human-to-human collaboration in various CQA scenarios. Lastly, we suggest design implications for future AI-assisted CQA systems.
BibTeX
@article{gao2024feasibility,
  author = {Jie Gao and Kenny Tsu Wei Choo and Junming Cao and Roy Ka-Wei Lee and Simon T. Perrault},
  title = {{Feasibility, Opportunities, and Challenges of Utilizing AI for Collaborative Qualitative Analysis}},
  journal = {ACM Trans. Comput. Hum. Interact.},
  year = {2024},
  doi = {10.1145/3617362},
}
Figure