A Computational Method for Measuring "Open Codes" in Qualitative Analysis

John Chen, Alexandros Lotsos, Lexie Zhao, Caiyi Wang, Jessica Hullman, Bruce Sherin, Uri Wilensky, Michael S. Horn

Abstract

Qualitative analysis is critical to understanding human datasets in many social science disciplines. Open coding is an inductive qualitative process that identifies and interprets "open codes" from datasets. Yet, meeting methodological expectations (such as "as exhaustive as possible") can be challenging. While many machine learning (ML)/generative AI (GAI) studies have attempted to support open coding, few have systematically measured or evaluated GAI outcomes, increasing potential bias risks. Building on Grounded Theory and Thematic Analysis theories, we present a computational method to measure and identify potential biases from "open codes" systematically. Instead of operationalizing human expert results as the "ground truth," our method is built upon a team-based approach between human and machine coders. We experiment with two HCI datasets to establish this method's reliability by 1) comparing it with human analysis, and 2) analyzing its output stability. We present evidence-based suggestions and example workflows for ML/GAI to support open coding.

Introduction

In qualitative research, inductive or "open" coding involves identifying themes and labels directly from raw data. Unlike deductive coding, which relies on predefined categories, inductive coding is exploratory and subjective. This subjectivity presents a significant challenge for evaluation: traditional metrics like Inter-Coder Reliability (ICR) assume a single "correct" set of codes exists. However, in inductive work, the goal is often depth and conceptual variation, where different researchers may identify different but equally valid themes.

As researchers increasingly turn to Generative AI (GAI) to assist in this labor-intensive process, there is a critical need for metrics that can distinguish high-quality analysis from "hallucinations" or "flooding" (the generation of redundant labels). This paper proposes a shift from "truth-based" metrics toward "team-based" metrics, introducing a framework to evaluate coding rigor without a fixed ground truth.

Methodology: The Aggregated Code Space

The core of our approach is the Aggregated Code Space (ACS), a collective conceptual map generated by merging individual codebooks. To achieve this, we developed a four-stage merging algorithm:

  1. Semantic Similarity (Stages 1 & 2): Initial merging based on text embeddings of code labels.
  2. LLM-Enhanced Definitions (Stage 3): An LLM generates definitions for each code based on the specific data examples to which it was applied. Merging then proceeds based on these conceptual definitions rather than just labels.
  3. Iterative Refinement (Stage 4): A dual-threshold system (lower and upper) uses a penalty system for codes that share too many or too few specific data examples, ensuring the final ACS is conceptually distinct.
Conceptual overview of merging individual codebooks into an Aggregated Code Space (ACS) and evaluating coders against it.

Once the ACS is established, we evaluate each coder (human or AI) using four key metrics:

  • Coverage: The proportion of the total conceptual space captured by the coder.
  • Overlap: The degree of alignment with the group consensus.
  • Novelty: The number of unique, valid concepts introduced by the coder.
  • Divergence: The extent to which a coder’s conceptual focus differs from the group average.

Experimental Design

We validated this method using a dataset of 127 online messages between educational tool designers and teachers. We compared four human researchers against various AI configurations, including "edge case" variants designed to simulate poor coding behaviors: "Flooding" (excessive, redundant codes) and "Hallucinating" (coding based on irrelevant data).

To ensure the framework's robustness, the merging algorithm was tested across multiple Large Language Models (LLMs), including Gemma3-27B, Qwen QwQ 32B, GPT-4o, and Gemini-1.5-Pro.

Key Results and Stability

Our analysis demonstrated that the metrics are remarkably stable. The relative ranking of coders remained consistent regardless of which LLM was used for the merging process, suggesting researchers can utilize open-source models for evaluation without sacrificing reliability. OLS regression analysis confirmed that the combination of the algorithm stage, the model used, and the coder's identity explained over 91% of the variation in metric outcomes ($R^2 > 0.91$).

Comparison of metrics across different coder behaviors, demonstrating the framework's ability to distinguish between baseline, flooding, and hallucinating behaviors.

Crucially, the metrics effectively diagnosed "AI pathologies." As shown in the experimental results, "Hallucinating" AI models showed a dramatic drop in Coverage and Overlap alongside a sharp spike in Divergence. "Flooding" models achieved high Coverage but demonstrated diminishing returns in Novelty, indicating redundancy rather than depth.

Discussion and Limitations

While the framework provides a powerful diagnostic tool, it is not without limitations. The current validation is limited to online conversation data; further research is needed to test its performance on longer texts like interview transcripts. Additionally, we observed that Gemini-1.5-Pro tended to merge codes more aggressively than other models, raising questions about whether such "deep" merging collapses important nuances.

Finally, the ACS is inherently dependent on the team. If a team of humans and AIs shares the same underlying bias, the metrics may fail to detect missing perspectives. Nevertheless, this method provides a systematic way to move beyond subjective impressions toward a rigorous, measurable understanding of qualitative "open codes."

Summary updated

Abstract

Qualitative analysis is critical to understanding human datasets in many social science disciplines. Open coding is an inductive qualitative process that identifies and interprets "open codes" from datasets. Yet, meeting methodological expectations (such as "as exhaustive as possible") can be challenging. While many machine learning (ML)/generative AI (GAI) studies have attempted to support open coding, few have systematically measured or evaluated GAI outcomes, increasing potential bias risks. Building on Grounded Theory and Thematic Analysis theories, we present a computational method to measure and identify potential biases from "open codes" systematically. Instead of operationalizing human expert results as the "ground truth," our method is built upon a team-based approach between human and machine coders. We experiment with two HCI datasets to establish this method's reliability by 1) comparing it with human analysis, and 2) analyzing its output stability. We present evidence-based suggestions and example workflows for ML/GAI to support open coding.
BibTeX
@misc{chen2024computational,
  author = {John Chen and Alexandros Lotsos and Lexie Zhao and Caiyi Wang and Jessica Hullman and Bruce Sherin and Uri Wilensky and Michael S. Horn},
  title = {{A Computational Method for Measuring "Open Codes" in Qualitative Analysis}},
  year = {2024},
  eprint = {2411.12142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.12142},
}

Loading related content...

Figure