Researchers now use large language models (LLMs) to help with thematic analysis: reading interview transcripts, tagging passages with short labels ("codes"), and grouping codes into broader patterns ("themes"). These tools have been validated almost entirely on English. In agglutinative languages such as Turkish or isiZulu, much of a sentence's meaning sits in chains of suffixes and prefixes attached to one root word. That meaning includes negation, "cannot," causation, and whether the speaker saw something or only heard about it. As a result, two teachers describing the same experience often use different surface words, while two passages that look almost identical can mean opposite things. We propose that automated coding pipelines fail in these languages in two predictable ways. They split one idea into many codes because the word forms differ. They also merge opposite stances because the forms look alike. We further propose that a simple fix repairs both failures. Each code is represented as a root word plus the few grammatical features that carry stance, and codes are merged on the root but kept apart on those features.
The application we care about is curriculum localization. Teachers adapt imported or AI-generated lesson materials to local language and culture. Qualitative work such as the Malaysian teacher study in Teaching English in the Age of AI shows that this adaptation is where AI materials break down. That study also shows it is studied through small, hand-coded interview samples, in that case six teachers. Scaling such studies to Turkish- and Zulu-speaking teachers requires automated coding that respects the language's structure. We would build that coding step, test it against human coders, and publish a translated benchmark that makes the failure measurable.
Core Claim
Our claim is specific. In an LLM-assisted thematic analysis pipeline running on Turkish or isiZulu text, the code-consolidation step does most of the damage. That step decides which candidate codes are "the same." Two cheap fixes each repair only one half of the problem. Lemmatizing (stripping words to their dictionary root) fixes over-splitting, but it erases stance carried by affixes, for example Turkish öğretebiliyorum "I can teach" versus öğretemiyorum "I cannot teach," or reported-past -mIş versus witnessed-past -DI. Translating to English first also fixes over-splitting, but it can flatten evidential and modal distinctions, and it sends sensitive transcripts through an extra system. A representation built on the Universal Dependencies scheme, which stores a root lemma together with a bundle of grammatical features, supports a middle path. The method, following the "content word primacy" conversion in Universal Dependencies for Turkish, is to merge candidate codes when their lemmas and meanings agree and to block the merge when a small, pre-declared set of stance features (polarity, ability/possibility mood, evidentiality, causative voice) disagrees.
What is new is not morphological analysis and not LLM-assisted coding, both of which exist. What is new is showing where in the qualitative pipeline morphology matters, and that the right intervention is selective feature retention at the merge step rather than normalizing the text before the pipeline starts.
Research Program
The pipeline reuses the four-stage human–LLM loop of LLM-in-the-loop, which consists of human exemplars, LLM initial coding, collaborative refinement, and double-coded evaluation with Cohen's κ. We change the refinement stage. When the LLM proposes merging codes, each code's head predicate and key arguments are passed through a morphological tagger. For Turkish, the tagger is trained on the Turkish Web Treebank from A Gold Standard Dependency Treebank for Turkish, which reports 94.74% morphological-feature accuracy. For isiZulu, it is the from-scratch neural tagger of Neural Morphological Tagging for Nguni Languages. A proposed merge that crosses a stance-feature boundary is rejected or flagged for the human coder. A merge the LLM missed is proposed when two codes share lemma, stance features, and semantic similarity above a threshold tuned on held-out data.
All models run locally on open weights. LLM-in-the-loop names privacy as the barrier to uploading real transcripts to commercial endpoints, and teacher interviews about their own classroom struggles fall under that concern.
We borrow one measurement idea from studies of preferential flow in rock, where injected fluid races through high-permeability channels and leaves much of the formation unswept (e.g. Three-dimensional physical simulation and optimization of water injection of a multi-well fractured-vuggy unit). The analogue we measure is "sweep": the fraction of source passages that end up supporting some final theme. We report sweep stratified by morphemes per word. Our hypothesis is that surface-level pipelines leave heavily inflected passages "unswept," because frequent, simple forms dominate the clusters. This is a diagnostic borrowed for its bookkeeping, not a claim that flow physics governs language models. We do not use pulsed schedules or other flow-inspired training tricks. A reviewer would rightly ask why frequency reweighting would not do the same job, and we have no evidence that it would not.
We need two data sources. The first is a controlled benchmark. We will have professional translators render the Music Shuffle (35 responses) and Password Manager (100-response sample) datasets from LLM-in-the-loop into Turkish and isiZulu. Their English gold codebooks then become a cross-lingual target. The second is new interview data from 15–20 Turkish-speaking and 15–20 isiZulu-speaking primary teachers about adapting AI-generated lesson materials. These interviews mirror the Malaysian study's protocol and are independently double-coded by native-speaker qualitative researchers. The interview topic matters for the method. Position: LLMs Can be Good Tutors in English Education and Dahal's argument against literal translation predict that teachers' accounts will turn on what they could not make work and what they heard from colleagues versus saw themselves. Those are exactly the distinctions carried by affixes.
Baselines
The naive reference point is single-prompt zero-shot coding in the source language: one instruction to the local LLM to produce codes and themes, with no exemplars or refinement. It shows how bad the problem is without any help.
The strongest existing method is the full LLM-in-the-loop procedure run directly on source-language text with the same local LLM, the same exemplars, and the same human refinement time. On English it reports human–LLM κ of 0.87 on Music Shuffle and 0.81 on Password Manager, against a human–human κ of 0.77 on Password Manager (LLM-in-the-loop). We found no reported result on Lacuna for any thematic-analysis pipeline on Turkish or isiZulu text. We also run a persona-conditioned multi-agent pipeline in the style of Auto-TA. That paper reports a +16.28 credibility gain from persona conditioning on English clinical transcripts, but no agreement figure comparable to κ. It is therefore a secondary competitor for full automation.
The fair-comparison rules are as follows. Every condition uses the same open LLM, the same exemplars translated by the same translators, an identical cap on human refinement minutes, and thresholds tuned on the same held-out split. Beating the strongest baseline means two things. The first is higher κ against the English gold codebook on the translated benchmark, with a paired bootstrap interval excluding zero. The second is a significant reduction in stance-merge errors on the minimal-pair probes described below, achieved without lowering sweep.
The ablations isolate the key ingredient. We compare surface forms only, lemma only (full lemmatization), lemma plus all tagger features, and lemma plus stance features only. We also run the stance-feature condition with gold versus predicted morphology. This matters most for isiZulu, where the Nguni paper reports macro F1 in the high 70s with gold canonical segmentation but only above 55% with predicted segmentation. We expect tagger error to be the main limit on the method there.
Why not just…?
Why not just translate everything into English and run the English pipeline? This is the most serious threat. If English κ carries over after translation, the morphology layer is unnecessary. Translation also moves a sensitive transcript through a second model, and English has no grammatical evidential. A Turkish speaker's "it was reportedly confusing" and "I saw it was confusing" can both come out as "it was confusing." We test this directly with minimal-pair probes that differ only in evidential or abilitative marking. If translate-then-code separates those pairs as well as our method and matches its κ, we will report that the simpler route wins.
Why not just lemmatize before coding? Lemmatization is the obvious repair for over-splitting, and we expect it to recover much of the κ gap on topical themes. Its predicted failure is merging negated with affirmed and able with unable. The lemma-only ablation measures this, and the error count on the stance probes is the deciding evidence.
Why not just trust a strong multilingual LLM to handle morphology internally? It may do so during code generation. Our hypothesis concerns the merge step, where short codes are compared. There is also a caution: the Nguni tagging study found fine-tuned multilingual pretrained models trailing from-scratch morpheme-level models by more than ten macro-F1 points. That result suggests pretrained models do not reliably encode affix-level distinctions in these languages. The surface-only condition with a strong model shows whether this concern holds for coding.
Proposed Experiments
The first check asks whether the problem exists. On the translated benchmark, we count near-duplicate codes that differ only morphologically under surface-level LLM-in-the-loop and compare the count with the English original. If fragmentation and κ loss are negligible, the project stops there with a useful negative result. The second check is the stance probe set: about 200 passages built by native speakers from the interview data, varying single affixes. We score each pipeline by how often it places a passage and its opposite-stance twin under the same code. The third check reports κ against gold codes and sweep stratified by morpheme count on both the translated benchmark and the new teacher interviews. The fourth check has native-speaker coders, blind to condition, rate whether each final theme faithfully reflects its supporting passages. This follows the reflexive tradition's insistence that themes answer to participants' accounts rather than to agreement statistics alone.
The proposal succeeds if stance-feature merging beats both lemmatization and translate-then-code on the probes while matching or exceeding them on κ. It fails, and we will say so, if either simpler route matches it on both measures.
References
- Teaching English in the Age of AI: A Qualitative Study of Primary English Teachers’ Perceptions
- Universal Dependencies for Turkish
- LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
- A Gold Standard Dependency Treebank for Turkish
- Neural Morphological Tagging for Nguni Languages
- Three-dimensional physical simulation and optimization of water injection of a multi-well fractured-vuggy unit
- Position: LLMs Can be Good Tutors in English Education
- Revolutionizing Education through AI-Powered Inclusive Learning Systems
- Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning