Language models are increasingly taught a culture's unwritten rules from conversations that another, larger model writes. One recent method has a large teacher model role-play social scenes set in China, Germany, Mexico, Japan and the UK. The teacher also labels the cultural intent behind each turn, for example "showing respect to an elder." A small student model then trains on these dialogues. The method works, but it rests on a single teacher whose picture of each culture is never checked. If that teacher holds a Western-tinted caricature of a Suzhou teahouse, the student learns the caricature at scale.
Qualitative social science has a standard defense against exactly this failure. Several independent coders label the same material, their agreement is measured, and interpretations are returned to the people being described for confirmation ("member checking"). This note proposes carrying that defense into synthetic cultural training. Each synthetic scene is labeled by several teacher models that come from different training cultures. Their agreement on the cultural intent tells us which examples to trust. A small panel of native-speaker raters checks whether that agreement actually tracks authenticity. The bet is that agreement across independent models from different cultures filters out caricatures that a single model, however often it is resampled, would keep repeating.
Core Claim
Agreement among teacher models from different cultural origins is a usable signal of which synthetic cultural examples are faithful. It works because their errors are less correlated than those of one model sampled many times. Training a student only on high-agreement examples, or weighting examples by agreement, should improve fit to real human value distributions more than three alternatives: training on everything one teacher produces, filtering by one teacher's self-consistency, or simply switching to a stronger single teacher.
The method being extended is Cultural Learning-Based Culture Adaptation of Language Models, which we call CLCA. In CLCA, GPT-4 adapts generic templates into local scenes, Llama 3.1 70B simulates the dialogues and labels their intents, and Llama 3.1 8B and Qwen 2.5 7B students are fine-tuned with LoRA, a lightweight fine-tuning method that trains a small number of added weights. The students are evaluated against 44 questions from Wave 7 of the World Values Survey. CLCA does not check its teacher against any second source or against human judgment. That missing check is the gap this proposal fills.
The motivation for the check comes from human qualitative research. Studies such as Ongwere et al. report inter-rater reliability of 0.9 across coding rounds. Smart Spaces, Private Lives closes its grounded-theory loop by having participants confirm its cultural privacy model. In both, triangulation and member checking are what make cultural interpretation credible. The engineering precedent is Menon & Serban, who found that single-pass LLM extraction from company filings was unreliable until they cross-checked it against independent databases and web retrieval. We apply the same logic to cultural labels instead of facts.
Research Program
The first step reproduces CLCA's generation pipeline for its five cultures. For every synthetic dialogue, we collect cultural intent labels from a panel of teachers drawn from different training origins. The panel includes Llama 3.1 70B, which is CLCA's original teacher, a large Qwen model, and at least one further model with a different regional training mix. Each teacher labels intents in a shared open-then-consolidated format, as a codebook would in human coding: free-text intents are mapped onto a fixed intent inventory by a separate, frozen mapping step. We compute agreement for each example $i$ as
$$a_i = \frac{1}{\binom{K}{2}} \sum_{j<k} \mathbb{1}[\ell_{ij} = \ell_{ik}],$$
where $K$ is the number of teachers, $\ell_{ij}$ is teacher $j$'s consolidated intent label for example $i$, and $\mathbb{1}$ is 1 when the two labels match. We also report Krippendorff's alpha, a chance-corrected agreement statistic, over each culture's corpus. This makes the overall reliability comparable to human coding studies.
The second step is the member check. For each culture, native-speaker raters judge whether a stratified sample of dialogues, drawn across agreement levels, is authentic or a caricature. Three raters per culture is a natural minimum, mirroring the three-researcher codebook of Phelps et al.. The raters' own agreement is reported. This step tests the central assumption directly. If teacher agreement does not predict human authenticity ratings, the method has no basis, and we will say so.
The third step trains students under several data conditions with identical example counts. The conditions are: CLCA's full single-teacher corpus; a high-agreement filtered corpus; an agreement-weighted corpus; and a soft-label variant that trains on the distribution of teacher intents rather than a single label. The soft-label arm takes seriously the lesson of LIVS. LIVS collapsed multi-stakeholder judgments into majority-vote binary preferences and thereby discarded about half its neutral evaluations. Disagreement may itself carry cultural information rather than noise. The experiment decides whether filtering or preserving disagreement works better.
Baselines
The naive reference points are the untuned student and the majority baseline from WorldValuesBench. The majority baseline always predicts the most frequent human answer, and small models often fail to beat it.
The strongest existing method is CLCA itself, trained with intent understanding. For Llama 3.1 8B, CLCA reports individual-level survey accuracy rising from 31.6% to 39.7% and KL divergence dropping by 0.0899. KL divergence measures how far the model's answer distribution is from the human one. We rerun CLCA ourselves with the same scenes, student models, LoRA settings, number of training examples and tuning budget. Our numbers are therefore compared against our reproduction rather than only against the published figures. A filtered or weighted condition beats CLCA if it improves individual accuracy and lowers KL on held-out questions across at least four of the five cultures, with seed variance reported.
The ablation that isolates the key ingredient is single-teacher self-consistency. Here Llama 3.1 70B labels each dialogue $K$ times at nonzero temperature, and agreement is computed exactly as above. The labeling call count is matched to the multi-teacher panel. A second ablation uses a panel of teachers that are different models but share a similar training origin. This separates the effect of model diversity from cultural-origin diversity.
The last baseline is a single stronger teacher alone, the best model in the panel. It tests whether the gains come from triangulation or simply from better labels.
Why not just…?
Why not just use a stronger single teacher? This is the most serious threat. If the strongest panel member alone matches the triangulated corpus, then triangulation is an expensive way to pick the best labeler. The "best single teacher" baseline is run under the same budget to test exactly this. The member check adds a second test: we measure whether the strongest teacher's disagreements with the others fall disproportionately on examples that native raters call caricatures. If they do not, we drop the multi-teacher claim. CDEval offers indirect evidence that strength alone may not suffice. In CDEval, capable models, including ones built outside the West, still sit close to the US cultural profile, and GPT-3.5 shows 0.78 similarity to it. This suggests that bias shared across a model family survives scaling.
Why not just resample one teacher and keep the consistent outputs? Self-consistency catches random slips but not systematic caricature, because a model that believes a stereotype repeats it reliably. That is the theoretical case, but it is an empirical question. The matched self-consistency ablation answers it, and we will report the result even if the cheap version ties.
Why not just prompt with demographic personas, or fine-tune directly on survey answers? WorldValuesBench found that adding demographic context made smaller open models worse. CQ-Bench found that models lean on broad demographic generalizations when reading implicit cues. Fine-tuning directly on survey answers would contaminate the very benchmark used for evaluation. CLCA also reports that factual cultural data from Wikipedia gives negligible gains, which argues against plain retrieval of facts. We still include a persona-prompting row so the gap is measured rather than assumed.
Proposed Experiments
The primary evaluation follows CLCA's World Values Survey protocol: individual-level accuracy and KL divergence per culture. To avoid tuning to the test set, the agreement threshold and weighting are chosen on a development split of the 44 questions and reported on the held-out remainder.
Transfer is tested on WVB-PROBE from WorldValuesBench. Its metric is the fraction of questions whose Wasserstein-1 distance to the human distribution is below 0.2. Reported reference points are 75.0% for GPT-3.5 Turbo and 25.0% for Vicuna-7B. Transfer is also tested on CQ-Bench attitude detection and value selection, where o1 reports F1 scores of 0.622 and 0.809 and humans score 0.689 on attitude detection. CQ-Bench matters because CLCA's gains might reflect fitting survey answers rather than reading implicit cultural cues. The open-ended value extraction task has no reported number on Lacuna, so we report our own results there without a reference figure.
Open-ended generation is scored with the LLM-as-a-jury approach of LLM-GLOBE, because that work showed closed Likert scales are distorted by how each model uses the scale. The jury includes models outside our teacher panel to avoid circular scoring.
The decisive checks are three, and each can falsify the proposal. First, teacher agreement must correlate positively with native-rater authenticity within each culture. Second, cross-origin agreement must outperform matched single-teacher self-consistency as a filter. Third, the gains must survive on CQ-Bench, not only on the World Values Survey questions CLCA was built around. If the first check fails, the proposal reduces to a negative result about synthetic cultural data. That result would still be worth publishing, because it would show that current synthetic cultural training cannot be validated by model consensus.
References
- Cultural Learning-Based Culture Adaptation of Language Models
- Artificial Intelligence for Medication Management in Discordant Chronic Comorbidities: An Analysis from Healthcare Provider and Patient Perspectives
- Smart Spaces, Private Lives: A Culturally Grounded Examination of Privacy Tensions in Smart Homes
- An Automated LLM-based Pipeline for Asset-Level Database Creation to Assess Deforestation Impact
- Patient perspectives about deployment of artificial intelligence decision support tools in a safety-net healthcare system
- LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spaces
- WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models
- CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
- Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Metacognitive Cultural Intelligence with CQ-Bench
- LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output