Syntax Versus Words as Two Opinions: Choosing Which Dementia Transcripts to Label

Screening for Alzheimer's disease from speech runs on very little labeled data. A label is a clinical diagnosis, not a quick annotation, so benchmark cohorts hold a few hundred speakers at most. The obvious way to stretch a labeling budget is active learning: let the model choose which recordings a clinician should assess next. Its usual selection signals, such as model uncertainty, come from a model trained on almost nothing and are therefore unreliable.

We propose a different signal. Train two cheap classifiers on the same transcript. One sees only its grammatical skeleton, meaning the sequence of part-of-speech tags (noun, verb, preposition...) with every word removed. The other sees only the words. A clinician is asked to label a transcript exactly when these two opinions disagree. Because the syntax-only model is blind to vocabulary and topic, its mistakes should rarely coincide with the word model's mistakes. Their disagreements should therefore land on the transcripts that actually teach something.

The core bet is empirical and falsifiable. It is not enough for two views to be different in some general sense. They must be jointly wrong rarely. We claim that delexicalized syntax is an unusually good partner for lexical models on this measure, better than a second lexical model. We claim this property is what makes disagreement-based querying pay off in the small-sample regime.

Core Claim

Recent theory gives a precise reason to expect this to work. Fast Co-Training under Weak Dependence via Stream-Based Active Learning reduces co-training to online learning. Two learners, each seeing one view of an unlabeled stream, request a label only when they disagree, so every query corrects at least one of them. The label count is $\tilde{O}(M/\alpha)$, where $M$ is the learners' mistake bound and $\alpha$ the minority-class fraction. It holds under a "weak dependence" condition that tolerates views being correlated some of the time. That paper is purely theoretical, on synthetic concept classes. Nobody has asked which real text views satisfy weak dependence well enough for the guarantee to matter.

We make the dependence measurable. For view classifiers $h_S$ (syntax) and $h_L$ (lexical), label $y$, and transcript $x$, define the joint-error excess

$$\Delta = P\big(h_S(x)\neq y,\ h_L(x)\neq y\big) - P\big(h_S(x)\neq y\big)\,P\big(h_L(x)\neq y\big).$$

$\Delta$ is zero when the two views' errors are independent and grows as they fail on the same cases. Joint errors are the cases disagreement querying can never see, because the two views agree and are both wrong. So a lower $\Delta$ should mean fewer wasted or missed queries.

Our claim has two parts. First, the part-of-speech view paired with a lexical view has markedly lower $\Delta$ than two lexical views, such as a pretrained-model embedding and bag-of-words. Second, across view pairs, $\Delta$ predicts how many clinical labels disagreement querying needs to reach a target accuracy. If the second part fails, the theory's weak-dependence condition is not the operative quantity on real clinical text. That would be a useful negative result in itself.

Why These Two Views

Three lines of evidence motivate the syntax view.

The first is work on stripping words to their tags so that style and structure can be measured apart from topic. Tomokiyo et al. did this to find non-native grammar without learning conversation topics. Tran et al. separated community style from content. Schwarm et al. measured reading level without domain vocabulary. That topic-blindness is exactly what should decorrelate errors from a word model.

The second is clinical: the syntax view is diagnostic on its own. Solorio & Liu classified language impairment in bilingual children by comparing perplexities of part-of-speech trigram models. Revealing the Roles of Part-of-Speech Taggers in Alzheimer Disease Detection reports 92.2% accuracy on DementiaBank using only frequencies of 27 tags. We treat that number cautiously, since the file does not tell us whether its evaluation was speaker-disjoint.

The third is that text views dominate audio here, which is why we pair two text views rather than text with audio. The ADReSS Challenge linguistic baseline reached 0.75 test accuracy against 0.625 for acoustic. The INESC-ID system found acoustic models trained from scratch near chance at 54.17%.

Other work constrains the design. A Corpus-Based Approach for the Prediction of Language Impairment found that automatic taggers make errors on spontaneous, disfluent speech. That noise could either decorrelate the views or simply cripple the syntax view, and we must measure which. Efficient Generalization via Multimodal Co-Training under Data Scarcity and Distribution Shift warns that co-training degrades when the unlabeled pool shifts. This matters because our pool (Pitt transcripts) and our target (ADReSS) differ in curation.

Two prior multi-view designs differ from ours. Syntactic Multi-view Learning for Open Information Extraction aligns two syntactic views (constituency and dependency trees) under full supervision. It uses neither disagreement to choose labels nor a syntax-versus-words split. INESC-ID concatenates tag embeddings into one network, which erases the separate opinion we need.

The Co-Training and Expansion analysis supplies the rationale for using agreement on unlabeled data. The self-training survey documents the confirmation-bias failure of single-view pseudo-labeling that co-training is meant to avoid.

Research Program

We deliberately keep both learners close to the theory: linear, online-updatable classifiers. With them the mistake-bound reduction of Zhang & Klivans applies approximately, since margin halfspaces are one of their analyzed classes.

The syntax learner sees Universal part-of-speech tag sequences from an off-the-shelf tagger. From these it builds tag n-gram counts plus a log perplexity ratio from class-conditional tag trigram models, in the style of Solorio & Liu. A logistic or perceptron classifier sits on top.

The lexical learner is a linear classifier on frozen embeddings from a pretrained language model. A second variant uses word bag-of-words.

The loop runs over an unlabeled pool in rounds. Each round, both learners score the pool. Transcripts where they disagree, ranked by the product of the two margins, are sent to the simulated clinician until the round's budget is spent, and both learners are updated. Confident agreements may be added as pseudo-labels, and we ablate that step. A cold start uses a small random seed set of labels, identical across all methods.

Alongside accuracy, we estimate $\Delta$ for every view pair on held-out labeled data at several budget levels. This gives the plot that tests the core claim: $\Delta$ against labels-to-target, across view pairs.

Baselines

The naive reference is random labeling. It uses the same seed set, the same per-round budget, and the same two learners, with each view's classifier and the concatenated-feature classifier all trained on randomly chosen labels. Disagreement querying must beat this at every budget, or it has no reason to exist. Active learning on tiny pools often fails to do so because of the cold-start problem.

The strongest competitor on accuracy is a single fine-tuned pretrained language model with prompt-based fine-tuning, as in Domain Adaptation via Prompt Learning for Alzheimer's Detection. It is paired with standard uncertainty sampling, the approach Deep Active Learning for Named Entity Recognition used to reach 99% of peak performance with 25% of the data. That paper reports a mean F1 of 0.8952 for adaptation from the Carolinas Conversation Collection to ADReSS. For fully supervised ADReSS, the reported points are 81.25% test accuracy for INESC-ID's multimodal system and 0.75 for the challenge linguistic baseline. We found no reported result on Lacuna for any active learning method on ADReSS or DementiaBank, so the learning curves themselves are new.

We run this baseline on the identical pool, seed set, and label budget. Its prompt and learning-rate search gets a tuning budget equal in GPU-hours to all tuning of our method combined. We count "beating it" as either of two outcomes. One is reaching its full-budget ADReSS accuracy with at most half the labels. The other is matching it at equal labels while being a linear model, which counts as a practical win but not a scientific one. The PLM-plus-uncertainty baseline also gets our disagreement rule, applied between it and the syntax learner, as an upgraded variant.

The ablation that isolates the key ingredient is disagreement querying with two lexical views: the PLM embedding classifier against the bag-of-words classifier. This is query-by-committee without syntax. A second ablation uses the lexical view against a feature-bagged copy of itself.

If POS-versus-lexical does not beat lexical-versus-lexical at equal budget, the syntax view is not what matters. We would then drop the claim.

A final ablation removes the agreement-based pseudo-labels to separate the active-learning effect from the co-training effect.

Why not just…?

Why not just fine-tune a pretrained model and use uncertainty sampling? This is the most serious threat, and we run it as the main baseline rather than argue it away. Two things count against it on paper. With a few dozen labels, uncertainty estimates come from a nearly untrained model. Prompt-tuning without domain prompts fell to 58–59% F1 under topic shift in Farzana & Parde, which suggests these models latch onto vocabulary. Neither point proves it loses on in-domain ADReSS; the learning curves in our main experiment settle this. If the fine-tuned model wins in-domain but loses under the conversational-transfer test, we would report that split honestly.

Why not just use a committee of two lexical models? Disagreement between any two decent models might carry most of the benefit, making syntax incidental. This is exactly the lexical–lexical ablation, and the $\Delta$ measurements explain the outcome either way. If two lexical views show $\Delta$ as low as the POS–lexical pair, our story is wrong.

Why not just concatenate tag features into one model and sample by uncertainty? Concatenation gives one opinion, and nothing can disagree with it. INESC-ID's fused system found linguistic features dominating, consistent with one view swamping the other. We run the concatenated classifier under uncertainty sampling at the same budget. If it matches our method, the separate-view machinery adds nothing.

Why not just self-train on the lexical view? Single-view pseudo-labeling reinforces its own confident errors, as the self-training survey describes. It is also cheap enough that we include it at the same budget. Removing pseudo-labels from our method shows whether any gain comes from querying rather than from unlabeled data.

Proposed Experiments

The unlabeled pool is the DementiaBank Pitt transcripts, with labels hidden and revealed only when queried. Every ADReSS test speaker is removed from the pool, because ADReSS is derived from Pitt and overlap would leak labels.

We evaluate in two ways. The first is on the ADReSS test split. The second is speaker-disjoint cross-validation on Pitt, since the ADReSS test set is only a few dozen speakers and single-split differences of a few points are noise.

Budgets range from a 10-label seed up to 200 labels. We report accuracy and F1 against labels, area under the learning curve, and labels needed to reach 0.75 and 81.25% ADReSS accuracy, each over at least 20 seeds with bootstrap intervals.

Three stress tests follow. First, we subsample the AD class in the pool to 10% and 25% to probe the $1/\alpha$ dependence that Zhang & Klivans leave open. Disagreement querying may starve the minority class, and we compare its query class-balance against uncertainty sampling. Second, we swap manual transcripts for automatic speech recognition output, to measure how tagger and recognizer noise changes the syntax view's accuracy and $\Delta$. Third, we test on Carolinas Conversation Collection transcripts, to see whether topic-blind syntax improves robustness to a change in conversation style.

The decisive checks are three. POS–lexical disagreement must beat both random labeling and the lexical–lexical committee at equal budgets, with non-overlapping intervals at 50 labels or fewer. Across all view pairs, measured $\Delta$ must rank-order labels-to-target. The method's advantage over the fine-tuned-model baseline must either persist under the imbalance and conversational-shift tests, or be shown not to.

Any single failure is informative. The second check, in particular, is the first real-data test of whether weak dependence is the quantity that governs disagreement-based labeling.