Grading Inflection, Not Characters: Honest Scores for Tiny Non-English Clinical Test Sets

Cross-lingual clinical NLP is often judged on test sets of a few dozen sentences, scored with metrics built for English. Two errors pile up in these numbers. The first comes from the language: in inflected languages a model that finds the right medical term but gives it the wrong case ending gets zero credit, as if it had found nothing. The second comes from the sample: with thirty test sentences, a gap of several F1 points between two transfer methods can be pure luck. This note proposes a scoring rule that gives credit for inflectional near-misses but refuses credit for character-level differences that change meaning. We pair it with an uncertainty procedure suited to tiny test sets, and we test both by checking whether they recover system rankings that we can verify on a large held-out pool.

The two problems are linked. A graded score that removes the arbitrary zeros from near-misses should also reduce noise, so the same thirty sentences separate systems more reliably. That only holds if the partial credit reflects correctness rather than surface similarity. Existing fixes measure surface similarity, and in clinical text surface similarity is dangerous: "hypotension" and "hypertension" differ by one character and mean opposite things.

Core Claim

Morphology-sensitive metrics exist. HeQ introduced Token-Level Normalized Levenshtein Similarity (TLNLS), a mix of word overlap and character edit distance, to stop Hebrew answers from scoring zero over an attached preposition. Advocating Character Error Rate for Multilingual ASR Evaluation showed that character error rate tracks human judgments better than word error rate in Malayalam and Arabic. Both weight every character difference equally, which the research file lists as an open limitation.

We claim that a lemma-anchored, inflection-tolerant span score does two things on small non-English clinical test sets. It agrees with human correctness judgments at least as well as character-distance metrics while giving near-zero credit to meaning-changing edits. And it reaches the true system ranking with fewer test sentences than strict span F1. "Lemma-anchored" means the prediction and the gold span must reduce to the same dictionary form (lemma), and may differ only in inflectional features such as case, number, or gender, as encoded in the UniMorph schema used by M2C. Differences in derivational prefixes, compound heads, or negation get no credit.

What is new is the combination: tolerance restricted to inflection, applied to clinical span extraction, and judged by ranking recovery at small sample sizes, not by correlation on large sets.

Research Program

The scoring rule works on aligned predicted and gold spans. Exact matches score one. When they differ, both are passed through a morphological analyzer for the language. If the lemmas match and the only differing features are inflectional, the span scores a tunable credit $c \in (0,1]$ (default one). Any other difference scores zero, or falls back to ordinary token overlap for boundary errors, exactly as relaxed F1 already does. Totals feed into the usual precision, recall, and F1.

The rule is deterministic and has one free parameter, which we fix before looking at test results. This matters because Reproducibility Issues for BERT-based Evaluation Metrics found that preprocessing choices swing embedding metrics by up to 11%, more in Russian and Turkish, and that one published correlation collapsed from 0.48 to 0.06 after a bug fix. A metric with many knobs would recreate the "vibration effects" that Cross-validation failure identifies as a route to accidental p-hacking.

For uncertainty, we report paired bootstrap confidence intervals over test sentences, not over mentions. Mentions within one sentence are correlated, and resampling them independently would repeat the underestimation that Cross-validation failure documents for fold-wise standard errors (a factor of 2 to 4). That paper also reports roughly ±15% accuracy intervals at $N=30$. This is the scale of noise we expect on GERNERMED.

The key methodological move is a calibration testbed where the truth is known. HeQ has about 30,000 question-answer pairs in a morphologically rich language. We can draw many subsamples of 30, 100, and 300 items, score competing systems with each metric, and ask how often the subsample ranking matches the full-set ranking judged by humans. This turns "which metric is better on tiny sets" into a measurable ranking-recovery rate. Clinical test sets cannot provide this, because they have no large pool behind them.

Baselines

The naive reference is strict exact-span F1 with point estimates and no intervals. This is how Multilingual Clinical NER: Translation or Cross-lingual Transfer? compares cross-lingual transfer, translate-train, and translate-test on MedNERF and the 30-sentence GERNERMED set. The file gives no reported F1 values for these sets (no reported result found on Lacuna for GERNERMED or MedNERF F1). We will therefore rerun that paper's three pipelines, using the same XLM-R and distilmBERT models and the same English n2c2 training data, to produce the system outputs that every metric scores.

The strongest existing competitor is TLNLS from HeQ. It is the only published metric designed for exactly the near-miss problem. On HeQ it reports mBERT at 71.42 F1 versus 78.2 TLNLS, a 6.8-point gap attributed to recovered near-misses. We alongside run relaxed token-overlap F1 and a character-error-style span distance, following Advocating Character Error Rate. All metrics score identical system outputs, so there is no training or tuning budget to equalize. Our only parameter, $c$, is fixed on a HeQ development slice that is disjoint from all evaluation subsamples.

Beating TLNLS means two things. First, equal or higher agreement with human span-correctness labels. Second, a higher ranking-recovery rate at $N=30$ and $N=100$, with non-overlapping bootstrap intervals on that rate. As a hard requirement, we also need a false-credit rate below TLNLS's on a check set of meaning-changing minimal pairs.

The ablation that isolates the key ingredient replaces lemma anchoring with plain suffix stripping. This asks whether the analyzer matters or whether crude normalization captures the gain. A second ablation keeps our metric but uses mention-level bootstrap, to show how much the sentence-level unit matters for interval width.

Why not just…?

Why not just keep strict F1 and add bootstrap confidence intervals? This is the most serious threat, and it may be sufficient for French and German. Intervals fix the noise problem but not the bias: if one transfer method systematically produces wrong case endings, strict F1 penalizes it for a harmless error, and tighter intervals only make that wrong conclusion look more certain. M2C found up to 50% of errors in inflected languages are pure morphological near-misses, but that was on synthetic templates. So our first experiment measures the near-miss rate in real clinical outputs. If it is small in MedNERF and GERNERMED, we will report that intervals alone suffice there, and restrict the metric's claim to languages where near-misses are frequent.

Why not just use TLNLS or character error rate, which already give partial credit? Character distance rewards the wrong edits in medicine. Swapping a negating or directional prefix changes the diagnosis but costs one or two characters. Our minimal-pair check set, built from prefix, compound-head, and antonym alternations in the clinical vocabularies, measures this directly. If TLNLS's false-credit rate is low in practice, its simplicity wins and we will say so.

Why not translate everything to English and score with English metrics? Multilingual Clinical NER already found translate-test weaker than direct transfer for large models. Projecting spans back through word alignment adds boundary errors, which would contaminate the very scores we want to trust. We include translate-test as one of the scored systems, not as the evaluation route.

Why not use a character n-gram or embedding similarity between spans? Medical Crossing showed that character n-gram tf-idf is a strong linker, and that SapBERT reaches 32.61% Acc@1 on filtered CodiEsp versus 6.49% for general BERT. That is a result about retrieving concepts, not about grading spans. Embedding scores also carry the preprocessing fragility documented in Reproducibility Issues. We include a SapBERT cosine-credit variant as an extra scored metric. If it matches our metric on human agreement and false-credit rate, that result is reportable, though it gives up determinism.

Proposed Experiments

Human judgments on HeQ. Annotators label a sample of mismatched predictions from several HeQ systems into three classes: correct, inflectional near-miss, or wrong. We report each metric's agreement with these labels.

Ranking recovery on HeQ. We draw 1,000 subsamples at each size from 30 to 1,000. At each size we compute the fraction of subsamples in which each metric reproduces the full-set human-judged pairwise ranking of systems. We also report the smallest $N$ at which a metric separates systems with 80% power.

Clinical re-scoring. We rerun the three transfer pipelines from Multilingual Clinical NER on MedNERF and GERNERMED. We also score entity mention spans on CodiEsp and Cantemist under the Filtered protocol of Medical Crossing, which shrinks test sets and so provides natural small-sample cases. For each, we report near-miss rates, every metric with sentence-level intervals, and whether any published pairwise conclusion survives.

Decisive checks. Three results would support the proposal. First, on HeQ at $N=30$, our metric recovers the true ranking more often than both strict F1 and TLNLS. Second, it assigns near-zero credit on the meaning-changing minimal pairs. Third, the suffix-stripping ablation falls short of it. If strict F1 with intervals matches it at every $N$, the metric contribution fails, and the note's conclusion becomes the purely statistical one: current cross-lingual clinical comparisons on 30-sentence sets are not distinguishable from noise.