Spending Expensive Model Calls Where They Move the Left-Right Score

Political scientists measure where parties stand by hand-coding every sentence of a manifesto into one of 57 policy categories (the MARPOR scheme). They then compute a document-level left-right score called RILE, roughly the share of right-coded sentences minus the share of left-coded ones. Hand coding cannot keep up with decades of parliamentary transcripts in dozens of languages. Automated coding brings a cost dilemma instead. Strategies for political-statement segmentation and labelling in unstructured text shows that a fast sequence tagger, a conditional random field (CRF: a model that labels each token while enforcing consistent label transitions) on top of XLM-RoBERTa, runs at over 3,000 tokens per second. A fine-tuned Flan-T5 XL is more accurate but runs at 26 tokens per second, and Llama-3.1 8B runs at under 3. The same paper reports something more consequential: sentence-level accuracy does not determine how faithful the resulting RILE score is. What matters is which errors are made and in which direction.

We propose a two-stage cascade. The fast model labels everything, and a small set of statements is passed to the slow model, or to a human coder. The routing rule is driven by how much each statement could move its document's left-right score, not by how unsure the fast model is about the fine-grained label. Two observations make this worthwhile. First, most confusions among the 57 categories are harmless to RILE, because many categories collapse into the same left, right, or neutral bin. Second, a single mislabeled sentence matters far more in a 200-sentence manifesto than in a 5,000-sentence one. A cascade that ignores both observations spends its expensive budget on errors that do not change the measurement social scientists actually use.

Core Claim

For a fixed budget of expensive labels, routing by expected contribution to document-level RILE error recovers most of the gap between the fast CRF and a full Flan-T5 (or gold human) run. It does so with a small fraction of the slow model's tokens, and it beats routing by ordinary instance-level uncertainty at equal budget.

To make this precise, let statement $i$ in document $d$ have a coarse RILE value $s_i \in \{-1, 0, +1\}$ (left, neutral, right). Let $N_d$ be the number of statements in $d$, so that $\text{RILE}_d = \frac{1}{N_d}\sum_{i \in d} s_i$. Write $p_i$ for the fast model's predictive distribution over $s_i$. This distribution is obtained by summing its 57-way probabilities within each bin and averaging over a cheap ensemble of checkpoints. Each statement then gets a routing score

$$ r_i = \frac{\mathrm{Var}_{p_i}(s_i)}{N_d^{2}\, c_i}, $$

where $c_i$ is the token cost of sending statement $i$ to the expensive labeler. The numerator is statement $i$'s contribution to the variance of $\text{RILE}_d$, assuming errors are independent across statements. Dividing by cost turns the score into a greedy knapsack ratio. The cascade sends the highest-scoring statements until the budget runs out. It also spends a small random fraction of the budget on audit labels. Those audit labels feed a difference estimator that corrects the remaining fast-model predictions for systematic bias:

$$ \widehat{\text{RILE}}_d = \frac{1}{N_d}\Big(\sum_{i \in d} \hat s_i\Big) + \frac{1}{N_d}\sum_{i \in d}\, \widehat{\mathbb{E}}[\,s_i - \hat s_i \mid \text{bin}(\hat s_i), \text{genre}\,]. $$

Here $\hat s_i$ is the routed label where available and the fast-model label otherwise. The correction term is estimated from the audit sample. Routing targets variance, and the audit targets bias. Nikolaev and Papay identify systematic error as the dominant threat, and the audit is there to address it.

Research Program

The routing objective decides where the two research directions meet. Ensembling here is a cheap uncertainty source and a cascade, not a way to raise accuracy by averaging.

The fast stage is the CRF from Nikolaev and Papay. We add a checkpoint ensemble of five snapshots from a single training run, the kind of trajectory ensembling One Model to Pronounce Them All used to smooth epoch-specific noise. This adds uncertainty information without retraining, and the extra CRF heads are cheap compared with one Flan-T5 call.

The slow stage is the fine-tuned Flan-T5 XL from the same paper. Because MARPOR has gold codes, we also run a simulated-human variant in which the "expensive labeler" is the gold annotation. This is the setting that bears directly on manual infeasibility. It asks how many sentences a human coder must read to bring an automatically coded archive within a target RILE error.

We study three properties of routing. The first is coarse-bin aggregation, which ignores confusions within a bin. The second is the $1/N_d^2$ leverage weight. The third is the audit correction. Each can be switched off independently. We also test how boundary uncertainty enters the score: a segmentation error can split one right-coded statement into two, which changes both numerator and denominator. We handle this by placing CRF boundary marginals into $N_d$ through sampling.

Finally, we test the scheme on Hansard, where the genre shift from edited manifestos to debate is severe. Nikolaev and Papay report procedural speech being mis-coded into substantive categories there. Under this shift, uncertainty computed inside the CRF's own predictions may be confidently wrong, which is the epistemic failure Hybrid Uncertainty Quantification for Selective Text Classification in Ambiguous Tasks warns about under extreme covariate shift. We therefore add a variant whose numerator mixes in a feature-space density term in the style of HUQ.

Relation to Prior Work

Three papers motivate the proposal. Nikolaev and Papay compare CRF, Flan-T5 and Llama in isolation and find the speed-accuracy split and the gap between micro accuracy and RILE fidelity. They never combine the models, and we do. Hierarchical Structured Model for Fine-to-Coarse Manifesto Text Analysis links sentence codes to document scores through a joint bi-LSTM and Probabilistic Soft Logic rules over coalitions and past elections. It raises Spearman correlation with Chapel Hill expert scores from 0.42 to 0.61. That calibration depends on relational metadata, which new parties and debates lack, whereas our correction needs only audit labels. HUQ supplies a method we reuse in part. It mixes aleatoric and epistemic uncertainty for selective classification and cuts area under the risk-coverage curve by 13.0% on 20 Newsgroups with an ELECTRA backbone. Its risk, however, is per-instance error. Our contribution is to redefine the risk being controlled as aggregate measurement error and to show that this changes which items should be deferred.

Two other papers relate more loosely. SPIO ensembles top-ranked tabular pipelines, and Political Text Scaling Meets Computational Semantics improves unsupervised scaling (Pearson 0.32 vs 0.15 for Wordfish). Neither addresses the problem of allocating a labeling budget.

Baselines

The naive reference point is the CRF alone on every statement, with RILE computed directly from its labels and no expensive calls. This is the cheapest system, and Nikolaev and Papay already describe it as giving comparatively good RILE estimates. It is the number every cascade must improve on. No reported RILE-error figure for it was found on Lacuna; the paper reports it only qualitatively, with segmentation-plus-labeling F1 within four points of a gold-boundary baseline. We therefore reproduce it on their 1,314-manifesto MARPOR subset with their released configuration.

The ceiling is full Flan-T5 XL on every statement, also from Nikolaev and Papay. This is the strongest existing automatic method for in-domain labeling, though no numeric F1 or RILE error appears in the file. We also report the gold-label RILE. A cascade beats the ceiling in the meaningful sense if it matches the full Flan-T5 RILE error within its bootstrap interval while using at most 15% of Flan-T5's tokens.

The strongest competitor on the question we actually ask is a standard confidence cascade. It routes the statements where the CRF ensemble's 57-way predictive entropy is highest, at the same budget, using the same Flan-T5 and the same checkpoint ensemble. A HUQ-scored version is included as a stronger form of this baseline. Our method wins if, averaged over budgets from 1% to 20% of tokens, its area under the curve of RILE error against budget is lower, with paired document-level bootstrap intervals that exclude zero.

The ablations remove one ingredient at a time from our score: the coarse-bin aggregation, the $1/N_d^2$ leverage weight, and the audit correction. A random-routing cascade at equal budget isolates the value of any targeting at all. All systems share the same train/dev/test manifesto split by country-election and the same CRF and Flan-T5 weights. Routing hyperparameters, namely the audit fraction and the ensemble size, are tuned only on the development split with an equal number of configurations per method.

Why not just…?

Why not just run the CRF everywhere, since its RILE is already good? On in-domain manifestos this is a real threat. If the CRF's RILE error is already near Flan-T5's, the cascade offers little. The error-versus-budget curve settles this directly: a flat curve is a negative result we will report. We expect the gap to show up mainly on short manifestos and on Hansard, where Nikolaev and Papay report systematic procedural mis-coding that no amount of sentence-level speed fixes.

Why not just correct aggregate counts with a confusion matrix, skipping routing? This is the most serious alternative. If CRF errors are mostly systematic, estimating a confusion matrix on held-out data and inverting it for each document's bin proportions could recover RILE at zero inference cost. We include it as a baseline. Our audit term is effectively a local version of this correction, so the comparison isolates what routing adds beyond bias correction. If confusion-matrix correction alone matches the full system, we will report that the problem is bias rather than variance, and the routing component will be dropped.

Why not just send a random sample of statements to Flan-T5 or human coders and estimate RILE from it? Random sampling is unbiased but has high variance for short documents. Its variance does not use the fact that most statements are confidently neutral or confidently on one side. The random-routing ablation and the audit-only variant measure this directly at matched budgets.

Why not just prompt one large language model for everything? At under 3 tokens per second for Llama-3.1 8B, as Nikolaev and Papay measure, a multi-decade Hansard pass is not practical. ClimateEval reports few-shot macro F1 below 0.34 on fine-grained domain classification tasks, which suggests prompting struggles exactly where the 57-way coding scheme is hardest.

Proposed Experiments

The primary experiment uses the MARPOR subset from Nikolaev and Papay with gold codes. For each method and each budget we report three quantities. The first is per-document absolute RILE error against gold-coded RILE. The second is Spearman correlation of country-level party rankings with gold-coded rankings. The third is Spearman correlation with Chapel Hill expert scores on the parties covered, for comparison with the 0.61 reported by Subramanian et al. We run this twice, once with Flan-T5 as the expensive labeler and once with simulated human coders drawing gold labels. The second run yields the practical figure: how many sentences per manifesto a coder must read.

The decisive check has two parts, and the claim fails if either fails. First, at equal budget, RILE-leverage routing must beat entropy routing and HUQ routing. Second, the leverage weight must matter, which shows up as the largest gains on short manifestos.

The transfer experiment runs the manifesto-trained cascade on Hansard. Gold labels do not exist there, so we hand-code a stratified sample of several hundred debate statements as an evaluation set. We measure two things on that sample: error in party-level RILE, and the rate at which procedural speech is mis-coded as substantive. We compare CRF-internal uncertainty with the HUQ-style density variant to test whether epistemic signals are needed under genre shift.

We also report throughput for every configuration, in tokens per second over the whole archive. The question the program must answer is practical: whether a Hansard-scale archive can be coded with an expensive-label budget a research group could actually afford.