Counterfactual Pairwise Augmentation Improves Small Pointwise LLM Judges

Mohammad Zangooei, Peyvand Teymoori, Mohammad Robati Shirzad, Soroosh ShahtalebiarXiv 2026 · arXiv

TL;DR

Counterfactual Pairwise Augmentation (CPA) significantly boosts the performance of small language models (SLMs) acting as "pointwise" judges by synthesizing minimal-edit response pairs that explicitly highlight decision boundaries. By training jointly on these "same-source" pairs rather than merging separate experts, a specialized Qwen3-8B model outperformed frontier models like Gemini 3.6 Flash by nearly 12 points on specialized evaluation criteria.

What Problem Does This Paper Address?

In modern agentic systems, small language models (SLMs) are often used as "pointwise judges"—gatekeepers that score a single response against a specific criterion (e.g., "is this answer factually correct?"). SLMs are ideal for this because they are fast and cheap to serve locally.

However, fine-tuning an SLM to be a reliable judge is difficult. Standard "pointwise" datasets (query-response-label triplets) often have labels that are "under-determined." Because different queries in the dataset vary across many dimensions at once—topic, style, length, and complexity—it is hard for the model to isolate exactly which feature triggered a "pass" or "fail" label. The model struggles to find the precise decision boundary.

Figure 1: Online pointwise judge gates each step.

Methods

The authors introduce CPA (Counterfactual Pairwise Augmentation) to make the deciding difference explicit during training. The pipeline consists of three main stages:

  1. Counterfactual Synthesis: For every item in a pointwise dataset, a "teacher" model (DeepSeek-V3.2) synthesizes a counterfactual response to the same query designed to carry the opposite label.
  2. Auditing: A second reasoning model (DeepSeek-R1) audits these pairs. It checks the pair in both response orders to ensure the synthesized response truly flips the label while keeping other attributes (style, length, etc.) fixed. Only verified pairs are kept.
  3. Joint Fine-Tuning: Instead of training a separate pairwise model and merging it (a common technique), the authors fine-tune a Qwen3-8B model using LoRA adapters jointly on both the original pointwise data and the new counterfactual pairwise data.
Figure 2: The CPA pipeline. Teacher Tsyn synthesizes a label-flipping counterfactual; auditor Taud verifies it in both pairwise orderings, retrying up to K times on failure. Both orderings of an accepted pair are retained as training data alongside the pointwise items. The judge is deployed pointwise-only.

A key contribution of the paper is a second-order mathematical analysis of why joint training is superior to model merging in this context. They argue that model merging is a "scalar" operation that moves all parameters by the same fraction. In contrast, joint training on counterfactual pairs allows the model to selectively absorb "curvature" (information) from the pairwise data only along the specific parameter directions that define the decision boundary, while leaving other parameters at their pointwise-optimal positions.

Datasets

The researchers evaluated CPA across six distinct binary criteria using public source corpora:

  • AI-ness: (HC3) Detecting if text is AI-generated.
  • Criminality: (Aegis 2.0) Identifying illegal or criminal content.
  • Safety Leakage: (Nemotron) Detecting harmful or inappropriate content.
  • Hallucination: (HaluEval) Checking for factual support in context.
  • Tool-parameter fulfillment: (Hermes) Verifying tool-call parameter accuracy.
  • Correctness: (TruthfulQA) Evaluating factual truthfulness.

Key Insights and Ideas

  • The Power of "Same-Source" Pairs: Unlike previous methods that use separate corpora for pairwise training, CPA uses the same queries. This "counterfactual minimality" ensures that the only difference between two responses is the satisfaction of the criterion, effectively "pointing" the model directly at the decision boundary.
  • Joint Training > Model Merging: The paper reconciles their findings with prior work (like Prometheus 2) that favored model merging. They show that merging is better when training sets are disjoint and errors are decorrelated. However, when using same-source counterfactuals, joint training is mathematically superior because it acts as a "matrix-weighted" average of the two objectives.
  • SLMs as Specialized Experts: The results suggest that for narrow, well-defined evaluation tasks, a specialized 8B model can significantly outperform massive "frontier" models that haven't been explicitly trained to see these specific boundaries.

Key Results

CPA demonstrated clear superiority over all baselines:

  • Pointwise Improvement: CPA improved the macro-F1 score by 2.5 points (90.4 vs 87.9) over training on pointwise data alone.
  • Beating the Merge: It outperformed the model-merging strategy on all six criteria at every tested merge weight.
  • Surpassing Frontier Models: Most impressively, the specialized Qwen3-8B (CPA) outperformed Gemini 3.6 Flash by 11.8 points and GPT 5.6 by 14.3 points on the macro-average across the six criteria.
Table 1: Per-criterion macro-F1 (%), mean ± sample standard deviation over 10 inference runs. Merged takes the best λ per criterion. The λ sweep itself is over 3 runs (Appendix A.3).

Limitations and Future Work

  • Binary Focus: The current study focuses on binary (pass/fail) criteria. The authors note that extending this to ordinal scales (e.g., 1-5 stars) would require synthesizing multiple counterfactuals to map every adjacent rubric level.
  • Synthesis Efficiency: Currently, counterfactuals are generated for all items. The authors suggest that future work could use "active learning" to target synthesis only for items near the decision boundary where the judge is most uncertain.
  • Rationale Quality: While they use a "student-first" rationale approach, they acknowledge that the quality of the reasoning chain remains a critical bottleneck for judge accuracy.

Summary updated

Abstract

Agentic systems increasingly gate individual steps with a pointwise judge that scores a response against a criterion. Small language models are attractive for this role because they can be served and specialized efficiently. Such a judge must locate the criterion's decision boundary, but pointwise labels under-determine that boundary in fine-tuning: labeled responses to different queries differ along many dimensions at once, so they do not reveal which features decide the label. We introduce CPA (Counterfactual Pairwise Augmentation), which makes the deciding difference explicit in training. For each pointwise item, CPA synthesizes and audits a counterfactual response to the same query intended to carry the opposite label. It frames the accepted pair as a pairwise task asking which of the two responses satisfies the criterion. The judge is fine-tuned jointly on both pointwise and pairwise datasets and deployed in pointwise mode only. A second-order analysis shows when this should outperform the prevailing strategy of merging separately trained pointwise and pairwise experts: merging moves every parameter direction toward the pairwise expert by the same fraction, whereas joint training can hold the pointwise optimum wherever the pairwise loss is flat and absorb its curvature where it is sharp. This is the regime that CPA's same-source, counterfactually minimal pairwise view is designed to produce. On separate Qwen3-8B LoRA adapters for six binary criteria, CPA improves criterion-averaged pointwise macro-F1 by 2.5 points over pointwise-only training and outperforms model merging on all six criteria at every tested merge weight. It also surpasses the strongest zero-shot frontier judge we evaluate, Gemini 3.6 Flash, by 11.8 points.
BibTeX
@misc{zangooei2026counterfactual,
  author = {Mohammad Zangooei and Peyvand Teymoori and Mohammad Robati Shirzad and Soroosh Shahtalebi},
  title = {{Counterfactual Pairwise Augmentation Improves Small Pointwise LLM Judges}},
  year = {2026},
  eprint = {2610.41149},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41149},
}
Figure