ScienceJudge: Pattern Perturbation Augmentation for Sentence-Level Scientific Claim Verification

Chuhai Cai, Xin Chen, Min Yang, Feng JiangarXiv 2026 · arXiv

TL;DR

ScienceJudge is a new framework designed to catch subtle, fluent errors in AI-generated scientific reports by verifying them against multimodal evidence (text, tables, and figures). By using a novel technique called Pattern Perturbation Augmentation (PPA) to generate realistic "hard negative" training examples, the system achieved first place in the NLPCC 2026 Shared Task 10 competition.

What Problem Does This Paper Address?

As Large Language Models (LLMs) increasingly assist in scientific writing, a critical risk emerges: "hallucinations" that are scientifically plausible and grammatically perfect but factually incorrect. These reporting errors are often subtle—misattributing a data point to the wrong figure, slightly overstating the scope of a finding, or reversing a causal relationship.

Detecting these errors is difficult because:

  1. Subtlety: The claims remain fluent and "look" right.
  2. Data Scarcity: There are very few labeled examples of these specific, fine-grained scientific errors compared to the vast amount of correctly supported text.
  3. Multimodality: Verification requires reasoning across text, complex tables, and scientific imagery simultaneously.

Methods

The authors introduce ScienceJudge, an ensemble framework powered by a sophisticated data augmentation strategy called Pattern Perturbation Augmentation (PPA).

Pattern Perturbation Augmentation (PPA)

Instead of using simple back-translation or random word swapping, PPA treats errors as specific "faithfulness-breaking transformations." The authors used GPT-5.5 to analyze existing errors in the training set and induce reusable patterns. PPA operates at three levels:

  • Pattern-Level Instantiation: Directly replicates specific error types (e.g., swapping Figure 5 for Figure 4) while keeping the rest of the sentence intact.
  • Operation-Level Abstraction: Extracts the underlying logic of an error—like "scope broadening"—and applies it to new scientific contexts.
  • Cross-Domain Transfer: Takes these error-generating operations and applies them to evidence from different fields like biomedicine and physics to improve the model's versatility.
Overview of the ScienceJudge framework, showing the PPA process and the ensemble inference stage.

The ScienceJudge Ensemble

The final system is an ensemble of seven predictors:

  1. Five Vision-Language Models (VLMs): Based on Qwen3-VL-8B, each fine-tuned on different slices of PPA-augmented data.
  2. Two Instruction-Guided LLMs: Zero-shot GPT-5.5 models using specific prompts to focus on boundary-aware and compact-criteria verification.

The system uses class-weighted voting to ensure that rare error categories (like "Unsupported Causal") aren't drowned out by the dominant "Supported" label.

Datasets

The primary benchmark is the NLPCC 2026 Shared Task 10 Track 1 dataset, which contains 3,333 paragraphs of AI-generated scientific claims paired with multimodal evidence bundles.

To solve the data imbalance, the authors created several augmented datasets using PPA, significantly increasing the number of negative examples (Contradictions, Overgeneralizations, etc.) available for training.

Table 1. Statistics of the base training set and PPA datasets. Columns report sentence counts for each label and the number of paragraphs.

Key Insights and Ideas

  • Errors are Structural, Not Random: The paper’s core "aha moment" is that scientific reporting errors follow recurring patterns. By modeling these patterns explicitly, we can generate "hard negatives" that are far more effective for training than random noise.
  • The "Identity" Trap: Standard models often fail when a claim is true but cited to the wrong figure. PPA specifically teaches the model to treat the figure/table reference as a semantic anchor that must match the evidence.
  • Diversity Beats Scale: The ensemble performed better when its components were trained on different augmentation strategies (Pattern vs. Operation vs. Domain) rather than just more data of the same type.
  • Multimodal Necessity: The authors found that text-only versions of their models performed significantly worse, proving that the models are actually utilizing the visual and tabular evidence provided.

Key Results

  • Leaderboard Success: ScienceJudge ranked 1st on the official NLPCC 2026 Task 10 Track 1 leaderboard with a final score of 0.6890.
  • Performance Gains: On a local development set, the framework improved the average score from 0.8124 (base fine-tuning) to 0.8692.
  • Error Detection: The Macro-F1 score reached 0.8466, and the Paragraph Exact Match (PEM) score—a very strict metric—hit 0.8919.
  • Ablation: Pattern-level augmentation was the single most effective strategy, particularly for catching "Unsupported Entity" errors (like wrong figure citations).

Limitations and Future Work

  • Post-hoc Evaluation: The authors acknowledge that the local development set was used for model selection and tuning, which might make the statistical tests less independent than a blind trial.
  • Human Audit Scale: The manual quality audit of the augmented data was limited to 100 samples; a larger-scale validation would be beneficial.
  • LLM Dependency: The system currently relies on GPT-5.5 for pattern induction and as part of the ensemble. Future work could explore if smaller, open-source models can handle these specialized roles.
  • Domain Expansion: While cross-domain transfer was tested, the full extent of how these patterns vary across highly specialized scientific niches remains to be explored.

Summary updated

Abstract

Sentence-level scientific claim verification determines whether each sentence in an AI-generated scientific paragraph is supported by a given multimodal evidence bundle. Fine-grained reporting errors often preserve fluent and plausible statements while introducing subtle evidence-claim deviations, making them difficult to detect under sparse and imbalanced supervision. We propose Pattern Perturbation Augmentation (PPA), which models reporting errors as label-specific faithfulness breaking transformations and induces reusable error patterns from labeled examples. PPA applies these patterns to supported claims to construct controlled hard negatives and expands their coverage through pattern-level, operation-level, and cross-domain augmentation. Based on these augmented views, we develop ScienceJudge, which integrates PPA-trained vision-language predictors and instruction-guided LLM predictors through class-weighted ensemble inference. On the local development set, ScienceJudge improves the average score from 0.8124 to 0.8692, achieving a Macro-F1 of 0.8466 and a Paragraph Exact Match score of 0.8919. Our submitted system ranks first on the official NLPCC 2026 Task 10 Track 1 leaderboard with a final score of 0.6890.
BibTeX
@misc{cai2026sciencejudge,
  author = {Chuhai Cai and Xin Chen and Min Yang and Feng Jiang},
  title = {{ScienceJudge: Pattern Perturbation Augmentation for Sentence-Level Scientific Claim Verification}},
  year = {2026},
  eprint = {2610.40238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40238},
}
Figure