TL;DR
ScienceJudge is a new framework designed to catch subtle, fluent errors in AI-generated scientific reports by verifying them against multimodal evidence (text, tables, and figures). By using a novel technique called Pattern Perturbation Augmentation (PPA) to generate realistic "hard negative" training examples, the system achieved first place in the NLPCC 2026 Shared Task 10 competition.
What Problem Does This Paper Address?
As Large Language Models (LLMs) increasingly assist in scientific writing, a critical risk emerges: "hallucinations" that are scientifically plausible and grammatically perfect but factually incorrect. These reporting errors are often subtle—misattributing a data point to the wrong figure, slightly overstating the scope of a finding, or reversing a causal relationship.
Detecting these errors is difficult because:
- Subtlety: The claims remain fluent and "look" right.
- Data Scarcity: There are very few labeled examples of these specific, fine-grained scientific errors compared to the vast amount of correctly supported text.
- Multimodality: Verification requires reasoning across text, complex tables, and scientific imagery simultaneously.
Methods
The authors introduce ScienceJudge, an ensemble framework powered by a sophisticated data augmentation strategy called Pattern Perturbation Augmentation (PPA).
Pattern Perturbation Augmentation (PPA)
Instead of using simple back-translation or random word swapping, PPA treats errors as specific "faithfulness-breaking transformations." The authors used GPT-5.5 to analyze existing errors in the training set and induce reusable patterns. PPA operates at three levels:
- Pattern-Level Instantiation: Directly replicates specific error types (e.g., swapping Figure 5 for Figure 4) while keeping the rest of the sentence intact.
- Operation-Level Abstraction: Extracts the underlying logic of an error—like "scope broadening"—and applies it to new scientific contexts.
- Cross-Domain Transfer: Takes these error-generating operations and applies them to evidence from different fields like biomedicine and physics to improve the model's versatility.
The ScienceJudge Ensemble
The final system is an ensemble of seven predictors:
- Five Vision-Language Models (VLMs): Based on Qwen3-VL-8B, each fine-tuned on different slices of PPA-augmented data.
- Two Instruction-Guided LLMs: Zero-shot GPT-5.5 models using specific prompts to focus on boundary-aware and compact-criteria verification.
The system uses class-weighted voting to ensure that rare error categories (like "Unsupported Causal") aren't drowned out by the dominant "Supported" label.
Datasets
The primary benchmark is the NLPCC 2026 Shared Task 10 Track 1 dataset, which contains 3,333 paragraphs of AI-generated scientific claims paired with multimodal evidence bundles.
To solve the data imbalance, the authors created several augmented datasets using PPA, significantly increasing the number of negative examples (Contradictions, Overgeneralizations, etc.) available for training.
Key Insights and Ideas
- Errors are Structural, Not Random: The paper’s core "aha moment" is that scientific reporting errors follow recurring patterns. By modeling these patterns explicitly, we can generate "hard negatives" that are far more effective for training than random noise.
- The "Identity" Trap: Standard models often fail when a claim is true but cited to the wrong figure. PPA specifically teaches the model to treat the figure/table reference as a semantic anchor that must match the evidence.
- Diversity Beats Scale: The ensemble performed better when its components were trained on different augmentation strategies (Pattern vs. Operation vs. Domain) rather than just more data of the same type.
- Multimodal Necessity: The authors found that text-only versions of their models performed significantly worse, proving that the models are actually utilizing the visual and tabular evidence provided.
Key Results
- Leaderboard Success: ScienceJudge ranked 1st on the official NLPCC 2026 Task 10 Track 1 leaderboard with a final score of 0.6890.
- Performance Gains: On a local development set, the framework improved the average score from 0.8124 (base fine-tuning) to 0.8692.
- Error Detection: The Macro-F1 score reached 0.8466, and the Paragraph Exact Match (PEM) score—a very strict metric—hit 0.8919.
- Ablation: Pattern-level augmentation was the single most effective strategy, particularly for catching "Unsupported Entity" errors (like wrong figure citations).
Limitations and Future Work
- Post-hoc Evaluation: The authors acknowledge that the local development set was used for model selection and tuning, which might make the statistical tests less independent than a blind trial.
- Human Audit Scale: The manual quality audit of the augmented data was limited to 100 samples; a larger-scale validation would be beneficial.
- LLM Dependency: The system currently relies on GPT-5.5 for pattern induction and as part of the ensemble. Future work could explore if smaller, open-source models can handle these specialized roles.
- Domain Expansion: While cross-domain transfer was tested, the full extent of how these patterns vary across highly specialized scientific niches remains to be explored.
Summary updated