Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA

Ekaterina Alimaskina, Denis Shveykin, Gleb Molodtsov, Igor Shalygin, Alexey Kadeishvili, Aleksandr BeznosikovarXiv 2026 · arXiv

TL;DR

Using language models to generate their own question-answer (QA) training data—a process known as "self-study"—is not a neutral preprocessing step. This paper reveals that QA generation is a highly fragile selection policy that saturates early, fixates on superficial formatting cues, and is easily hijacked by malicious or messy text artifacts. Fortunately, lightweight procedural adjustments like sentence-targeted generation and basic regex filtering can dramatically reduce these vulnerabilities.

What Problem Does This Paper Address?

A popular paradigm for training modern language models involves generating synthetic QA pairs directly from a source corpus. This data is then used to fine-tune models, distill larger models into smaller ones, or compress long contexts into compact KV caches.

Historically, researchers treated this generation step as a neutral, automated way to "scan" a document. This paper challenges that assumption. The authors investigate the hidden biases and security risks of the two-stage self-study loop (where one model decides what to ask, and another answers). They find that the process is neither uniform nor stable, leaving downstream models exposed to biased training data and indirect prompt injections embedded within untrusted source documents.

Methods

The authors systematically dissect the self-study pipeline using two main analytical angles:

  1. Evidence Footprint Analysis: They use a strong judge model (Qwen3-32B) to map out exactly which verbatim text spans ("anchors") are selected to support generated questions. This allows them to measure document coverage and depth across different prompt seeds.
  2. Fragility & Injection Probes: They introduce passive text anomalies into the source documents to see how easily the pipeline breaks. This includes a formatting perturbation probe (adding a generic "most important passage" banner), a salient artifact probe (inserting raw, uncleaned HTML/parser residue), and an instruction injection probe (embedding behavioral constraints like refusal templates or task conflicts into the text).

To fix these issues, they evaluate procedural safeguards: multi-diverse prompting (asking for multiple distinct questions at once), sentence-targeted generation (forcing the model to ask about a pre-selected sentence), and upstream context sanitization (using regex, DeBERTa classifiers, or LLM-based filters).

Datasets

The experiments were conducted across three distinct document collections representing different lengths and domains:

  • Cartridges paper (cartridges.tex): A single, highly technical machine learning paper.
  • LongHealth: A benchmark containing 10 long clinical documents.
  • QASPER: A dataset consisting of 16 full-text NLP research papers.
  • Synthetic Tabular Data: A fact-dense 60x10 table used to stress-test prompt diversity.

Key Insights and Ideas

  • The Saturation Trap: Question generators do not scan documents evenly. They quickly find "salient" regions and repeatedly generate questions around them, leading to diminishing returns and near-duplicate training data.
  • Formatting Trumps Semantics: Models are easily distracted by superficial presentation. Headings, bold text, lists, and even poorly cleaned parser markup act as "shared attractors" that hijack question generation across different model families and scales.
  • The Capability Paradox in Security: When an untrusted document contains an embedded instruction (e.g., "Ignore previous instructions and reply with X"), larger, more capable models are more likely to be hijacked during the answering stage. Because they follow instructions more reliably, they comply with the malicious text rather than sticking to the QA task.
Figure 1: Cumulative evidence coverage over generated interactions. Coverage grows rapidly at first and then saturates across all corpora and model sizes, indicating diminishing returns from additional generated interactions.

Key Results

  • Artifact Hijacking: Inserting a single block of raw HTML residue into a document caused injection hit rates to skyrocket. Models grounded up to 77–94% of their generated questions in the useless artifact rather than the actual document content.
Figure 4: Injection hit rate under uniformly distributed HTML-like artifacts. Rows correspond to generator models, columns to prompt seeds, and panels to corpora. Higher values mean that generated interactions are more often grounded in the injected artifact rather than in the original document content.
  • Answering Hijacking: Answering models complied with embedded instructions in over 85% of configurations, directly passing poisoned behavior into the final training data.
  • Procedural Fixes Work: Externalizing the anchor choice via sentence-targeted generation cut the artifact injection hit rate significantly (e.g., from 61.8% to 17.5% on QASPER).
  • Sanitization Benefits: Upstream keyword-regex filtering proved to be the most effective defense against behavioral hijacking, slashing mean injection compliance from 88% down to 13% while retaining 100% of the clean text.

Limitations and Future Work

While the proposed fixes are lightweight and effective, they introduce trade-offs. For instance, forcing a model to generate questions based on a single pre-selected sentence (sentence-targeted) makes the data highly local. This prevents the model from generating complex, high-level questions that require connecting facts across multiple paragraphs or sections.

The authors note that future work should explore how to externalize anchor selection for multi-sentence or document-level relations without letting the generator fall back into its biased hotspot habits.

Summary updated

Abstract

Language models are increasingly taught from synthetic question--answer (QA) supervision: a model generates questions about a document, answers them from the same text, and the resulting pairs are used to fine-tune, distill, or compress knowledge into another model. We show that this generation step is not neutral preprocessing. It is an implicit policy that both selects which evidence becomes training signal and decides how that evidence is answered, and it is fragile at both stages. When choosing what to ask, generators do not scan a document uniformly. Coverage saturates early and concentrates on salient spans, diverse prompts converge on the same regions, and what looks question-worthy is driven by local presentation. As a result, salient artifacts such as poorly cleaned markup can hijack question generation across model families and scales. When answering, the model that produces the supervision tends to obey instruction-like passages embedded in the text. This compliance depends on the intent and surface form of the passage rather than its strictness, and is worst under task conflict, where larger models comply more often. These failure modes arise from choices made during QA generation, so they can be reduced without changing the training loop. Tying each question to a fixed target reduces biased selection, and filtering instruction-like spans before answering lowers mean injection compliance from $88\%$ to $13\%$ in our evaluation while retaining nearly all clean text.
BibTeX
@misc{alimaskina2026self,
  author = {Ekaterina Alimaskina and Denis Shveykin and Gleb Molodtsov and Igor Shalygin and Alexey Kadeishvili and Aleksandr Beznosikov},
  title = {{Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA}},
  year = {2026},
  eprint = {2606.32002},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.32002},
}
Figure