Degeneracy Filtering of Retrieved Evidence for Professional Exam QA

When a question-answering system for a specialized field such as pharmacy, law or engineering lacks the needed domain knowledge, the standard fix is to retrieve textbook passages and knowledge-base facts and paste them in front of the question. Much of what comes back is off-topic, and it misleads the reader model. On the Chinese National Licensed Pharmacist Examination (NLPEC), Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text attributes 32% of its system's errors to retrieval noise. That is second only to knowledge that was simply missing. This note proposes a cheap, unsupervised filter. It pools everything retrieved for a question into one word co-occurrence graph, in which words are nodes and an edge joins two words that appear within a few tokens of each other. It then keeps only the sentences whose words sit in the graph's densely interlinked core and connect to the question's own terms. The peripheral, loosely attached material is discarded before the reader sees it.

The bet rests on a specific prior result. On meeting transcripts with speech-recognition word error rates up to 37%, scoring words by graph degeneracy beat centrality-based selection clearly. Degeneracy asks how deeply a word is embedded in a mutually connected cluster, and the k-core, the largest subgraph in which every word has at least k neighbors, is its standard measure. Centrality measures such as PageRank instead score how often a random walk reaches a word (Combining Graph Degeneracy and Submodularity for Unsupervised Extractive Summarization). Retrieval noise in a professional exam looks structurally like transcription noise. It consists of fragments that share a few surface words with the question but do not form a coherent cluster with the rest of the evidence. We test whether the same structural signal removes it.

Core Claim

Our claim is that core-based selection over a pooled, question-anchored word graph gives a reader a cleaner context than passage-by-passage relevance scoring at the same token budget. Filtering at the sentence level also turns the retrieved knowledge-graph facts and textbook snippets into a single evidence set. The resulting gain on NLPEC should be concentrated in the errors that are due to retrieval noise.

The contribution is a specific combination that does not exist in the file. Graph-of-words degeneracy has been used for keyword extraction (Main Core Retention on Graph-of-Words for Single-Document Keyword Extraction), which showed that the weighted main core outperforms PageRank and HITS on Hulth2003 (61.24% versus 58.94% and 57.86% F1). It has also been used for summarization, but never as an evidence filter for domain question answering. The domain systems, for their part, run graph networks only over curated entity graphs and read the retrieved text as a flat sequence. KMQA does exactly this: it converts CMeKG triples into pseudo-sentences and runs a GCN over the entity subgraph (Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text). Of the connecting work, the closest is From Graph to Word Bag: Introducing Domain Knowledge to Confusing Charge Prediction. It projects a legal knowledge graph onto a flat bag of discriminative words that steers attention. It never builds a graph over the input text, and it addresses classification rather than evidence selection. We do the reverse. Our graph is built from the retrieved text itself, and we use its structure to decide what to keep.

Research Program

For each question–option pair in NLPEC, we take the retrieved textbook snippets and the CMeKG triples, with the triples rendered as pseudo-sentences exactly as KMQA does. We segment the text with a medical Chinese word segmenter, remove stop words, and add the question and option text. We then build one undirected co-occurrence graph with window size 3 or 4. The file reports consistent degradation at larger windows (Text Level Graph Neural Network for Text Classification, Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks). Edges are weighted by co-occurrence count.

We compute weighted core numbers and the CoreRank score of each word, which is the sum of its neighbors' core numbers. We then score each sentence $s$ as

$$\text{score}(s) = \sum_{v \in s} \text{CR}(v)\,\exp(-\lambda\, d(v, Q)),$$

Here $\text{CR}(v)$ is the CoreRank of word $v$, $d(v,Q)$ is the shortest-path distance in the graph from $v$ to the nearest question or option word, and $\lambda \ge 0$ controls how strongly evidence must connect to the question. Sentences are chosen greedily under a fixed token budget with the coverage-plus-diversity objective used by Tixier et al. Diversity matters because the six or so retrieved snippets often repeat the same drug monograph.

The anchoring term is the part of the design that needs care. An unanchored main core captures whatever topic dominates the retrieved pool. When retrieval drifts, for example to many snippets about a sibling drug, the dominant topic is the noise. The distance penalty keeps structural cohesion from overriding connection to the question.

The filtered context is then passed to two readers so the result does not depend on one model. The first is a KMQA-style RoBERTa reader, retrained on filtered contexts. The second is an off-the-shelf instruction-tuned open LLM prompted zero-shot with the same budget. The filter has two free parameters, $\lambda$ and the budget, plus the window size. It has no trained weights, so its cost is negligible next to either reader.

Baselines

The naive reference points are the reader with no retrieved context and the reader given the full unfiltered retrieval, truncated to the budget in retrieval order. Together they bound how much retrieval helps and how much noise hurts.

The strongest reported system on this benchmark is KMQA, which achieves 61.8% accuracy on the NLPEC test set (Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text). We will reproduce it with its original retrieval and treat our reproduced number as the reference. We will report both our reproduced figure and the published 61.8%. Swapping only the context-selection step inside this reproduction is the cleanest test of the filter.

The strongest competitor on the actual question, which is how to select evidence, is a supervised cross-encoder reranker. This is a model that reads the question and each passage together and outputs a relevance score. We fine-tune it on NLPEC training questions, using the reader's correctness as weak supervision, and then take its top sentences under the same budget. No reported result was found on Lacuna for a reranker on NLPEC, so we will produce this number ourselves. It is the baseline the proposal has to beat.

A second non-graph competitor selects sentences by dense-embedding similarity to the question, with maximal marginal relevance for diversity. It captures cohesion and relevance without any graph.

Two ablations isolate the key ingredient. The first uses the same graph and the same anchoring but replaces CoreRank with PageRank, mirroring the comparison Tixier et al. ran on noisy speech. The second drops anchoring ($\lambda = 0$).

Every filter receives the same retrieval pool, the same token budget, and the same reader. Each also gets the same dev-set hyperparameter search of 30 configurations. We count it as beating the reranker only if the filter improves test accuracy over it by at least one point, significant under a paired bootstrap, with both readers. The expected ceiling is modest. If every noise-caused error of a roughly 38%-error system were fixed, accuracy would rise by about 12 points, so gains of 2–4 points would already be meaningful.

Why not just…?

Why not just use a fine-tuned reranker? This is the real threat, and we do not claim to know the answer. A reranker scores each passage against the question on its own. The degeneracy filter asks whether a sentence belongs to the mutually reinforcing cluster that the whole pool forms around the question. That signal should matter most when noisy snippets are lexically close to the question, which is exactly the case where independent scoring fails. The head-to-head at a matched budget settles it, and so does a combined variant: reranker top-k followed by degeneracy pruning. If the reranker alone matches the combination, the graph adds nothing and we will say so.

Why not just let a long-context LLM read everything and ignore the noise? The file shows that more capable or better-prompted models do not reliably use extra domain text. Adding explicit theorem definitions to prompts gave no significant gain in TheoremQA: A Theorem-driven Question Answering Dataset, and reasoning-focused models did not clear 50% F1 on AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios. Neither result is about noise specifically, however. Our LLM-reader condition with the full unfiltered context tests this alternative directly.

Why not just retrieve fewer passages? Truncating to the top one or two raises precision but cuts the multi-hop facts that pharmacology questions need, since KMQA's error analysis attributes 44% of errors to missing information. We sweep the budget for all methods and report accuracy against context length. The filter should dominate the whole curve, not just one point on it.

Why not just use embedding-based selection? The dense MMR baseline tests this directly. If it matches the graph filter, then cohesion itself is doing the work and the graph structure is not needed.

Proposed Experiments

The main experiment reports NLPEC test accuracy for every baseline and both readers at three token budgets.

The decisive mechanistic check is error re-annotation. Using KMQA's error categories, we label 300 test errors from the reranker condition and 300 from the filter condition, blind to condition. The filter should lower the retrieval-noise share relative to the reranker without raising the missing-information share. If accuracy rises but the noise share does not fall, the gain is coming from something other than noise removal.

A controlled stress test injects lexically similar distractor snippets into the pool at rates of 0%, 25%, 50% and 100% of the original pool size. The distractors are drawn from monographs of drugs in the same therapeutic class. The degeneracy filter should degrade more slowly than the reranker, and more slowly than PageRank on the same graph, as it did under speech noise.

On a 200-question subset, annotators will mark which retrieved sentences are necessary. We then report evidence recall and precision for each filter, which separates "keeps the right evidence" from "the reader happens to do better."

As a limited transfer check, we apply the unchanged filter to TheoremQA with retrieved theorem descriptions. The file reports GPT-4 at 51.0% there with Program-of-Thoughts prompting. A null result would be informative too, since TheoremQA suggests the bottleneck there is applying knowledge rather than noise.