Keeping the Qualifier: Facet-Preserving Extraction for Ambiguous Questions over Web Pages

When someone asks an ambiguous question such as "who won the 100m gold?", a good answer lists several values and says which one goes with which reading: men's or women's, which year, Summer or Winter Games. On real web pages the value usually sits in a paragraph or table cell, and the label that says which reading it belongs to, which we call the qualifier, sits higher in the page structure: a section heading, a table caption, a column header. The usual pipeline separates the two in two places. The cleaner that strips navigation menus, ads and footers (boilerplate) also drops or moves headings and captions. The retriever then cuts the page into flat chunks or keeps only the most specific matching element, so the label ends up somewhere else. The system can still produce the correct value attached to the wrong reading. We call this a binding error. Standard extraction scores and standard answer scores do not show where it came from.

This note proposes three things. The first is a way to measure, before any answer is generated, whether extracted text still keeps each value next to its own qualifier. It uses permutation distances, which score how far content blocks have moved from their original order. The second is a selection rule that keeps body text free of duplicates but attaches to each selected passage only the ancestor labels that tell it apart from the other selected passages. The third is a test of whether this closes the gap between ambiguous-question answering on clean Wikipedia paragraphs and the same task over raw web pages.

Core Claim

Our hypothesis is that on ambiguous questions over raw HTML, a large share of answer errors are binding errors rather than missing facts. A binding-integrity score computed on the extracted context should predict downstream ambiguous-QA accuracy across different extraction pipelines better than extraction F1 does. A contrastive qualifier frontier should recover most of the lost accuracy at a fixed token budget. The frontier is a set of passages in which no passage contains another, and each passage carries only the ancestor labels where its path in the page tree splits from the paths of the other selected passages.

Where This Sits

The QA side motivates the problem. ASQA pairs ambiguous factoid questions with long-form answers and scores them with Disambig-F1 (a QA model must recover each reading's answer from the generated text) and the DR score. It finds a large synthesis gap even with oracle passages. All of its passages, however, are clean Wikipedia text. Tree of Clarifications builds a tree of candidate interpretations and prunes it with retrieval-backed self-verification. That tree is over interpretations, not over page structure, and it retrieves flat passages. Our work keeps the question side fixed and changes the document side. It also tests whether interpretation trees still help once qualifiers are lost.

The structured-retrieval literature supplies both the method and its constraints. Greedy overlap filtering (Ogilvie et al.) drops any ancestor or descendant of an already-selected element. Strict antichains are sets of tree nodes where none contains another. Contrastive Antichains in Hierarchies uses them to guarantee zero vertical redundancy and scores nodes by how much they contrast with each other. Both rule out returning a parent heading together with its child passage. Vu et al. show that fixed leaf-only or root-only pruning biases retrieval toward the wrong granularity. Work on the redundancy paradox (Gallinari et al.; Kazai) warns that returning whole branches duplicates text and clutters results. We borrow the contrastive idea but apply it to labels rather than whole nodes. Body text stays an antichain, while qualifier strings from ancestors may be attached. This is a deliberate relaxation, and it addresses the specific failure these papers leave open: losing the framing header.

Web2Text is the cleaning method we build on. It merges single-child DOM nodes into a "collapsed DOM" and labels blocks as content or boilerplate with CNN potentials and an HMM. It reports F1 0.88 on CleanEval and lifts ClueWeb12 MAP from 0.137 to 0.154. Both scores ignore where content blocks end up relative to each other. Naive tag stripping is known to flatten clutter into the prose (Xu et al.), and leftover markup is known to leak into generated answers (CMUQA).

The permutation work supplies the measuring tool. Kendall's τ tracks human judgments of sentence order (Lapata) and improves MT metrics when combined with lexical overlap (LRscore). AMBER2 adds a jump-width penalty that is lenient on contiguous block moves and harsh on scattering. That distinction is exactly the one we need: a sidebar moved to the end of the page is harmless, while a column header scattered away from its cells is not. None of these metrics has been applied to the output of a web extractor.

Research Program

Benchmark: ASQA-over-HTML. For each ASQA question we fetch the raw rendered HTML of the source Wikipedia articles. This brings back infoboxes, navigation boxes, sidebars, reference lists and footers, which together form a controlled boilerplate regime where the gold answers are already known. For each disambiguated question-answer pair we locate the answer string in the page tree. We then mark its governing qualifier, defined as the nearest ancestor-or-header node whose text contains the distinguishing slot (year, edition, category). This marking is done automatically by string-matching the slot values that differ between the disambiguated questions, and a sample is checked by hand. The result is a set of (qualifier, answer) pairs grounded in the page structure.

Binding-integrity diagnostics. Each extractor produces a sequence of blocks. We compare it with the rendered document order of the content blocks using the normalized Kendall distance

$$K(\pi) = \frac{2\,\#\{(i,j): i<j,\ \pi(i)>\pi(j)\}}{n(n-1)},$$

where $\pi$ maps each of the $n$ surviving content blocks to its position in the extracted sequence. We pair this with AMBER2's jump width, so that benign block moves can be separated from scattering.

The quantity we care most about is pair-level binding integrity:

$$B = \frac{1}{|P|}\sum_{(q,a)\in P} \mathbb{1}\left[q \text{ appears before } a \text{ in the context, with no competing qualifier for the same slot in between}\right],$$

where $P$ is the set of gold (qualifier, answer) pairs and a competing qualifier is another value of the same slot, for example "Winter" between "Summer" and the answer. $K$ tells us whether order survived. $B$ tells us whether the specific link that ambiguity resolution depends on survived.

Method: contrastive qualifier frontier. Retrieval scores elements of the collapsed DOM after Web2Text labeling. Boilerplate-labeled blocks are dropped as body text but kept as candidate qualifiers if they are headings, captions or table headers. We choose body text as a greedy antichain, as in Ogilvie et al. For each selected node we then find the lowest ancestors at which its root-to-node path splits from the paths of the other selected nodes. We attach the qualifier strings found at those split points, together with the row and column headers for table cells, and nothing else. Two passages from the same table therefore receive their differing column headers, not the full repeated heading path. The rule is deterministic, needs no training, and adds a number of tokens that grows only with the number of split points.

Generation. The same few-shot language model reader is used for every condition. We run it both as a single-pass long-form answerer and inside Tree of Clarifications, so we can see whether interpretation trees and qualifier preservation help independently.

Baselines

The naive reference point strips tags from the HTML, splits the text into 100-word chunks as in standard dense retrieval, and retrieves with the same dense retriever and reader. It shows how much the HTML regime costs relative to clean ASQA.

The cleaning baseline runs Web2Text before chunking. It is the strongest published structured boilerplate remover in the file, so it isolates whether better cleaning alone fixes binding.

The structural-filtering baseline applies greedy overlap filtering over collapsed-DOM elements with no attached qualifiers. It is the direct ancestor of our selection rule, and comparing against it isolates the effect of attaching qualifiers.

The heading-path baseline prepends the full page-title-to-section heading path to every chunk. This is the cheap fix a practitioner would try first and our most serious competitor. We run it at two settings: at its natural length, and truncated so that the total context tokens match our method.

The strongest QA method is Tree of Clarifications. It reports Disambig-F1 33.7 on clean ASQA with 5-shot prompting, against 26.4 for supervised T5-Large with JPR. ASQA reports a best DR of 32.1 (T5-O-5) and an oracle DR of 52.5. No reported result was found on Lacuna for ASQA over raw HTML pages, so we rerun Tree of Clarifications ourselves on both clean ASQA and ASQA-over-HTML. We use the same reader model, retriever and number of calls in both settings, so that its HTML drop is measured rather than assumed.

The ablations remove the contrastive part (attach all ancestor qualifiers), remove the antichain constraint (allow parent and child body text together), and restrict qualifiers to headings only (no captions or table headers).

Every system gets the same retriever, reader, prompt, context budget in tokens, and a tuning budget of one shared development split. We count it as beating the baselines if, at matched tokens, our method improves Disambig-F1 over both Web2Text-plus-chunks and token-matched heading paths. The improvement must exceed bootstrap confidence intervals, and it must close at least half of the gap between the naive HTML condition and clean ASQA.

Why not just…?

Why not just prepend the heading path to every chunk? This is a real threat, and it may capture most of the gain on prose-heavy pages. Our reason to expect a gap is structural. On many ambiguous questions the distinguishing qualifier is a table column header or a caption, not a section heading. Full paths also repeat the same tokens across every chunk from a page, and that repetition crowds out content at a fixed budget. The token-matched comparison and the headings-only ablation settle the question directly. If heading paths match our method, we will report that heading preservation is the whole story, and the contribution becomes the diagnostic plus the benchmark.

Why not just give a long-context model the whole cleaned page, or the raw HTML? Prior work reports that raw markup exhausts the context and dilutes signal (WebWalker) and that markup leaks into generated answers (CMUQA). We include a whole-page condition at the largest budget we can afford. It serves as an upper bound on what the reader can do when nothing is cut, and it tells us whether binding fails in reading rather than in extraction.

Why not just ask an LLM judge whether the answer is coherent? A judge scores the output and cannot say which pipeline stage broke the link between value and label. $B$ is computed before generation and is cheap. Lapata's results show that order-based measures can track human judgments ($r=0.45$ against an inter-judge $r=0.56$). We will check whether $B$ predicts per-question Disambig-F1 better than extraction F1 and better than a judge's score of the extracted context.

Decisive Checks

We will run four checks. First, a hand-labeled error analysis on 200 ASQA-over-HTML questions under Web2Text-plus-chunks, which estimates what share of Disambig-F1 losses are binding errors. If that share is small, the premise fails and we stop. Second, a correlation test across all extractors and ablations of $B$, $K$ and extraction F1 against downstream Disambig-F1. Third, the matched-token comparison against heading paths. Fourth, a ClueWeb12 retrieval check, which confirms that keeping qualifiers does not undo Web2Text's MAP gain of 0.137 to 0.154 on ad-hoc search.