Structure-Gated Late Chunking for Retrieval over Raw Web Pages

Retrieval systems split long documents into chunks and embed each chunk as a vector. That splitting loses context: a chunk that says "its population grew" no longer knows which city it describes. The strongest recent fix is late chunking. It runs the whole document through a long-context encoder first and only then cuts the token vectors into chunks, so every chunk carries document-wide context. This works on clean prose. On real web pages, though, the "whole document" includes navigation menus, cookie banners, related-link lists and footers. The same global attention that repairs lost context now mixes this clutter into every chunk vector.

The usual response is to delete the clutter first with a boilerplate remover, which labels each block of a page as content or boilerplate, and then embed what survives. That has two costs. Deletion is irreversible, so every mistake the remover makes is final. It also flattens the page, discarding which heading governs which paragraph. We propose a two-stage alternative borrowed from decoupled parsing. A cheap first stage recovers an unlabeled "scope skeleton" of the page: how likely each block is to be content, and which heading each block falls under. A second stage runs late chunking under an attention bias derived from that skeleton. Content tokens are softly steered away from boilerplate and toward their governing headings, and nothing is deleted. The claim is testable and could fail. If hard deletion followed by late chunking does as well, the soft, scope-aware design is unnecessary, and the experiments below are built to show exactly that if it is true.

Core Claim

On raw web pages, dense retrieval quality is limited by two separate failures. One is boilerplate leaking into content vectors through global attention. The other is heading scope lost when pages are flattened into text. A structure-gated late-chunking encoder can fix both at once, and it degrades gracefully when the first stage makes mistakes. Concretely, we expect it to beat three pipelines on web retrieval benchmarks: (a) late chunking on tag-stripped text, (b) late chunking on text cleaned by a state-of-the-art neural boilerplate remover, and (c) ancestor-chain slicing on cleaned text. We expect the margin over (b) to grow as the first stage's error rate grows.

Where This Sits

The "decouple structure from labels" idea comes from parsing, and it supplies our method.

We take the structural-first ordering from this work. We take the warning from the Wang and Li relation gap: hard, discrete handoffs between stages cost accuracy. So our first stage passes continuous probabilities, not decisions, into the second.

Boilerplate removal supplies our first stage and our main competitor.

  • Web2Text collapses the page's HTML tree, scores blocks with convolutional networks, and decodes content/boilerplate labels with a hidden Markov model.
  • BoilerNet replaces the hand-built features with a bidirectional LSTM over tag paths and tokens.

Both output a flat set of kept blocks. Neither says how kept blocks relate to each other or to headings. We reuse BoilerNet's architecture but give it a second output: the governing heading of each block.

Context-restoring encoders motivate the problem and supply our second stage.

  • Late Chunking reports a roughly 3.6% relative nDCG@10 gain over naive chunking on BeIR.
  • Landmark Embedding inserts boundary tokens and learns context-conditioned sentence vectors.

Both were evaluated on clean, linear text, and both attend over every token. TreeJC, from Does Structure Matter?, prepends each leaf's chain of ancestor headings and reaches 54.9 F1 on NQStruct. It assumes clean encyclopedic trees, and on open-web pages its ancestor chains would pull in sidebar paths.

DOM-LM encodes the page tree with structure-aware positions. It ingests all nodes, boilerplate included, and slides windows over subtrees. LiveWeb-IE shows that extractors lose about 15% on live pages because templates change. That drift is exactly why a first-stage remover will make errors on new sites, and why the second stage should not trust it blindly.

The closest existing combination is the obvious pipeline: run BoilerNet or Web2Text, then late-chunk the survivors. We found no paper that reports this. Our proposal differs from it in three ways. It replaces deletion with a soft, learnable attention bias. It adds heading scope as an explicit attention target. It measures robustness to first-stage error directly.

Research Program

Stage 1: the scope skeleton. Stage 1 is a BoilerNet-style sequence model over the leaf blocks of the page's HTML tree. For each block $j$ it produces two outputs:

  • a content probability $p_j$;
  • a pointer to its governing heading block, chosen among preceding heading-like blocks, or a "none" option.

Training uses two sources. Content labels come from CleanEval and GoogleTrends-2017. Scope labels are harvested automatically from pages with well-formed heading hierarchies, then perturbed with injected boilerplate, so the pointer head learns to skip headings that sit inside sidebars and menus. Stage 1 is small and runs in milliseconds per page. For reference, Web2Text reports 54 ms per page.

Stage 2: the gated encoder. The page is linearized in document order with all blocks kept, and passed once through a long-context embedding model. Every attention layer receives an additive bias:

$$\text{score}_{ij} = \frac{q_i^\top k_j}{\sqrt{d}} + \lambda \log p_{b(j)} + \mu\, s_{ij}$$

The terms are:

  • $q_i$ and $k_j$ are the query and key vectors of tokens $i$ and $j$, and $d$ is their dimension.
  • $b(j)$ is the block containing token $j$.
  • $s_{ij}$ equals 1 when token $j$ lies in the governing heading, or an ancestor heading, of token $i$'s block, and 0 otherwise.
  • $\lambda$ and $\mu$ are learned scalars, one pair per layer.

When $\lambda$ is large, the model approaches hard deletion. When it is zero, the model reduces to plain late chunking. Letting the model learn $\lambda$ is the point: the encoder can recover content that Stage 1 wrongly marked as boilerplate when the surrounding context makes it relevant. After encoding, chunk vectors are formed by mean-pooling token vectors weighted by $p_{b(j)}$ over structure-aligned spans, so chunks never straddle a heading boundary. We fine-tune only $\lambda$, $\mu$, and a small adapter on the encoder, using contrastive query–passage pairs from web QA data.

Analysis. We will measure how much attention mass content tokens place on boilerplate tokens, with and without the gate. We will also track retrieval of passages whose key entity or condition appears only in a heading or in an earlier block. The file's anaphora example from Late Chunking, where a Berlin query's similarity rose from 0.75 to 0.85, shows the kind of case we will log.

Baselines

The naive reference is naive fixed-size chunking of tag-stripped text. A user-attention ranking study shows that tag stripping leaves sidebar text flattened into the content stream. This baseline tells us how bad the problem is. A second weak reference is late chunking on tag-stripped text. It isolates what global context alone buys on noisy pages. Late Chunking reports about a 3.6% relative nDCG@10 gain on BeIR. We found no reported result on Lacuna for late chunking on raw web pages.

The strongest competitor is BoilerNet with hard deletion, followed by late chunking on the surviving blocks. It is strongest because it combines the best reported boilerplate remover with the best context-restoring encoder in the file, and it is the design a practitioner would build today. BoilerNet reports gains of 11% in recall and 7% in F1 for the content class over Web2Text on GoogleTrends-2017. On downstream retrieval, the best reported number in the file is Web2Text cleaning on ClueWeb12, with MAP 0.154 versus 0.137 on raw content.

For fairness, this pipeline and ours will share:

  • the identical Stage-1 network and weights, with only the handoff differing (hard threshold versus soft bias);
  • the same encoder, adapter size, fine-tuning data and steps;
  • an equal hyperparameter search budget, including tuning the deletion threshold.

Beating it means a statistically significant nDCG@10 gain on the noisy benchmarks with no loss on clean ones. We will count it as a real win only if the gap widens under the injected Stage-1 errors described below.

We also run a TreeJC-style baseline: ancestor-chain prepending over BoilerNet-cleaned blocks, with naive chunk embedding. On clean trees, TreeJC reports 54.9 F1 and 44.2 exact match on NQStruct.

Three ablations isolate our ingredients:

  • $\mu = 0$ removes scope and keeps only gating.
  • $\lambda = 0$ keeps scope but applies no gating.
  • An oracle Stage 1 with gold content labels and gold scopes sets the ceiling.

Why not just…?

Why not just strip tags and let late chunking's global attention sort it out? Attention has no reason to ignore a cookie banner that appears on every page. DataScience-Polimi at SemEval-2022 found that boilerplate noise hurt news-similarity embeddings more than model choice. WebSRC found that flattened text trails HTML-aware models, 61.67 versus 70.12 exact match. Our dose-response experiment measures this directly for late chunking.

Why not just run BoilerNet, delete boilerplate, and late-chunk the rest? This is the real threat. If Stage 1 were near-perfect, soft gating would add little. Two things argue otherwise. Template drift of about 15% in LiveWeb-IE means Stage 1 will not be near-perfect on new sites. And deletion does nothing to restore heading scope. The head-to-head with shared Stage-1 weights, repeated at controlled Stage-1 error rates, settles it. If hard deletion ties us at every error rate and the scope ablation adds nothing, we will report the simple pipeline as sufficient.

Why not just fine-tune the embedder contrastively on raw pages so it learns to ignore clutter? Such a model could learn some suppression implicitly. However, it never sees explicit scope, and it spends context length on boilerplate. We include this as a baseline, with the same encoder and fine-tuning budget but no gate. If it matches us, the structural prior is not earning its keep.

Why not feed raw HTML to a long-context model or DOM-LM? WebWalker reports that raw HTML exhausts context and drowns the signal. DOM-LM still slides windows over subtrees, reintroducing fragmentation. We include DOM-LM representations with chunk pooling as a comparison on WebSRC.

Proposed Experiments

Stage 1 fidelity. We evaluate Stage 1 on CleanEval and GoogleTrends-2017 against the reported Web2Text F1 of 0.88 and BoilerNet's gains. Pointer accuracy for heading scope is evaluated on held-out sites.

Retrieval on real web collections. On ClueWeb12, we compare against the 0.154 and 0.137 MAP reference points, adding nDCG@10.

Controlled interference benchmark. We take the long documents in LongEmbed and the NQStruct pages and wrap them in real boilerplate templates harvested from GoogleTrends-2017 and Common Crawl pages. We sweep the boilerplate share from 0% to 70% of tokens and plot the nDCG@10 curve for each method. The clean 0% point checks that gating costs nothing when there is no clutter.

Reading comprehension. WebSRC exact match, compared against the reported 70.12 and 73.22, tests whether gated chunks help reading, not only retrieval.

Drift. Pages from LiveWeb-IE sites not seen in training test drift robustness.

Decisive checks. The first decisive check is the error-injection sweep. We flip Stage-1 content labels at rates from 0% to 30% and corrupt scope pointers at matching rates. Soft gating should degrade more slowly than hard deletion. If it does not, the core claim fails. The second is the $\mu = 0$ ablation, evaluated specifically on queries whose answer depends on heading context. If scope adds nothing there, the parsing-inspired half of the design is unnecessary and we will say so.