Retrieval systems for long documents cut them into chunks. Once a passage is separated from its section, it loses the information that says what it is about. A common fix is to prepend the ancestor section titles to every chunk before indexing. Hybrid retrievers score each chunk twice: once with an exact-keyword score such as BM25, and once with a dense embedding, a single learned vector compared by inner product. In these systems the prepended context is usually fed identically to both scorers. We argue this is the wrong default. Keyword scoring can absorb extra context terms cheaply and additively. A single dense vector has fixed capacity and gets blurrier as its input grows. So context should mostly go to the keyword side, and the embedding should stay focused on the local passage.
The proposal has a second part. Document structure has to be recovered before it can be used, and recovering it costs ingestion time. That cost ranges from about 0.1 s per page for geometric heuristics to roughly 1.5 s per document for a 4B-parameter model that predicts section hierarchies. We borrow two ideas from efficient syntactic parsing: writing a tree as a sequence of per-token labels, and using cheap filtering passes before expensive ones. With these we build a cheap structure parser and a cascade that calls the expensive model only when needed. The central empirical question is how much retrieval quality depends on structure accuracy, and whether the keyword channel tolerates structure errors better than the embedding does.
Core Claim
We make two claims. First, in a hybrid retriever, routing inherited document context (ancestor headers and salient terms of the parent section) into a separately weighted keyword field, while embedding only the local chunk, retrieves evidence better than prepending the same context to both channels. The gain should grow as chunks get shorter relative to their context and on questions whose evidence spans several chunks. Second, this asymmetric design is more robust to errors in the recovered hierarchy. That robustness moves the best operating point on the ingestion-cost versus retrieval-quality curve toward cheap structure parsers.
The mechanism comes from Sparse, Dense, and Attentional Representations for Text Retrieval. That paper shows that a fixed-size dense vector keeps its margin between relevant and distractor passages only if its dimension grows with text length. Prepending headers lengthens the text, which is exactly the case that paper warns about. Its ME-HYBRID results also show the keyword side adds value even on top of multi-vector dense encoders. Using the result of Ji Ma et al. that lexical scores can be written as sparse dot products, everything fits in one index. Each chunk $c$ gets the concatenated vector $[\mathbf{d}(c_{\text{local}}) \,\|\, \lambda_l\,\mathbf{s}(c_{\text{local}}) \,\|\, \lambda_c\,\mathbf{s}(c_{\text{ctx}})]$. Here $\mathbf{d}$ is the dense encoder and $\mathbf{s}$ is the sparse lexical vector. $c_{\text{local}}$ is the chunk text, and $c_{\text{ctx}}$ is its inherited context. The weights $\lambda_l$ and $\lambda_c$ set the importance of local and context keyword matches. A query $q$ is scored as
$$\text{score}(q,c) = \langle \mathbf{d}(q), \mathbf{d}(c_{\text{local}})\rangle + \lambda_l\,\text{BM25}(q, c_{\text{local}}) + \lambda_c\,\text{BM25}(q, c_{\text{ctx}}),$$
which is one nearest-neighbour search with no separate context pipeline.
Relation to Prior Work
Two systems show that context injection helps. ChunkNorris recovers a table-of-contents tree from PDFs with geometric rules and prepends ancestor titles to each chunk. Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations prepends a fixed 128-word header to every dialogue chunk. Neither separates the context by scoring channel. ChunkNorris evaluates only dense retrievers. Our proposal is the missing factorial comparison: context sent to the dense side, the sparse side, both, or neither.
HiChunk takes a different route. It predicts multi-level hierarchies with Qwen3-4B, then merges retrieved children back into their parent section at query time. That is a query-time, dense-only use of structure. We use the same hierarchy at indexing time, in the keyword channel, and we reuse its HiCBench annotations as our gold trees. Mix-of-Granularity and MoC make chunk size adaptive, at the cost of routing latency or 1.5B-parameter segmentation models. We do not change chunk sizes. We change where context is scored.
Two further papers show the sparse channel can carry information the dense channel loses. DENSPI pairs dense phrase vectors with sparse TF-IDF for lexical filtering. Xueguang Ma et al. show BM25 recovers what aggressive dense compression throws away, reaching 16× compression with a 0.2% accuracy gain over uncompressed DPR-768. Neither studies document structure. TreeRare is the closest paper that uses parse trees for retrieval, but it parses only the query and leaves the corpus flat. We parse the corpus's document structure and leave queries alone.
The parsing literature contributes methods rather than motivation. Gómez-Rodríguez and Vilares encode a constituency tree as one label per word boundary: the change in shared-ancestor depth plus a category. This runs at up to 957 sentences per second. We apply the same encoding at line granularity, tagging each document line with its relative change in heading depth. That turns hierarchy recovery into a single encoder pass, and clipping the depth changes always yields a well-formed tree. Vine Pruning trains its cheap first pass with a loss that penalizes only discarding true structure. AdaParse routes PDFs to a heavy parser only when needed, gaining 17× throughput. Following both, our cascade escalates a document to the LLM structurer only when the cheap tagger is unsure. The router is trained with an asymmetric loss that penalizes wrongly accepted structure far more than unnecessary escalations.
Research Program
We will build one hybrid index with three structure tiers. The first tier is ChunkNorris-style heuristics. The second is the line-level depth tagger: a small encoder trained on HiCBench hierarchies plus silver trees from heuristics run on born-digital documents. The third is HiChunk's LLM structurer. Each tier produces a tree. The tree defines chunk boundaries and each chunk's context string: ancestor titles, plus the top TF-IDF terms of the parent section, since titles alone are often uninformative.
For every tier, we compare four placements of context: none, dense only, sparse only, and both. The dense encoder and BM25 implementation are held fixed across all conditions. We also run a multi-vector dense variant in the ME-BERT style to test whether extra dense capacity closes the gap.
To separate the effect of structure accuracy from the effect of the specific parser, we corrupt the gold HiCBench trees in controlled ways. We drop headers, assign wrong heading levels, and attach chunks to sibling sections. This traces retrieval quality as a function of tree F1 for each placement. The three real parsers are then placed on that curve.
Finally, we sweep chunk length at fixed context length. This directly tests the length-capacity prediction: the dense-only placement should degrade relative to sparse-only as the ratio of context to chunk length rises.
Baselines
The naive reference is flat fixed-length chunking with no structure and standard BM25-plus-dense fusion. Fusion weights are tuned on held-out questions. This tells us whether structure matters at all on these documents.
The main practical competitor is the approach used in practice today: ChunkNorris-style prepending of ancestor titles to the chunk text, indexed identically by BM25 and the dense encoder, with fusion weights tuned. It is the strongest competitor because it costs nothing extra and is what practitioners already do. The asymmetric design must beat it with the same trees, encoder, chunk boundaries, and fusion-weight search budget. We allocate an equal grid over $(\lambda_l, \lambda_c)$ versus $(\alpha, \beta)$, with the asymmetric condition receiving no more tuning points than the symmetric one.
The strongest existing structure-aware method is HiChunk with Auto-Merge (the HC200+AM configuration) on HiCBench. HiChunk reports that this configuration gave the highest Evidence Recall and Fact-Coverage, especially on the multi-chunk tasks $T_1$ and $T_2$. However, no reported numeric evidence-recall value was found on Lacuna for HiChunk on HiCBench. We will therefore rerun it with its released structurer, on our encoder, as the comparison point. We also combine it with our method, since Auto-Merge operates at query time and is compatible with context in the keyword channel.
For the hybrid implementation itself, the sanity anchor is ME-HYBRID's MRR@10 of 0.343 on MS MARCO Passage, reported by Luan et al.. Our fusion code should approach that level before we trust it on structured documents. No reported numeric retrieval result was found on Lacuna for ChunkNorris on PIRE, because the values appear only in figures. We will rerun it.
The key ablations are four. Context in the sparse field only, with the dense side untouched, isolates the placement effect. Titles-only versus titles-plus-section-terms isolates the content of the context. A separate BM25 field versus the same terms appended to the local BM25 text shows whether separate weighting matters. Gold versus predicted trees measures parser sensitivity.
Beating the practical baseline means a gain on $T_1$/$T_2$ evidence recall whose paired bootstrap interval excludes zero, with no significant loss on $T_0$. Beating HiChunk on cost means reaching its retrieval quality within its bootstrap interval while calling the LLM structurer on a minority of documents.
Why not just…?
Why not just prepend headers to the text and let fusion weights sort it out? This is the real threat. Well-tuned fusion weights might make symmetric prepending as good as asymmetric placement. The four-way placement comparison under an equal tuning budget settles this directly. If symmetric prepending matches asymmetric placement across the chunk-length sweep, the length-capacity mechanism does not bite at realistic header lengths, and we will report that as the finding. We will not claim a gain that is not there. The prediction that distinguishes the two hypotheses is an interaction: the asymmetric advantage should widen as chunks shrink. A constant gap, or no gap, would count against our explanation.
Why not just use a multi-vector dense encoder, which has capacity to spare? Multi-vector encoding does relieve the length bottleneck. But Luan et al. report a 3× to 8× larger index and still find a gain from adding sparse retrieval (0.334 to 0.343 MRR@10). We include the multi-vector variant with context in both channels. If it matches our single-vector asymmetric index, the useful conclusion is that our design reaches the same quality at a fraction of the memory. We will report index size next to recall.
Why not just run the LLM structurer on everything, or skip structure and use semantic chunking? Semantic chunking is already weak on these documents. HiChunk reports a hierarchy F1 of 0.0616 for semantic chunking on Gov-Report versus 0.9882 for its structurer. MoC finds semantic chunks entangled with their neighbours. Running the LLM structurer everywhere is accurate but costs about 1.5 s per document, against about 105 ms per page for heuristics. The noise-injection curve shows whether that accuracy translates into retrieval gains once context is in the tolerant keyword channel. If the curve is flat above the heuristic's tree F1, the cascade is unnecessary and the cheapest tier wins. That would be a clean result in its own right.
Proposed Experiments
The primary benchmark is HiCBench. It supplies human-annotated hierarchies for gold trees and noise injection, and evidence-recall and fact-coverage metrics stratified by whether evidence sits in one chunk or several. PIRE is the second benchmark: 100 PDFs with 300 single-chunk and 32 multi-chunk questions, where we report Recall@10 and NDCG@10. It is the natural testbed for the PDF heuristic tier and for per-page ingestion energy. Qasper supplies scientific papers with real section structure and a runtime reference for structuring cost.
For each condition we report retrieval quality, index size, and ingestion seconds and energy per document. Results are plotted as a quality-versus-cost frontier across the three structure tiers and the cascade.
Three results would be decisive. First, sparse-only context must beat symmetric prepending on multi-chunk questions, with the gap widening as chunks get shorter. Second, under noise injection, the sparse-only curve must stay flatter than the dense-only curve. Third, the cascade must reach LLM-tier retrieval while escalating only a minority of documents.
If the first result fails, the placement claim is wrong. If the second fails, cheap parsers lose their advantage. If both hold, the practical recommendation is simple: for hybrid indices over structured documents, embed the local passage, send inherited context to a separately weighted keyword field, and parse structure cheaply.
References
- Sparse, Dense, and Attentional Representations for Text Retrieval
- Hybrid First-stage Retrieval Models for Biomedical Literature
- ChunkNorris: A High-Performance and Low-Energy Approach to PDF Parsing and Chunking
- Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
- HiChunk: Evaluating and Enhancing Retrieval Augmented Generation with Hierarchical Chunking
- Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation
- MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System
- Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index
- Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval
- TreeRare: Syntax Tree-Guided Retrieval and Reasoning for Knowledge-Intensive Question Answering
- Constituent Parsing as Sequence Labeling
- Vine Pruning for Efficient Multi-Pass Dependency Parsing
- AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling Engine