Anytime Document Structuring While the Model Is Still Loading

Before a long document can be searched or questioned by a language model, it has to be cut into pieces. The best current cutters run a fine-tuned language model over the text to find section and subsection boundaries, which works far better than slicing every few hundred tokens. These cutters are usually measured warm, with the model already sitting in GPU memory. Real ingestion is often cold. A clinician uploads one molecular pathology report and wants an answer immediately, so the segmentation model must first be read off disk. For a 4-billion-parameter model that read takes seconds, longer than the segmentation itself.

This note proposes running the structuring model layer by layer as its weights arrive. Small boundary-prediction heads are attached at intermediate depths, so a provisional segmentation is available after only the first few layers have loaded. The provisional boundaries also decide where the deeper layers look, which replaces wasteful 50%-overlap sliding windows with windows aligned to likely sections. The bet is that most section boundaries can be decided from shallow features such as headings, enumeration and paragraph breaks, and that only a minority of ambiguous boundaries need the full depth. If so, time-to-usable-segmentation from a cold start drops from "load everything, then infer" to roughly "load a fraction, infer, refine."

Core Claim

Structural boundary detection is an unusually good fit for depth-progressive inference, for three reasons. Its output is a sparse set of discrete cut points. Its strongest cues are lexical and typographic, and Shi et al. show that explicit end-of-paragraph tokens alone teach a language model to recognize thematic transitions. Its errors are cheap to revise downstream. We claim three things. Boundary heads at the first quarter to third of a fine-tuned structuring model's layers recover most of the full model's boundary accuracy on well-formatted documents. Pipelining layer loading with layer execution, so that layer $k$ runs over the whole document while layer $k+1$ is still being read, yields an anytime segmenter whose quality rises monotonically during cold start. Using shallow boundaries to place the deep model's context windows keeps cross-boundary context at lower cost than uniform overlap. The claim should fail, in a measurable way, on degraded OCR and unformatted text dumps, where boundaries are semantic rather than typographic. Characterizing that failure is part of the contribution.

Where This Sits

Two systems define the structuring side. HiChunk fine-tunes Qwen3-4B to predict section boundaries over overlapping windows and merges retrieved child chunks into parent sections. It reports a structural $F1_{Lall}$ of 0.9882 on Gov-Report and about 1.5 seconds per Qasper document, against 5.5 seconds for LumberChunker. All of these are warm numbers. LongRefiner reduces indexing cost by generating only structural tags and replacing paragraph bodies with skip tokens, which cuts indexing tokens by about 90%. Both shrink how much the model must generate. Neither touches the cost of getting the model ready to run. Our proposal reuses HiChunk's boundary labels and model scale and attacks that second cost.

The systems literature motivates the gap and supplies the pipelining idea. Yi et al. show that cold inference on edge devices is 1.5× to 443× slower than warm inference, dominated by loading and weight transformation. Their NNV12 overlaps the read-and-transform of layer $N+1$ with execution of layer $N$. We borrow that overlap but add something NNV12 lacks: useful output before the last layer arrives. FlowLoader speeds loading 2.8× to 10× over PyTorch and ServerlessLLM by saturating idle interconnects and caching across tiers, and it treats the model as opaque bytes. WarmServe avoids cold starts by prewarming models predicted from seasonal traffic. The file notes that this degrades under bursty, out-of-pattern requests, which is exactly the sporadic single-document upload regime. These loaders are complementary to our method rather than competitors, and we will stack our method on top of the fastest one. NNsight/NDIF report that above roughly 3B parameters, moving weights to the GPU exceeds a forward pass. This is why we target a 4B structuring model rather than a toy.

On context preservation, InstructCoder finds 50% window overlap the best trade-off for fixed-length windows, which doubles compute. MuLD shows how much naive chunking costs: chunked T5 scores 54.01 F1 against Longformer's 82.58 on character archetype classification. We keep overlap, but put it only around uncertain boundaries.

The application regime is clinical fusion reporting. RNA-sequencing-based gene fusion detection produces documents that oncologists must interrogate quickly. Examples are assay validation reports like Tempus xT, guideline consensus such as the NRG1/2 fusion consensus, case reports like the ROS1/entrectinib case, and method papers whose answers hinge on structure. The reading-frame and domain-retention evidence that Pegasus and FuGePrior compute lives in specific results sections and supplementary tables. These documents arrive one at a time, often as degraded PDFs, and with no seasonal pattern. The fusion direction is therefore the deployment regime and stress test, not the mechanism. We do not claim anything about fusion calling itself.

Research Program

We first reproduce a HiChunk-style boundary model by fine-tuning Qwen3-4B on HiChunk's sentence-level section and subsection labels, so that every variant shares one backbone. We then attach linear boundary heads to the residual stream at evenly spaced depths. These heads are trained with the backbone frozen, so the full-depth model is unchanged and any gain cannot come from retraining it. Each head outputs, per sentence, a boundary probability for each hierarchy level. Its confidence is calibrated on held-out data.

The streaming executor loads weights from SSD in layer order. Whenever layer $k$ becomes resident, it runs over the cached hidden states of every window in the document, and whenever a head exists at depth $k$, it emits a revised segmentation. Hidden states for a document of tens of thousands of tokens fit easily in GPU memory, so no recomputation is needed.

After the first head fires, windows are re-anchored to provisional section starts. Overlap is kept only around boundaries whose calibrated confidence falls below a threshold $\tau$. As a result, the deep layers see section-aligned context rather than arbitrary 50%-overlap slices. The quantity we optimize is the area under an anytime curve: boundary quality, or downstream retrieval quality, as a function of wall-clock time since the request arrived with weights on disk.

Two analyses accompany the system. The first is a per-boundary "decision depth" study, recording the shallowest head that agrees with the final model. It is broken down by boundary type (explicit heading, numbered list, implicit topic shift) and by document condition (clean markup, PDF text extraction, OCR-degraded). The second is a cost accounting of redundant tokens processed under uniform overlap versus boundary-aligned windows.

Baselines

The naive reference is fixed-length chunking at 512 tokens with 50% overlap, following InstructCoder's recommended rate. It needs no model load at all, so it sets the floor for both quality and latency. Beside it sits HiChunk's semantic-chunking comparison point, which HiChunk reports at 0.0616 $F1_{Lall}$ on Gov-Report. That figure shows how far "cheap and immediate" falls behind on structure.

The strongest existing method is cold-start HiChunk. This is our reproduction of the fine-tuned Qwen3-4B structurer, loaded fully and then run, with weights read through the fastest loader we can deploy. We use FlowLoader-style multi-channel loading if available and plain safetensors otherwise, and report both. Its reported warm quality is 0.9882 $F1_{Lall}$ on Gov-Report and its reported warm latency about 1.5 s per Qasper document. No reported cold-start figure for HiChunk or LongRefiner was found on Lacuna. We will measure it on identical hardware, storage and documents. Our method beats it if it reaches within 0.01 $F1_{Lall}$ of the full model's Gov-Report score in substantially less cold wall-clock time (we will report the ratio). Because our final layer reproduces the full model exactly, end-state accuracy can never be lower.

The second strong competitor is a two-model cascade. A small model, fine-tuned on the same labels and comparable in parameter count to the shallow prefix of our model, runs first. The 4B model is loaded in the background and refines only low-confidence boundaries. This competitor gets the same labels, the same number of tuning runs, and the same storage path.

Three ablations isolate the key ingredients. Early-exit heads without streaming tests whether the gain comes from overlap with loading. Streaming with uniform 50% overlap instead of boundary-aligned windows tests the context-placement contribution. Heads only at the final quarter tests whether shallow depth genuinely carries the signal.

Why not just…?

Why not just keep the model warm or load it faster? Persistent instances and prewarming work when traffic is predictable or volume justifies a dedicated GPU. WarmServe's own reliance on seasonal forecasting, and the switching-cost problem Niyato et al. describe for edge caching, mark sporadic uploads as the case where neither holds. Faster loaders like FlowLoader shrink the load term but do not remove it. Ours composes with them: we report results stacked on the fastest loader. If fast loading makes cold time negligible relative to inference on our hardware, the anytime curve flattens and we will say so.

Why not just parse headings with rules? On clean Markdown or HTML, a heading parser is nearly free and probably nearly as good, and we expect our shallow heads to largely rediscover it there. This is a genuine threat on clean data, and the rule parser is included as a baseline on every split. The case for a learned model rests on PDF-extracted and OCR-degraded documents, where the file identifies missing structure as the open problem. If rules match the shallow heads on the degraded clinical PDFs as well, the shallow-exit contribution reduces to "use rules first," and the paper reports that.

Why not just use a small distilled segmenter? This is the most serious alternative, and it is why the cascade is a full baseline rather than a footnote. A small model loads fast and may match shallow heads. The argument for our approach is that the shallow layers are part of the large model, so no bytes are loaded twice and refinement is continuous rather than a second cold start. Whether that argument matters empirically is settled by comparing anytime curves under equal storage bandwidth. If the cascade's curve dominates ours, we will recommend the cascade.

Proposed Experiments

Boundary quality is measured on Gov-Report with HiChunk's $F1_{Lall}$, and on Qasper for latency, reporting the full anytime curve from request arrival with weights on SSD. Downstream effect is measured on HiCBench's sparse, single-chunk-dense and multi-chunk-dense tasks. We report evidence recall and fact coverage when retrieval runs on whatever segmentation exists at fixed time budgets. HiChunk reports HC200 with Auto-Merge as best on these, but the file gives no numeric values, so no reported number was found on Lacuna for HiCBench evidence recall. We will re-run that configuration ourselves. Robustness is tested by re-rendering Gov-Report and Qasper documents to PDF and to synthetically degraded OCR, then re-extracting the text.

For the clinical regime, we build a small annotated set from fusion-related documents of the kinds listed above, with hand-labeled section hierarchy. Questions target fusion partners, reading-frame status, retained kinase domains, and recommended inhibitors. Requests arrive under a bursty arrival process. This set is small and reported as a secondary, regime-specific check, not a leaderboard.

The decisive checks are three. The first is whether a head at roughly a third of the network's depth reaches within 0.01 $F1_{Lall}$ of the final layer on clean Gov-Report. If it needs most of the depth, the core premise is false. The second is whether boundary-aligned windows match uniform 50% overlap on HiCBench while processing measurably fewer redundant tokens. The third is whether, on degraded PDFs, the gap between shallow and deep heads widens as predicted, which would show that the method knows when to wait.