Hidden Loss Weights in Hierarchical Document Embeddings

Long documents have structure. Words form passages, and passages form documents. One natural embedding model would therefore produce one vector per passage and one vector for the whole document, and train both with a contrastive loss, meaning a loss that pulls matching items together and pushes mismatched ones apart. At search time both vectors are compared by cosine similarity, the angle between them. Practitioners who train such models find the result very sensitive to how the passage loss and the document loss are weighted. They either spend many runs searching for weights or cut gradient flow between the levels entirely. This note argues that much of that sensitivity comes from two controls nobody sets on purpose. The first is the length of each vector before it is normalized to unit length, which silently rescales that level's gradient. The second is the backward path through which each level's loss reaches the other level's representation.

The proposal is to pin the first control and make the second one graded and measured. Concretely, we fix each level's pre-normalization vector length so that the nominal loss weights mean what they say. We then replace the all-or-nothing "stop-gradient" (an operator that passes values forward but blocks gradients backward) with a partial gradient valve between levels. The valve opens when the two levels' gradients agree and closes when they conflict. If the diagnosis is right, performance on document-aware passage retrieval and scientific-document benchmarks should become nearly flat across a wide range of loss weights. It should also match or exceed tuned multi-task balancing methods at lower cost. If the diagnosis is wrong, the experiments will show that plainly as well.

Core Claim

Suppose an embedding $e_\ell = z_\ell/\|z_\ell\|$ at hierarchy level $\ell$ is scored by cosine similarity, and receives upstream gradient $g_\ell = \partial \mathcal{L}_\ell/\partial e_\ell$ from its loss $\mathcal{L}_\ell$. Then the gradient that reaches the unnormalized vector $z_\ell$ is

$$\frac{\partial \mathcal{L}_\ell}{\partial z_\ell} = \frac{1}{\|z_\ell\|}\left(I - e_\ell e_\ell^\top\right) g_\ell .$$

This is the standard derivative of vector normalization. It means that a loss weighted by $w_\ell$ acts on the shared encoder with effective weight roughly $w_\ell/\|z_\ell\|$. In a hierarchical encoder, a document anchor that summarizes thousands of tokens and a passage anchor that summarizes fifty have no reason to keep equal lengths $\|z_\ell\|$ during training. The effective balance between levels therefore drifts even when the nominal weights are fixed.

Our first hypothesis is that this drift explains most of the weight sensitivity. Our second hypothesis concerns what remains after the drift is removed. The remaining interaction between levels is a real but variable conflict, which is sometimes helpful and sometimes harmful. A continuous valve steered by measured gradient agreement handles it better than either full gradient flow or a hard stop-gradient.

The file supplies the motivation from three directions. Metric mismatch between training and inference is known to cost accuracy. ReMP reports up to 5% degradation when training geometry and inference geometry disagree. Content-based audio representations with Siamese networks finds retrieval peaks only when the inference similarity matches the training loss. Angular Triplet-Center Loss shows Euclidean objectives leave embeddings poorly arranged for angular search. All three study flat, single-vector models. None of them asks what normalization does to the balance between several objectives, which is our question.

Research Program

The backbone is the Hierarchical Document Transformer. HDT inserts sentence, section, and document "anchor" tokens and restricts attention to siblings and parent/child anchors. It reports 5,909 MB of memory at 32k tokens versus 21,259 MB for dense attention, and convergence in 10k pretraining iterations versus 64k for Longformer. HDT supplies the method here, but it trains with masked language modeling and lets all gradients flow freely across levels. We keep its architecture and change only the training objectives and the backward pass. We will use a released checkpoint if one exists. Otherwise we will pretrain following its recipe.

On top of HDT we attach two cosine-scored contrastive losses. A passage-level loss uses section/sentence anchors and query–passage pairs. A document-level loss uses the document anchor and citation-proximity pairs in the style of SPECTER. For the similarity inside the loss we test two options. One is standard cosine InfoNCE. The other is the arccosine logit of SimACE, which replaces cosine with an angle function so that gradients stay large for near-duplicate positives, and which improved average STS from 76.25 to 78.20 on BERT-base. SimACE is a flat sentence encoder. We reuse its loss, and our contribution does not depend on which of the two wins.

The first intervention, norm pinning, removes the hidden weight. It places a LayerNorm with frozen unit gain and no bias before each anchor's L2 normalization. This forces $\|z_\ell\|$ to a constant $\sqrt{d}$, so the effective weight equals the nominal weight up to a shared constant. The fix is deliberately the simplest one possible. The research claim is not the trick itself but the attribution: that this trick, and not a more elaborate balancing scheme, is what flattens the sensitivity curve.

The second intervention is a graded cross-level valve. It generalizes the stop-gradient that DELG places between local and global heads, and that Stop-Gradient Softmax Loss places on class proxies. The valve sits where the document anchor reads the passage anchor states $h$. At that point we replace $h$ by

$$\tilde h = \lambda_t\, h + (1-\lambda_t)\,\mathrm{sg}[h],$$

where $\mathrm{sg}[\cdot]$ is stop-gradient. The forward value is unchanged ($\tilde h = h$), and only a fraction $\lambda_t\in[0,1]$ of the document loss's gradient flows back into the passage states. The coefficient $\lambda_t$ is an exponential moving average of $\max(0,\cos(\partial\mathcal{L}_p/\partial h,\ \partial\mathcal{L}_d/\partial \tilde h))$, where $\mathcal{L}_p$ and $\mathcal{L}_d$ are the passage and document losses. Both gradients are taken with respect to the anchor activations only, so the cost is two backward passes through the small head, not through the whole network. A symmetric valve governs the passage loss's path into the document anchor. When $\lambda_t$ is pinned to 0, the scheme reduces to DELG-style isolation. When it is pinned to 1, it reduces to plain HDT end-to-end training.

The analysis component logs, per level and per step, $\|z_\ell\|$, the effective weight $w_\ell/\|z_\ell\|$, the gradient cosine, and $\lambda_t$. The question is whether final quality is better predicted by nominal or by effective weights.

Baselines

The naive reference point is HDT trained end to end with both cosine contrastive losses at equal nominal weight. It has no norm pinning and no valve. All other methods are measured against it.

The strongest external reference on scientific documents is SPECTER2. It uses task-format control codes and per-task adapters on a frozen base, and reports an average score of 71.2 on the 24-task SciRepEval benchmark, against 67.5 for SPECTER and 68.8 for SciNCL. It is the strongest competitor because it already handles negative transfer between tasks, through parameter isolation instead of gradient control. SPECTER2 reads titles and abstracts. For a fair comparison we therefore evaluate our model both on titles and abstracts and on full text. Beating it means exceeding 71.2 on the average under the title-and-abstract setting, or showing that full-text hierarchy adds points on top of our own title-and-abstract score. HDT-E is reported competitive on SciRepEval proximity tasks, but no reported result was found on Lacuna for an exact HDT-E score. We will rerun it ourselves.

For document-aware passage retrieval, the reference is DRAGON+ on DAPR. DAPR reports 0.0 nDCG@10 for DRAGON+ on the NQ-hard stress test, attributes 53.5% of errors in top retrievers to missing document context, and reports that prepending titles helps on Wikipedia but hurts on Genomics. We run DRAGON+ with and without title prepending on every DAPR subset. Beating it means gains on NQ-hard without the Genomics regression that title prepending shows.

The strongest methods on the actual research question are GradNorm and PCGrad. GradNorm rescales task weights so that per-task gradient norms track relative training rates. PCGrad projects away the conflicting component of one task's gradient onto the other's normal plane. Their reported gains come from other domains: GradNorm reports about 5% lower depth error on NYUv2 than equal weighting, matching a 100-run grid search. PCGrad reports CIFAR-100 routing accuracy rising from 67.7% to 77.5%. No reported result was found on Lacuna for either method on hierarchical text encoders.

Both are applied to the same HDT model, data, and number of training steps. GradNorm receives its asymmetry parameter over its stated range 1.0 to 1.5, and a choice among three shared reference layers. PCGrad uses full per-task backward passes. Each method, including ours, gets the same tuning budget of runs. We also report wall-clock and memory, since PCGrad pays roughly one extra full backward pass per task.

The key-ingredient ablations are as follows. Norm pinning is tested without the valve. The valve is tested without norm pinning. The valve is also tested with fixed $\lambda\in\{0, 0.5, 1\}$. Finally, a valve driven by random $\lambda$ with the same mean as the learned schedule separates "agreement-driven" from "merely partial".

Why not just…?

Why not just grid-search static loss weights? With two levels, a grid search is affordable, and this is the alternative most likely to be good enough. Our claim is not that grid search fails to find a good point. The claim is that the good point moves, because the effective weight $w_\ell/\|z_\ell\|$ drifts as anchor lengths change over training and across datasets. The weight sweep below settles this directly. If the best static ratio transfers unchanged across DAPR subsets and SciRepEval, and unpinned models are as flat as pinned ones, the norm diagnosis is wrong and we will say so. GradNorm's finding that one adaptive run beat 100 grid runs suggests static weights are brittle, but it was measured on NYUv2, not on text hierarchies.

Why not just use GradNorm? This is the real threat. GradNorm measures gradient norms at a shared layer, and that measurement already includes the $1/\|z_\ell\|$ factor, so it could compensate for norm drift automatically. If GradNorm matches pinning plus valve at equal tuning budget, our method contributes a cheaper and parameter-free route plus an explanation, not a better optimizer. We will report that honestly. We expect a difference for a different reason. GradNorm can rescale magnitudes but cannot remove directional conflict. The valve targets exactly the directional conflict, and the comparison with the valve ablations isolates it.

Why not just use a hard stop-gradient like DELG, or PCGrad? DELG's hard stop-gradient works when the local head clearly harms the global backbone. DAPR's finding that most passage errors come from missing document context suggests that document-level signal should shape passage representations at least some of the time. The $\lambda=0$ ablation is DELG's choice, and the $\lambda=1$ ablation is HDT's choice. If either fixed endpoint matches the adaptive valve, the gradient-agreement signal is not earning its place. PCGrad also resolves directional conflict. It does so in parameter space with full per-task backward passes, while our valve acts on one activation edge of the hierarchy. The head-to-head comparison at matched compute decides whether that locality costs accuracy.

Why not just use a flat long-context model, or train separate models and merge them? Flat windows do not fix long-range synthesis: on SCROLLS, LED with 16k tokens scored 29.16 against BART's 29.01 with 1k tokens. Model merging avoids joint-training conflict, but passages then cannot attend to document context during encoding. That context is precisely what DAPR says is missing. We include a Longformer-based flat encoder trained with the same contrastive loss as a reference. We do not treat merging as a main baseline.

Proposed Experiments

The central experiment is a sensitivity sweep. We train seven passage-to-document weight ratios, log-spaced from 1/27 to 27, for each of four conditions: naive, norm pinning alone, valve alone, and both. Each run uses three seeds, with cosine InfoNCE and the SimACE loss as a secondary factor. We evaluate every run on the DAPR subsets (nDCG@10, with NQ-hard and Genomics reported separately) and on the SciRepEval average and its four format groups. Sensitivity is summarized by the spread of each metric across the sweep and by the area under the passage/document trade-off curve.

The first decisive check tests the norm hypothesis. In unpinned runs, final performance plotted against the time-averaged effective weight $w_\ell/\|z_\ell\|$ should collapse onto a single curve, which plotting against nominal $w_\ell$ does not do. Pinned runs should also show a markedly smaller spread than naive runs. If neither happens, the hidden-weight hypothesis is rejected.

The second decisive check tests the valve. Under norm pinning, the agreement-driven valve must beat all three fixed-$\lambda$ endpoints and the random-$\lambda$ control. Its logged $\lambda_t$ should vary meaningfully over training rather than sit at 0 or 1. Where the valve stays nearly closed, we expect passage retrieval on NQ-hard to lose less than the full-flow model on the Genomics subset. That would indicate the valve blocks document-level noise where title prepending fails.

The third check is the external comparison. At matched data, steps, and tuning budget, pinning plus valve should match or beat GradNorm and PCGrad on the DAPR and SciRepEval averages at lower overhead, and should approach or exceed SPECTER2's 71.2 in the title-and-abstract setting. A tie with GradNorm is an informative outcome. It would reframe the contribution as a zero-parameter explanation of why balancing is needed at all.