A common recipe for imbalanced classification is to train in two stages. First, train the network normally so it learns a representation of the input. Then freeze that representation and re-tune only the final classifier so that rare classes get a fair share of predictions. The standard explanation for why this works is that imbalance damages the classifier, not the features. This note turns that explanation into something that can be measured on clinical text, where a single note must be split into dozens of mutually exclusive section types, most of them rare. The idea is to ask, for a given frozen encoder, which combinations of per-class accuracy any re-tuned classifier on top of it could achieve. That set of combinations is the encoder's attainable set. Two encoders can then be compared by whether one attainable set contains the other, not by a single averaged score.
The obstacle is dimensionality. With 50 section classes the attainable set lives in a 50-dimensional space. The standard measure of such sets, the hypervolume (the volume of objective space dominated by the set, above a reference point), cannot be computed exactly past roughly five objectives. The key observation is that freezing the encoder makes a different route cheap. Hypervolume can be written as an average over directions of how far the set reaches along each direction. Along any one direction, "how far can a re-tuned classifier reach" is a small calibration problem on cached features that takes seconds. Two-stage training is therefore what makes the high-dimensional geometry measurable. In turn, the geometry gives a sharper test of the claim that motivates two-stage training.
Core Claim
For a frozen encoder $A$, let $\theta$ be the classifier-calibration parameters (per-class logit offsets and weight scales, the family used by learnable weight scaling and $\tau$-normalization). Let $s_i(\theta)$ be the held-out F1 of class $i$, and let $r$ be a reference vector of minimally acceptable scores. For a direction $\lambda$ on the positive unit sphere, define the attainment of $A$ along $\lambda$ as
$$ t_A(\lambda) = \max_{\theta} \; \min_{i} \; \frac{s_i(\theta) - r_i}{\lambda_i}, $$
which is how far along $\lambda$ the best calibrated classifier on $A$ can push all classes simultaneously. Two facts make this useful. First, $A$'s attainable set contains $B$'s exactly when $t_A(\lambda) \ge t_B(\lambda)$ for every $\lambda$, because downward-closed sets above $r$ are determined by their radial reach. Second, the hypervolume of the attainable set equals a dimension-dependent constant times the average of $t_A(\lambda)^M$ over uniformly drawn directions, where $M$ is the number of classes. This identity underlies the scalarizing-function approximation of hypervolume.
The proposal makes two claims. The first is that attainment profiles computed this way give a dimension-free, directly interpretable comparison of clinical text encoders. The second is that on section identification they will show whether the "features are fine, the classifier is biased" account holds for clinical text or whether the rare sections are limited by the encoder itself.
Where This Sits
The mechanism comes from two lines of work. Decoupling Representation and Classifier for Long-Tailed Recognition introduced classifier re-training, $\tau$-normalization, learnable weight scaling and nearest-class-mean heads on frozen features. It found that instance-balanced feature learning plus head re-tuning beats joint rebalancing, with learnable weight scaling reaching 49.9% overall accuracy on ImageNet-LT against 37.7% for OLTR. That paper supplies our calibration family and the hypothesis we test. Its evidence consists of single accuracy numbers per method, which cannot tell "this encoder reaches a better trade-off" apart from "this head picked a better point on the same trade-off." Havaei et al. motivates the same split for medical segmentation, training features on balanced data and then refitting the decision layer on the true distribution. Yu et al. constrains the story: contrastive features can be misaligned with categorical targets. This is exactly the case where no head re-tuning helps, and where an attainment profile should visibly fall short.
The directional estimator comes from Proposal of Approximating Hypervolume by Scalarizing Functions for Evolutionary Many-Objective Optimization and its follow-up, which approximate hypervolume of a finite population by casting rays from the reference point. We apply the same rays to a continuous attainable set rather than a finite point cloud. Each ray becomes an optimization rather than a lookup. The file notes that ray counts needed for accurate hypervolume grow badly with dimension. That limitation is a constraint we take seriously: we treat the full profile and the containment test as the primary object, and treat 50-dimensional hypervolume as a secondary scalar whose variance we report. For exact ground truth in low dimension we use the box-decomposition algorithm of Efficient Computation of Expected Hypervolume Improvement Using Box Decomposition Algorithms, which is fast for up to about five objectives. Hypervolume-driven Bayesian optimization methods such as qNEHVI and rank-based alternatives such as BOtied solve a different problem: choosing expensive black-box evaluations. Here each evaluation is cheap, and the question is the shape of a set, not where to sample next.
The clinical side supplies the regime. MedSecId labels 2,002 MIMIC-III notes with 50 section classes. Its token-level BiLSTM-CRF reaches micro F1 0.955 but macro F1 0.787, with common sections above 0.99 and rare ones far lower. That gap between micro and macro F1 is what a long tail looks like. Crucially, sections are mutually exclusive per token, so raising one class's logit offset steals predictions from others, and the 50 objectives genuinely conflict. CheXbert is itself a two-stage pipeline: weak supervision from a rule labeler, then fine-tuning on about 1,000 expert-labeled reports. It reaches 0.798 average F1 against a 0.805 radiologist benchmark, with 14 independent observation heads. Its four-state labels (positive, negative, uncertain, blank) come from CheXpert. They are mutually exclusive within an observation and independent across observations, which gives us a natural control case described below. No work in the file measures attainable trade-off sets for clinical extraction models. Existing clinical evaluations report micro or macro F1, and macro F1 is one fixed linear weighting of the 50 objectives.
Research Program
The first component is the estimator. Backbone features for the training, validation and test notes are cached once. For each direction $\lambda$, the max-min problem is solved by bisection on the attainment level. At a candidate level $t$, we search for per-class offsets and scales that make every class clear its target $r_i + t\lambda_i$. Because F1 is not differentiable, feasibility is checked with coordinate search over offsets on validation counts. A smooth surrogate (soft-count F1) provides a warm start. Calibration uses one validation split. The reported attainment is the held-out test score of the calibrated head along that direction, so optimistic bias from tuning on validation does not inflate the profile. Directions are shared across encoders, so profiles can be compared direction by direction. Confidence intervals come from bootstrapping over notes, since rare-section F1 on a few hundred tokens is noisy.
The second component is validation of the estimator where exact answers exist. We merge the 50 sections into three to five groups: frequent, medium and rare, split further by note type. On these grouped objectives we densely sample calibration parameters, compute exact hypervolume of the resulting points with the box-decomposition algorithm, and compare against the directional estimate. This tells us how many directions are needed at each dimension and whether the estimator's variance is usable at $M=50$.
The third component is the scientific question. On MedSecId we train encoders under several regimes. These are the paper's own BiLSTM-CRF; a clinical BERT-style encoder with instance-balanced sampling; the same encoder with class-balanced sampling; the same encoder trained jointly with a focal loss; and a two-stage weak-then-expert variant in the style of CheXbert, with regex section labels as weak supervision. The long-tail result for images predicts that instance-balanced features have the largest attainable set, and that class-balanced and joint-focal training produce profiles that are dominated or cross. If the rare-section profiles of all encoders collapse to the same low attainment regardless of head, the bottleneck is the features or the labels, not the classifier. That would contradict the decoupling account for clinical text.
Baselines
The naive reference point is the uncalibrated classifier from standard training, reported as a single point in the 50-dimensional space. Its micro and macro F1 anchor the scale; for the BiLSTM-CRF, MedSecId reports micro F1 0.955 and macro F1 0.787. Any calibrated head should dominate this point along most directions. If it does not, the calibration family is too weak.
The strongest existing competitors for the "which encoder plus head is best" question are the decoupled heads of Kang et al.: classifier re-training, $\tau$-normalization and learnable weight scaling. Each is run on every frozen encoder with its standard tuning procedure and the same validation split and grid budget as our calibration search. They are the right competitors because they are exactly the single-point answers that practitioners use. On clinical text we found no reported result on Lacuna for these heads on MedSecId. The closest reported figure is learnable weight scaling at 49.9% on ImageNet-LT, which we will attempt to reproduce with the released recipe as a sanity check of our implementation. Each such head is one point. The attainment method "beats" this baseline in a specific, checkable sense. Either the profile shows a direction where a different calibration of the same encoder improves the rare sections by more than the bootstrap interval while holding the others at the baseline's levels, or the profile reverses an encoder ranking that the single-point heads imply.
For the CheXbert setting the strongest system is CheXbert itself at 0.798 average F1. We rerun it from its described two-stage recipe and use it as one of the encoders.
The key ablation removes the frozen-encoder structure. Instead of computing attainment by optimizing a head per direction, we compute the directional hypervolume estimate over a finite cloud of checkpoints and randomly calibrated heads, which is what the scalarizing approximation does in evolutionary search. If this finite-cloud estimate matches the optimized attainment, the per-direction optimization is unnecessary. We expect it to underestimate reach badly in rare-class directions, because random calibrations almost never land there.
Why not just…?
Why not just compare encoders by macro F1 after the best decoupled head? Macro F1 is one fixed weighting of the classes, and if attainment profiles never cross it loses nothing. That is a real possibility, and we treat it as a falsifiable outcome. We count encoder pairs whose profiles cross, meaning one wins in some directions and loses in others, and pairs whose macro-F1 ranking disagrees with profile dominance. If crossings are rare and within the noise, the honest conclusion is that macro F1 after re-tuning is sufficient for clinical section identification. The containment test will still have confirmed that conclusion rather than assumed it.
Why not just tune each class's threshold independently? This works when classes do not compete, and the CheXbert setting is exactly such a control. Across its 14 observations the labels are independent, so the attainable set should factor into a product of per-observation sets. Our estimator should reproduce that product, which checks it against a known answer. Section identification is different. Each token gets exactly one section, so adjusting one class's offset changes every other class's predictions, and independent tuning is not even well defined. The per-direction coupled search is needed there.
Why not compute exact hypervolume after grouping classes into a few buckets? Grouping is what we do for validation, and it is cheap and exact. It averages away trade-offs inside a group: two rare sections that compete for the same tokens, such as allergies against medications at admission, disappear into one "rare" number. We report grouped exact hypervolume alongside the full profile and show which conclusions change when the grouping is removed.
Why not use a many-objective optimizer such as BOtied or qNEHVI to search the head space? Those methods allocate expensive evaluations over a black-box function. Here an evaluation of a calibrated head on cached features costs milliseconds, and exhaustively solving each direction is affordable. Their scale-invariant or noise-aware machinery addresses a cost we do not have. They also report the quality of a finite sampled front, not whether one encoder's reachable set contains another's.
Proposed Experiments
The decisive checks come in order. First, on grouped objectives with three to five dimensions, the directional attainment-based hypervolume must agree with exact box-decomposition hypervolume on densely sampled heads within its bootstrap interval. We also report how many directions this takes as dimension grows; if the count explodes before 50, we fall back to profiles and containment and drop the scalar. Second, on CheXbert-style report labeling, the profile must factor across observations, as the independence of the labels requires. Third, on MedSecId, we compare attainment profiles of the five encoder regimes on held-out notes. We report the fraction of directions where each encoder dominates each other encoder, the attainment restricted to directions that emphasize rare sections, and where each of the Kang et al. heads sits inside its encoder's profile.
The result that would most change practice is a finding that class-balanced or focal-loss encoders are strictly contained in the instance-balanced encoder's attainable set. That would confirm, in the full 50-class space, that for clinical text the tail is a classifier problem. The opposite result would also be informative: rare-section attainment stays flat across all encoders and heads, meaning more calibration will not help and the effort belongs in encoders or labels. A secondary robustness check trains on discharge summaries and evaluates on the other MedSecId note types, testing whether attainable sets shrink uniformly under this template shift or collapse in specific directions.
References
- Decoupling Representation and Classifier for Long-Tailed Recognition
- Deep learning trends for focal brain pathology segmentation in MRI
- Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation
- Proposal of Approximating Hypervolume by Scalarizing Functions for Evolutionary Many-Objective Optimization
- Incorporation of Hypervolume Approximation with Scalarizing Functions into Indicator-Based Evolutionary Multiobjective Optimization Algorithms
- Efficient computation of expected hypervolume improvement using box decomposition algorithms
- Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement
- BOtied: Multi-objective Bayesian optimization with tied multivariate ranks
- A New Public Corpus for Clinical Section Identification: MedSecId
- Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT
- CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison