Texture recognition networks discard where things are in an image and keep only statistics of what local patterns occur. This is the right choice for lung CT tissue, fabric or terrain, where the exact position of a pattern carries no meaning. The pooling layers that do this well (Fisher vectors, bilinear pooling, learned dictionary encoders) work by producing very long feature vectors, often tens of thousands of numbers per image. These datasets are also multi-label and badly imbalanced. A CT slice can show several disease patterns at once, and some patterns, such as honeycombing, are rare. The usual fix is to give each label its own yes/no output, a sigmoid trained with binary cross-entropy, and to upweight the rare labels' errors. We argue that this fix has a predictable breaking point. Each independent label is its own binary problem with its own positive rate, and recent theory says loss weighting cannot balance a binary classifier's errors once the feature dimension is too large relative to the number of rare positives. Texture pooling inflates the feature dimension, so it may push exactly the rarest labels past that point.
The proposal is to test this prediction on real multi-label texture data and to build the fix it implies. The prediction is that a single per-label ratio, computable before training, sorts labels into those that weighting can rescue and those it cannot. The fix is to give each rare label's output a projection of the texture descriptor whose width is capped by that label's own positive count, while frequent labels keep the full descriptor. If the prediction holds, it explains a known failure: honeycomb detection on lung CT reaches an F-score of only 0.3385 while other patterns exceed 0.89 (Gao et al.). It also turns choosing a dimension for imbalanced texture heads from guesswork into a rule.
Core Claim
Thumb on the Scale studies retraining only the last layer on frozen features of dimension $d$ with $n$ samples. It proves that when $d/n > 2\pi_+$, where $\pi_+$ is the minority fraction, no choice of loss weights equalizes minority and majority error. That paper works with single-label problems. Under independent label loss, the analysis applies separately to each label $k$, with its own positive fraction $\pi_k$. This motivates a per-label score:
$$r_k = \frac{d}{2\,\pi_k\, n},$$
where $d$ is the dimension of the input to label $k$'s output layer and $n$ is the effective number of training images. The theory predicts that labels with $r_k > 1$ will show a minority-error gap that does not close however heavily they are weighted. Labels with $r_k < 1$ should respond to weighting.
Our claim has two parts, and the first can fail independently of the second. First, $r_k$ predicts which texture labels resist weighting in practice, even though the networks are deep and the theory's assumptions are idealized. Second, capping each rare label's input dimension near $2\pi_k n$ recovers its accuracy, while frequent labels keep the full descriptor. Global dimension reduction cannot do this without hurting frequent labels. The decomposition of multi-label learning into per-label binary problems is what allows a separate treatment for each label. On label dependence and loss minimization shows that for decomposable metrics such as Hamming loss, predicting each label's marginal is Bayes-optimal. So treating labels separately loses nothing for the per-label F-scores we report.
Relation to Prior Work
The Gao et al. papers on interstitial lung disease (holistic classification, label propagation and unordered pooling, multi-label regression) already combine all three directions. They use orderless Fisher-vector pooling, sums of per-label logistic losses and static class-balancing weights on whole CT slices. These papers motivate our work: balancing raised overall F-score from 0.7146 to 0.7656, yet honeycomb stayed far behind. They never asked why weighting stopped helping on that label, and they reduced dimension with one global PCA. The obvious next step, Deep TEN with weighted BCE on LTRC, is the candidate gap the literature points to. It is an engineering swap, and we use it as a baseline rather than as the contribution.
The method comes from three sources. Deep TEN is a learnable dictionary encoder trained end to end, evaluated only with single-label softmax; we use it as the scalable orderless backbone. Deep Filter Banks and Bilinear CNNs supply the high-dimensional descriptors whose width we vary. VS-loss (Kini et al.) shifts and scales logits to change the trained classifier's decision boundary. Kini et al. also constrain the design: they show that plain weights lose their effect once training reaches zero error. Because VS-loss alters the direction of the decision boundary rather than its offset, it is the natural competitor to dimension capping. The neural-collapse view of reweighting (Rethinking Loss Reweighting) is derived for softmax and has no stated counterpart for independent sigmoids, so we do not build on it.
Research Program
The first stage is a controlled measurement in the setting the theory covers. We freeze a backbone and extract orderless descriptors: Fisher vectors and Deep TEN encodings with codebook sizes from 8 to 128, plus compact bilinear features. We then retrain only the per-label linear outputs with weighted BCE. For each label we sweep the weight over a wide grid and record the smallest achievable gap between false-positive and false-negative rates. We also sweep the dimension $d$ with random projections. The test is whether the point where the gap stops closing lines up with $r_k \approx 1$ across labels and widths.
Estimating $n$ needs care. LTRC slices come from only 658 patients, and neighboring slices are nearly duplicates. We therefore compute $r_k$ twice, once using slice counts and once using patient-level positive counts. We report which choice predicts the measured breaking point. This is itself a finding, because it tells practitioners which sample count matters for medical data.
The second stage builds the fix. Each label's output layer reads from its own learned linear projection of the shared descriptor. The projection width is set to $\min(d, c\cdot 2\pi_k n)$, with one global constant $c$ tuned on validation. Frequent labels effectively keep the full descriptor. The third stage trains everything end to end on Deep TEN. It tests whether the rule still holds once the backbone adapts to the data, where the theory gives no guarantee.
Baselines
The naive reference is a ResNet-50 with global average pooling, meaning one mean per channel and no higher-order texture statistics. It is trained with unweighted per-label BCE and uses a fixed 0.5 threshold. It shows how much any of the machinery matters.
The strongest published system on our main benchmark is the Gao et al. holistic pipeline. Their smooth $L_1$ regression model reports an overall F-score of 0.8028 on LTRC at a presence threshold of 6000 pixels, with honeycomb at 0.3385 (Gao et al.). Their multi-label classification CNN reports 0.7959 overall (Gao et al.). They used a weaker CNN-F backbone. We therefore reimplement their class-balanced per-label logistic loss on our backbone, using the same patient-level 5-fold split, and also quote their numbers. Beating them means exceeding 0.8028 overall. More importantly, it means raising honeycomb F-score with non-overlapping fold-wise confidence intervals while losing no more than one point on the other three patterns.
The strongest modern competitor is Deep TEN with inverse-frequency weighted BCE, VS-loss logit adjustment for each label, and validation-tuned thresholds for each label. Every method gets the same backbone, the same epochs and the same number of hyperparameter trials, 30 random-search trials per method. The ablation that isolates the key ingredient keeps every other component and replaces the per-label width cap with one global PCA to the same total parameter budget. A further variant uses per-label widths chosen at random rather than by $r_k$.
On DTD's 47-attribute task, the reported reference is 64.5% mAP for SIFT Fisher vectors with independent SVMs (Describing Textures in the Wild). The file gives no multi-label mAP for FV-CNN or Deep TEN on DTD; no reported result found on Lacuna for those, so we will run them ourselves.
Why not just…?
Why not just tune a decision threshold for each label on validation data? This is the most serious threat, because F-score is very sensitive to thresholds and threshold tuning costs almost nothing. Moving the threshold shifts where the boundary sits but does not change its direction. In the overparameterized regime, the theory says the direction is the problem: the boundary is set by the handful of rare positives. If threshold tuning alone closes the honeycomb gap, our hypothesis is wrong for this data. For that reason every method, including the baselines, is reported both with and without per-label thresholds.
Why not just reduce dimension globally with PCA, as Gao et al. did? A global cut that is small enough for the rarest label discards variance the frequent labels use. A cut that is large enough for the frequent labels leaves rare labels above $r_k = 1$. The matched-budget global-PCA ablation tests exactly this trade-off. If global PCA matches our method, the per-label structure adds nothing and we will say so.
Why not just use VS-loss and skip the dimension rule? VS-loss was shown to improve long-tailed CIFAR-100 from 38.82% to 43.52% top-1 (Kini et al.). It may be enough. However, its proven benefit concerns where the classifier converges. It does not escape the limit on equalizing errors in last-layer retraining, which constrains any loss applied to features that are too wide. We run VS-loss alone, the width cap alone, and both together. If VS-loss alone matches the combination on labels with $r_k > 1$, that would show the limit is not binding in practice.
Why not just use global average pooling, which is already position-invariant? Fisher-vector pooling beat fully connected pooling on DTD by 7.1 points for VGG-M and 13 points for VGG-VD (Deep Filter Banks). So higher-order statistics matter for textures. The interesting question is whether the low dimension of average pooling helps rare labels anyway. Our naive baseline reports per-label results to show this.
Why not just oversample rare images? In multi-label data, repeating a honeycomb slice also repeats the ground-glass pattern on the same slice. Thumb on the Scale also reports that properly weighted training beats downsampling. We include label-aware oversampling as a secondary baseline but do not expect it to change the conclusion.
Proposed Experiments
The decisive check is a plot of each label's lowest reachable error gap, the best over all weights, against $r_k$. Each point is a (label, width, dataset) combination, pooled from LTRC's four patterns and DTD's 47 attributes. The hypothesis survives if the gap departs from zero near $r_k = 1$ for one consistent definition of $n$. It fails if there is no visible transition, or if the transition sits far from 1 in a way that differs between datasets.
The second check is on LTRC, using patient-level 5-fold cross-validation. We report overall and per-pattern F-score, with honeycomb as the primary outcome. We compare against the full baseline set, with and without per-label thresholds.
The third check is DTD multi-label mAP, broken out by attribute frequency. It asks whether the width cap helps exactly the rare attributes that $r_k$ flags and leaves the others unchanged.
The fourth check is a transfer test. We take $c$ tuned on LTRC and apply it unchanged to DTD. This shows whether the rule needs any fitting per dataset.
References
- Multi-label Deep Regression and Unordered Pooling for Holistic Interstitial Lung Disease Pattern Detection
- Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining
- On label dependence and loss minimization in multi-label classification
- Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks
- Interstitial Lung Diseases via Deep Convolutional Neural Networks: Segmentation Label Propagation, Unordered Pooling and Cross-Dataset Learning
- Deep TEN: Texture Encoding Network
- Deep Filter Banks for Texture Recognition, Description, and Segmentation
- Bilinear CNN Models for Fine-Grained Visual Recognition
- Label-Imbalanced and Group-Sensitive Classification under Overparameterization
- Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of View
- Describing Textures in the Wild