Bias or Rigidity? Diagnosing and Repairing Rare-Class Failure in Adapted Language Models

When a text classifier is trained on skewed data, such as a corpus where 90% of news paragraphs are benign and only a few hundred are condescending, it misses the rare classes. The usual repairs in natural language processing work on the loss: errors on rare classes are weighted more heavily. These repairs treat two different failures as one. The first is classifier bias. The features already separate the rare class, but the output layer favors frequent classes. That kind of bias can be fixed cheaply after training. The second is representation rigidity. The pretrained model, kept frozen or adapted only through small add-on modules, never places rare-class texts apart from some frequent "confuser" class. No reweighting of the output layer can fix that.

We propose to measure, class by class, how much of the rare-class error comes from each source. We then propose a targeted fix for the rigid part. The measurement uses a prototype classifier, which represents each class by a single point in feature space and labels an input by its nearest point. Built with balanced priors, it gives a bias-free reading of what the current features can separate. The fix is a prototype-anchored low-rank adapter. During adaptation, frequent-class examples are held near their original frozen positions, so the adapter's small capacity goes to moving rare-class examples away from their confusers.

Core Claim

We claim that in long-tailed text classification with parameter-efficient adaptation, most tail error left after post-hoc bias correction comes from entanglement in the frozen space. Our hypothesis is that standard low-rank adaptation (LoRA, which adds small trainable low-rank matrices to frozen weights) underuses its capacity on that entanglement. Because head-class gradients dominate training, its update subspace aligns with directions that separate head classes from each other. The proposal makes three falsifiable predictions. First, a per-class "rigidity gap," computed from frozen features alone, predicts which classes gain from adaptation and which are fixed by bias correction alone. Second, anchoring head-class features in place moves adapter capacity toward tail-versus-confuser directions. Third, this raises tail macro-F1 over LoRA trained with logit-adjusted loss at equal parameter count.

Relation to Prior Work

Two vision papers motivate the bias half of the story. Decoupling Representation and Classifier for Long-Tailed Recognition showed that long-tailed failures stem largely from classifier weight skew. Retraining or rescaling the classifier on a frozen backbone recovers much of the tail. Learning Prototype Classifiers for Long-Tailed Recognition explained the cause. In a dot-product classifier, weight norms grow with class frequency. Euclidean prototypes avoid this and keep roughly equal norms across classes. We reuse the second paper's classifier as our measuring instrument and as the head of our method. Neither paper asks what remains after bias is removed when the backbone cannot be fully retrained, and neither studies text.

Parameter-Efficient Long-Tailed Recognition (LIFT) constrains our design. It found that full fine-tuning distorts tail features while lightweight updates preserve them, and it initialized classifiers from text embeddings. We accept its premise that light adaptation is the right regime. LIFT, however, adapts uniformly across classes and was evaluated only on images.

The rigidity literature supplies the second failure mode. It is almost entirely visual or medical. Few-shot Adaptation of Medical Vision-Language Models found prompt tuning rarely beats a linear probe when the backbone lacks domain features. Bridging the Pathology Domain Gap and Learning Federated Visual Prompt in Null Space for MRI Reconstruction report adapter ceilings when target structure is absent from pretraining. These works describe rigidity at the level of a whole domain. Our contribution is to measure it per class under imbalance, where the open problem is that narrow low-rank subspaces may collapse rare-class signal.

Three adapter-plus-prototype or adapter-plus-imbalance papers sit closest to us:

Our anchors are the head classes' own frozen positions, derived from data rather than imposed. This respects the point that orientations matter without fixing any geometry. Our anchoring is asymmetric by class frequency, which none of these papers does.

On the text side, Robust Text Classification: Analyzing Prototype-Based Networks established that prototype heads work on BERT-family backbones. That study used fully fine-tuned backbones on balanced data. It also reported sensitivity to how tightly prototypes cluster, a warning we take seriously for scarce tail classes. The imbalanced-text systems we compete with all attach linear heads to fully fine-tuned transformers and rely on Focal Loss or weighted cross-entropy:

Research Program

The first component is the diagnostic. Let $z = f(x;\Theta_0)$ be the frozen feature of text $x$, and let $c_k$ be the mean training feature of class $k$. On frozen features we fit two classifiers. One is a linear head trained with plain cross-entropy. The other is a nearest-prototype classifier with uniform priors. The per-class bias gap is the tail recall that the prototype classifier recovers over the linear head.

The rigidity gap needs a reference for what better features could achieve. We obtain it in two ways. In controlled settings, we build long-tailed versions of balanced corpora (AG_NEWS, DBPedia) by subsampling classes to decay exponentially with imbalance ratio 100. This follows the CIFAR-100-LT recipe. The rigidity gap is then the difference between frozen-feature prototype accuracy on class $k$ and the accuracy of a model fine-tuned on the full balanced corpus. In natural corpora, where no balanced version exists, we use a label-free proxy. For each tail class $k$, the proxy is the ratio of its within-class spread to its distance from the nearest head prototype. We validate the proxy against the true gap in the controlled settings before relying on it.

The second component is the method, prototype-anchored adaptation. We attach rank-$r$ LoRA modules to a frozen RoBERTa-large backbone, giving adapted features $z' = f(x;\Theta_0,\Delta\theta)$. We train them with a logit-adjusted Euclidean prototype loss plus a frequency-weighted anchor:

$$\mathcal{L} = -\log \frac{\exp(-\|z'-c_y\|/T + \log \pi_y)}{\sum_j \exp(-\|z'-c_j\|/T + \log \pi_j)} \;+\; \beta\, \frac{n_y}{n_{\max}}\, \|z' - z\|^2 .$$

Here $c_j$ are learnable prototypes, $T$ is a temperature, $\pi_j$ is the empirical class frequency, $n_y$ is the training count of the example's class, $n_{\max}$ is the largest class count, and $\beta$ sets anchor strength. The $\log \pi$ terms apply logit adjustment, which removes frequency bias during training. The anchor term barely restrains tail examples and strongly restrains head examples. The adapter therefore cannot buy its loss reduction by reshuffling the head, and must move the tail. We initialize tail prototypes from frozen embeddings of class definitions taken from the annotation guidelines. This tests whether LIFT's text-anchor idea helps when the "label text" is a gloss such as "The Saviour" in the condescending-language taxonomy. We report this initialization only as an ablation.

The third component is mechanistic analysis. For each run we compute the principal angles between the LoRA update subspace (the column space of the learned low-rank matrices in the top layers) and the discriminant direction separating each tail class from its nearest head confuser. The hypothesis predicts that this angle is smaller under anchoring than under plain LoRA. It also predicts that the reduction is larger for classes with high rigidity gaps.

Baselines

The naive reference is a frozen backbone with a linear head trained by ordinary cross-entropy. It shows the raw size of the imbalance problem under the same compute as every other method.

The bias-only reference has two parts. One is the nearest-prototype classifier with uniform priors on frozen features. The other is a frozen-feature linear head with post-hoc logit adjustment and per-class threshold tuning on the validation split, following the decoupling recipe of Decoupling Representation and Classifier for Long-Tailed Recognition. Anything this reference recovers is, by construction, not rigidity. A method beats it only if it improves tail macro-F1 on classes the diagnostic flags as rigid.

The strongest parameter-efficient competitor is LIFT ported to text: LoRA at the same rank, logit-adjusted loss, and a linear head initialized from class-gloss embeddings, as in Parameter-Efficient Long-Tailed Recognition. No reported result was found on Lacuna for LIFT or any prototype classifier on long-tailed text. Its only reported figure is visual, 78.3% overall accuracy on ImageNet-LT with 0.62M parameters, so we will run it ourselves.

The strongest existing text systems use full fine-tuning with Focal Loss or weighted cross-entropy. On EDOS Task C, iREL at SemEval-2023 Task 10 reports 54.62% validation macro-F1 with RoBERTa-large and Focal Loss. On SMM4H 2024 Task 3, PCIC at SMM4H 2024 reports 0.655 test macro-F1. That figure comes from XLNet-large with combined losses, context extraction and paraphrase augmentation. On Don't Patronize Me, the dataset paper reports 70.63% binary F1 for RoBERTa. The shared-task overview reports top Subtask 2 systems in the high 40s macro-F1.

We will rerun full fine-tuning with Focal Loss ourselves on the same splits and backbone. We cite the reported numbers only as sanity anchors, since their protocols differ: validation versus test, and extra augmentation in the SMM4H system.

The key ablations isolate the anchor. The first sets $\beta=0$, which leaves LoRA with the prototype head and logit adjustment. The second applies the anchor uniformly to all classes, an ordinary feature-preservation regularizer. Only if frequency-weighted anchoring beats uniform anchoring is the asymmetry, rather than generic regularization, doing the work.

All methods share the following controls:

  • the same backbone and splits;
  • the same trainable-parameter budget for adapter methods;
  • an identical 30-trial random hyperparameter search over learning rate, temperature, rank (for $\beta=0$) and $\beta$ (for ours);
  • five seeds each.

Beating the strongest baseline means a tail macro-F1 gain over LIFT-text whose seed-level 95% interval excludes zero, on at least two of the three natural multi-class benchmarks, with head-class F1 no worse than one point lower.

Why not just…?

Why not just use Focal Loss or class weights, as the shared tasks do? Reweighting rescales gradients but leaves the dot-product preference for high-norm head weights intact. Learning Prototype Classifiers for Long-Tailed Recognition reports a prototype head at 53.41% versus 46.12% for a softmax baseline on CIFAR-100-LT. Even so, iREL's Focal Loss system sits at 54.62% macro-F1 on 11-class EDOS. Our full fine-tuning plus Focal Loss baseline tests this directly. If it matches our method on rigid classes, our claim that light adaptation needs targeting is wrong for this regime.

Why not just tune thresholds or apply logit adjustment after training? For binary tasks such as Don't Patronize Me Subtask 1, this is a real threat. With two classes, frequency bias reduces to a shifted threshold, and threshold tuning should capture most of it. We therefore expect our method to help little there and do not claim otherwise. The multi-class benchmarks are where post-hoc correction cannot separate a tail class from a specific confuser. The diagnostic predicts which classes those are before the method is run. That prediction can be checked.

Why not just run LoRA with a logit-adjusted loss, LIFT-style? This is the most plausible equal competitor, which is why the proposal does not rest on beating it alone. If anchoring gives no gain, the diagnostic still stands as the contribution. In that case it would show whether the remaining tail error under LoRA is bias or rigidity, and whether higher rank or full fine-tuning is needed. The principal-angle analysis decides between the two accounts. If plain LoRA's update subspace already aligns with tail-versus-confuser directions, the head-dominance hypothesis is false and we drop the method claim.

Why not just oversample or paraphrase the tail? Decoupling Representation and Classifier for Long-Tailed Recognition found class-balanced resampling degrades learned representations. PCIC reports paraphrase augmentation with mixed, backbone-dependent effects. We include class-balanced oversampling under LoRA as a further baseline at the same budget.

Proposed Experiments

The controlled study comes first. We use long-tailed AG_NEWS and DBPedia at imbalance ratios 10, 50 and 100. Here the true rigidity gap is known, so we can test whether the frozen-space proxy ranks classes correctly. The decisive check is a Spearman correlation of at least 0.6 between proxy and true gap across classes and ratios. AG_NEWS has only four classes, so DBPedia's 14 carry most of the statistical weight.

The natural study uses EDOS Task C (11 classes), Don't Patronize Me Subtask 2 (7 categories) and SMM4H 2024 Task 3 (4 classes), with Don't Patronize Me Subtask 1 as the binary control. Metrics are macro-F1 overall, plus F1 split into head, medium and tail thirds by training frequency.

Three results would decide the proposal:

  1. Across all classes and datasets, the per-class gain of adaptation over the bias-only reference correlates with the rigidity gap measured before training. If it does not, the decomposition is not useful.
  2. Frequency-weighted anchoring beats both $\beta=0$ and uniform anchoring on high-rigidity tail classes. If it does not, the anchor is not the active ingredient.
  3. The LoRA-subspace-to-confuser angle shrinks under anchoring. If it does not, the mechanism we propose is not the one at work, even if accuracy improves.

A secondary check addresses the clustering sensitivity reported for prototype networks in Robust Text Classification. We sweep the temperature $T$ and report tail-F1 variance across the sweep for prototype heads versus linear heads. If prototype heads are much less stable on scarce tails, that is a cost practitioners need to know about.