Abstract
Contrastive learning has revolutionized classification by clustering similar samples, but its application to regression is hindered by the continuous nature of labels and the non-linearity of standard similarity metrics. We present ACCon, a novel regularizer that introduces "angle compensation" to ensure feature representation similarity linearly reflects label distance. By projecting features onto a semi-hypersphere and adjusting for cosine non-linearity, ACCon achieves superior performance on imbalanced and small-scale datasets across vision and NLP benchmarks.
Introduction
Deep regression tasks, such as age estimation or textual similarity scoring, require models to understand the continuous relationship between labels. While contrastive learning (e.g., SupCon) effectively groups discrete classes, it often fails in regression because it treats all non-identical samples as equally "negative." Existing distance-aware methods attempt to weight losses by label distance but typically use cosine similarity directly. This leads to a distorted feature space because the cosine function is non-linear, causing representations to cluster unevenly or "clump" on the hypersphere.
The ACCon Method
The core innovation of this work is the Angle-Compensated Contrastive (ACCon) loss. Unlike previous methods that focus solely on label ranking or direct distance weighting, ACCon aims to create a feature space where the geometric angle between samples accurately and linearly represents the numerical distance between their labels.
Key technical components include:
- Angle Compensation: The loss function incorporates a "compensation" term to linearize the relationship between cosine distance and label distance, preventing the geometric distortions common in standard contrastive frameworks.
- Semi-hypersphere Projection: To respect the monotonic nature of regression labels, ACCon projects features onto a semi-hypersphere rather than a full sphere, better preserving the continuous order of targets.
- Plug-and-Play Regularization: ACCon acts as a regularizer added to standard regression losses (like MSE or MAE), guiding the encoder toward more logical feature representations without requiring architectural changes.
Experimental Results
We evaluated ACCon on AgeDB (facial age), IMDB-WIKI (large-scale age estimation), and STS-B (semantic textual similarity) benchmarks. ACCon consistently outperformed state-of-the-art (SOTA) methods like RankSim, RNC, and ConR.
Feature Space Analysis
Visualizations confirm that ACCon produces a more continuous and symmetric feature space. As shown in the t-SNE plots, ACCon creates a smooth gradient that mirrors the label space, whereas vanilla models or RankSIM often result in fragmented or overly compressed representations. Quantitatively, ACCon achieves a significantly higher linear correlation (Pearson Corr: -0.735) between feature similarity and label distance compared to other methods.
Performance Highlights
- Accuracy Gains: ACCon improved Mean Absolute Error (MAE) by 4.54% on AgeDB, 9.27% on STS-B, and 3.72% on IMDB-WIKI compared to vanilla regression models.
- Data Efficiency: On a reduced AgeDB dataset (1/8th size), ACCon maintained significantly lower error rates, demonstrating its ability to learn the underlying task logic with fewer examples.
- Imbalanced Data: In "few-shot" label regions (labels with very few training samples), ACCon showed up to 34% improvement over vanilla models on the AgeDB-DIR benchmark.
Conclusion
ACCon addresses a fundamental mismatch between contrastive learning and continuous regression. By linearizing the feature-label relationship through angle compensation, it provides a robust regularizer that excels in data-scarce and imbalanced scenarios. Future work will explore more diverse augmentation strategies to further enhance performance in non-visual domains like NLP where standard image augmentations are not applicable.
Summary updated