ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression

Botao Zhao, Xiaoyang Qu, Zuheng Kang, Junqing Peng, Jing Xiao, Jianzong Wang

Abstract

In deep regression, capturing the relationship among continuous labels in feature space is a fundamental challenge that has attracted increasing interest. Addressing this issue can prevent models from converging to suboptimal solutions across various regression tasks, leading to improved performance, especially for imbalanced regression and under limited sample sizes. However, existing approaches often rely on order-aware representation learning or distance-based weighting. In this paper, we hypothesize a linear negative correlation between label distances and representation similarities in regression tasks. To implement this, we propose an angle-compensated contrastive regularizer for deep regression, which adjusts the cosine distance between anchor and negative samples within the contrastive learning framework. Our method offers a plug-and-play compatible solution that extends most existing contrastive learning methods for regression tasks. Extensive experiments and theoretical analysis demonstrate that our proposed angle-compensated contrastive regularizer not only achieves competitive regression performance but also excels in data efficiency and effectiveness on imbalanced datasets.

Introduction

Traditional contrastive learning frameworks, such as Supervised Contrastive Learning (SupCon), were designed for classification tasks where the goal is to pull samples of the same class together and push different classes apart. However, applying these methods to deep regression is non-trivial because labels are continuous (e.g., age, price, or similarity scores). In regression, "negatives" are not binary; a sample with a label close to the anchor is less negative than one with a distant label.

Existing "distance-aware" methods attempt to address this by weighting the loss based on label differences. However, these methods often ignore a critical mathematical reality: cosine similarity is non-linear. Directly weighting a non-linear function leads to distorted feature spaces, which hampers generalization, particularly in scenarios with limited or imbalanced data.

Methodology: Angle Compensation

The core contribution of this work is the Angle-Compensated Contrastive Regularizer (ACCon). The authors hypothesize that for optimal regression performance, there should be a linear negative correlation between label distances and representation similarities.

Comparison of representation learning methods for regression. ACCon (ours) uses angle compensation on a semi-hypersphere to maintain linear label-distance relationships.

To achieve this, ACCon introduces three key components:

  1. Linear Hypothesis: The model is forced to learn feature representations where the similarity between embeddings maps linearly to the distance between their continuous labels.
  2. Angle Compensation: Instead of simple weighting, the method derives an "ideal angle" between samples based on label differences. It then applies a compensation term to the contrastive loss to account for the non-linearity of the cosine function.
  3. Semi-Hypersphere Projection: Features are projected onto a semi-hypersphere. This geometric constraint prevents feature collapse and ensures that the relative distances between continuous labels are preserved in the embedding space.

ACCon is designed as a "plug-and-play" regularizer. The final training objective combines a standard regression loss (like MSE) with the ACCon loss: $\mathcal{L} = \mathcal{L}_{reg} + \gamma \cdot \mathcal{L}_{ACCon}$.

Experimental Results

The researchers evaluated ACCon across diverse domains, including age estimation (AgeDB, IMDB-WIKI) and natural language processing (STS-B). The method was tested under both natural distributions and the Deep Imbalanced Regression (DIR) benchmark.

Performance and Data Efficiency

ACCon consistently outperformed state-of-the-art baselines. On the AgeDB-Natural dataset, it reduced Mean Absolute Error (MAE) by 4.54% over the vanilla model. The gains were even more significant on imbalanced datasets; for rare "few-shot" labels in AgeDB-DIR, ACCon improved MAE by 34% compared to the vanilla baseline.

Data efficiency comparison. ACCon consistently achieves lower Mean Absolute Error (MAE) than Vanilla and RankSIM baselines across varying proportions of training data.

Furthermore, the method demonstrated exceptional data efficiency. As shown in the comparative analysis, ACCon maintained lower error rates than both Vanilla and RankSIM models even when training data was reduced to 1/8th of the original size. This suggests that the structured feature space learned by ACCon allows the model to generalize better from fewer examples.

Conclusion

ACCon addresses a fundamental limitation in applying contrastive learning to regression by linearizing the relationship between label distances and feature similarities. By enforcing a global geometric structure on the feature space, it significantly improves performance on imbalanced and small-scale datasets. While currently limited by the computational intensity of contrastive pairs and the requirement for supervised labels, ACCon provides a robust, compatible framework for enhancing a wide variety of deep regression tasks.

Summary updated

Abstract

In deep regression, capturing the relationship among continuous labels in feature space is a fundamental challenge that has attracted increasing interest. Addressing this issue can prevent models from converging to suboptimal solutions across various regression tasks, leading to improved performance, especially for imbalanced regression and under limited sample sizes. However, existing approaches often rely on order-aware representation learning or distance-based weighting. In this paper, we hypothesize a linear negative correlation between label distances and representation similarities in regression tasks. To implement this, we propose an angle-compensated contrastive regularizer for deep regression, which adjusts the cosine distance between anchor and negative samples within the contrastive learning framework. Our method offers a plug-and-play compatible solution that extends most existing contrastive learning methods for regression tasks. Extensive experiments and theoretical analysis demonstrate that our proposed angle-compensated contrastive regularizer not only achieves competitive regression performance but also excels in data efficiency and effectiveness on imbalanced datasets.
BibTeX
@inproceedings{zhao2025accon,
  author = {Botao Zhao and Xiaoyang Qu and Zuheng Kang and Junqing Peng and Jing Xiao and Jianzong Wang},
  title = {{ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34435},
}
Figure