ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression

Botao Zhao, Xiaoyang Qu, Zuheng Kang, Junqing Peng, Jing Xiao, Jianzong Wang

Abstract

Contrastive learning has revolutionized classification by clustering similar samples, but its application to regression is hindered by the continuous nature of labels and the non-linearity of standard similarity metrics. We present ACCon, a novel regularizer that introduces "angle compensation" to ensure feature representation similarity linearly reflects label distance. By projecting features onto a semi-hypersphere and adjusting for cosine non-linearity, ACCon achieves superior performance on imbalanced and small-scale datasets across vision and NLP benchmarks.

Introduction

Deep regression tasks, such as age estimation or textual similarity scoring, require models to understand the continuous relationship between labels. While contrastive learning (e.g., SupCon) effectively groups discrete classes, it often fails in regression because it treats all non-identical samples as equally "negative." Existing distance-aware methods attempt to weight losses by label distance but typically use cosine similarity directly. This leads to a distorted feature space because the cosine function is non-linear, causing representations to cluster unevenly or "clump" on the hypersphere.

The ACCon Method

The core innovation of this work is the Angle-Compensated Contrastive (ACCon) loss. Unlike previous methods that focus solely on label ranking or direct distance weighting, ACCon aims to create a feature space where the geometric angle between samples accurately and linearly represents the numerical distance between their labels.

Comparison of ACCon with existing contrastive methods. (A) contrasts order-aware and distance-aware approaches with ACCon's angle compensation. (B) shows standard SupCon for classification. (C) illustrates the ACCon projection onto a semi-hypersphere.

Key technical components include:

  1. Angle Compensation: The loss function incorporates a "compensation" term to linearize the relationship between cosine distance and label distance, preventing the geometric distortions common in standard contrastive frameworks.
  2. Semi-hypersphere Projection: To respect the monotonic nature of regression labels, ACCon projects features onto a semi-hypersphere rather than a full sphere, better preserving the continuous order of targets.
  3. Plug-and-Play Regularization: ACCon acts as a regularizer added to standard regression losses (like MSE or MAE), guiding the encoder toward more logical feature representations without requiring architectural changes.

Experimental Results

We evaluated ACCon on AgeDB (facial age), IMDB-WIKI (large-scale age estimation), and STS-B (semantic textual similarity) benchmarks. ACCon consistently outperformed state-of-the-art (SOTA) methods like RankSim, RNC, and ConR.

Feature Space Analysis

Visualizations confirm that ACCon produces a more continuous and symmetric feature space. As shown in the t-SNE plots, ACCon creates a smooth gradient that mirrors the label space, whereas vanilla models or RankSIM often result in fragmented or overly compressed representations. Quantitatively, ACCon achieves a significantly higher linear correlation (Pearson Corr: -0.735) between feature similarity and label distance compared to other methods.

Qualitative and quantitative analysis of the feature space. (a) t-SNE visualizations show ACCon's superior structural continuity. (b) Distribution plots demonstrate that ACCon achieves a much stronger linear correlation between feature similarity and label distance.

Performance Highlights

  • Accuracy Gains: ACCon improved Mean Absolute Error (MAE) by 4.54% on AgeDB, 9.27% on STS-B, and 3.72% on IMDB-WIKI compared to vanilla regression models.
  • Data Efficiency: On a reduced AgeDB dataset (1/8th size), ACCon maintained significantly lower error rates, demonstrating its ability to learn the underlying task logic with fewer examples.
  • Imbalanced Data: In "few-shot" label regions (labels with very few training samples), ACCon showed up to 34% improvement over vanilla models on the AgeDB-DIR benchmark.

Conclusion

ACCon addresses a fundamental mismatch between contrastive learning and continuous regression. By linearizing the feature-label relationship through angle compensation, it provides a robust regularizer that excels in data-scarce and imbalanced scenarios. Future work will explore more diverse augmentation strategies to further enhance performance in non-visual domains like NLP where standard image augmentations are not applicable.

Summary updated

Abstract

In deep regression, capturing the relationship among continuous labels in feature space is a fundamental challenge that has attracted increasing interest. Addressing this issue can prevent models from converging to suboptimal solutions across various regression tasks, leading to improved performance, especially for imbalanced regression and under limited sample sizes. However, existing approaches often rely on order-aware representation learning or distance-based weighting. In this paper, we hypothesize a linear negative correlation between label distances and representation similarities in regression tasks. To implement this, we propose an angle-compensated contrastive regularizer for deep regression, which adjusts the cosine distance between anchor and negative samples within the contrastive learning framework. Our method offers a plug-and-play compatible solution that extends most existing contrastive learning methods for regression tasks. Extensive experiments and theoretical analysis demonstrate that our proposed angle-compensated contrastive regularizer not only achieves competitive regression performance but also excels in data efficiency and effectiveness on imbalanced datasets.
BibTeX
@inproceedings{zhao2025accon,
  author = {Botao Zhao and Xiaoyang Qu and Zuheng Kang and Junqing Peng and Jing Xiao and Jianzong Wang},
  title = {{ACCon: Angle-Compensated Contrastive Regularizer for Deep Regression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34435},
}
Figure