BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning

Gustav Wagner Zakarias, Lars Kai Hansen, Zheng-Hua Tan

Introduction

Self-supervised learning (SSL) has become a cornerstone of modern AI, enabling models to learn robust representations from vast quantities of unlabeled data. However, a significant challenge remains: the "alignment" problem. Features learned through general pretext tasks (e.g., contrastive learning) often do not perfectly align with the specific requirements of downstream tasks. This discrepancy can lead to a "feature shift" during fine-tuning, where the model may discard useful pretrained knowledge or fail to adapt efficiently to the target domain.

The BiSSL Framework

To address this, we introduce BiSSL (Bilevel Self-Supervised Learning), a framework that inserts a specialized "preparatory" stage between pretraining and fine-tuning. BiSSL utilizes Bilevel Optimization (BLO) to refine model representations such that they satisfy both the original pretext objective and the target downstream objective simultaneously.

Comparison between conventional SSL and the proposed BiSSL pipeline, illustrating the nested optimization loop.

The optimization is structured as a nested problem:

  1. Lower-Level (Pretext): The model continues solving the SSL task (like SimCLR or BYOL) but is regularized to remain near parameters that benefit the downstream task.
  2. Upper-Level (Downstream): The model optimizes for the final target task, effectively "guiding" the lower-level updates to ensure the backbone produces features highly relevant to the end goal.

To handle the computational complexity of second-order derivatives (the Hessian), BiSSL employs the Implicit Function Theorem and a Conjugate Gradient (CG) method for efficient approximation.

Experimental Evaluation

BiSSL was rigorously tested across a diverse suite of vision benchmarks, including ImageNet-1K for pretraining and 12 downstream classification datasets (e.g., CIFAR, CUB200 Birds, Stanford Cars). It was also evaluated on dense prediction tasks such as PASCAL VOC for object detection and Cityscapes for semantic segmentation. The framework proved to be architecture-agnostic, showing gains across ResNet-50 and Vision Transformers (ViT).

Key Results

Our experiments demonstrate that BiSSL consistently outperforms standard fine-tuning. It improved top-1 accuracy on 11 out of 12 classification datasets. The most significant gains were observed in fine-grained recognition tasks, where domain-specific features are critical; for instance, BYOL-pretrained models saw a 5.6% boost on FGVC Aircraft and a 4.4% boost on CUB200 Birds.

Beyond classification, BiSSL showed substantial improvements in dense prediction tasks, as detailed in the table below.

BiSSL achieves consistent performance gains in object detection (VOC07+12) and semantic segmentation (Cityscapes) across different SSL baselines.

Analysis of the parameter space revealed that BiSSL-trained backbones have higher cosine similarity to their final fine-tuned states compared to standard SSL backbones, confirming a smoother transition and better alignment during the final adaptation phase.

Discussion and Limitations

While BiSSL provides a powerful bridge for transfer learning, it introduces additional computational overhead—updates typically take 2-4 times longer than standard gradients due to the CG iterations. Furthermore, the method currently requires access to the original pretext data and assumes identical backbone architectures for both stages. Future work will explore integrating BiSSL with Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA to mitigate these costs and enhance flexibility.

Summary updated

BibTeX
@article{zakarias2025bissl,
  author = {Gustav Wagner Zakarias and Lars Kai Hansen and Zheng-Hua Tan},
  title = {{BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=GQAGlqOpyA},
}
Figure