BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning

Gustav Wagner Zakarias, Lars Kai Hansen, Zheng-Hua TanSubmission to ICLR 2025 · Rejected

TL;DR

Self-supervised learning (SSL) typically separates pre-training and downstream fine-tuning into two completely disjoint stages, risking the loss of useful representations due to task and distribution misalignment. BiSSL bridges this gap by formulating pre-training and downstream adaptation as a unified bilevel optimization (BLO) problem inserted as an intermediate phase. By coupling the objectives mathematically, BiSSL produces a pre-trained backbone initialization that is structurally primed for downstream fine-tuning, improving downstream accuracy across diverse computer vision benchmarks.

What Problem Does This Paper Address?

The classic recipe for transfer learning with SSL is a two-step dance:

  1. Train a backbone on a large unlabeled dataset using an unsupervised pretext task (e.g., contrastive learning like SimCLR).
  2. Attach a task-specific head and fine-tune on labeled downstream data.

However, because the pre-training data distribution often diverges from the downstream task, fine-tuning can trigger representational collapse or catastrophic forgetting, where rich semantic features learned during pre-training quickly vanish. Standard pipelines treat these two stages as separate silos. The authors ask: Can we explicitly model the interdependence between pretext learning and downstream adaptation so they inform each other before final fine-tuning begins?

Methods

The authors introduce BiSSL, an intermediate training stage that frames pretext learning and downstream fine-tuning as a constrained bilevel optimization problem:

  • Lower-Level Objective (Pretext Task): Optimizes the pretext loss $\mathcal{L}^P$ (SimCLR's NT-Xent) along with a quadratic regularization term $\frac{\lambda}{2} \|\theta_D - \theta_P\|_2^2$ that tethers the lower-level backbone $\theta_P$ to the downstream backbone parameters $\theta_D$.
  • Upper-Level Objective (Downstream Task): Evaluates downstream classification loss $\mathcal{L}^D$ using the optimal lower-level representation $\theta_P^*(\theta_D)$ (plus an auxiliary term for convergence stability).
                      ┌────────────────────────────────────────┐
                      │        Upper-Level (Downstream)        │
                      │  min  L_D(θ_P*(θ_D), φ_D) + γ L_D(...) │
                      └──────────────────┬─────────────────────┘
                                         │ Guides via
                 Implicit Gradient (CG)  │ Regularization
                                         ▼
                      ┌────────────────────────────────────────┐
                      │          Lower-Level (Pretext)         │
                      │   min  L_P(θ_P, φ_P) + λ ||θ_D - θ_P||²│
                      └────────────────────────────────────────┘

Key Technical Details:

  • Implicit Differentiation: Computing the gradient of the upper-level objective with respect to $\theta_D$ requires differentiating through the lower-level solution (the implicit gradient). Under local optimality and convexity assumptions, the authors derive this derivative in closed form using the Implicit Function Theorem.
  • Conjugate Gradient (CG) Approximation: Computing the inverse Hessian-vector product directly is intractable for deep networks. BiSSL employs an iterative, layer-wise CG method (using stored mini-batches of pretext data) to approximate these updates without explicitly materializing the Hessian matrix.
  • Training Pipeline: A brief pretext warm-up (500 epochs) and downstream linear head warm-up (20 epochs) initialize the parameters. BiSSL then alternates between lower- and upper-level updates for 500 alternations (roughly equivalent to 100 pretext epochs) before running standard full fine-tuning.

Datasets

All models were evaluated using a ResNet-18 backbone pre-trained on:

  • Pretext Pre-training: STL10 unlabeled partition (100,000 images, 96×96 resolution).

Evaluation spanned 14 downstream image classification benchmarks encompassing generic recognition, fine-grained classification, scenes, and textures:

  • STL10 (labeled split), CIFAR-10, CIFAR-100, Caltech-101, FashionMNIST, Food-101, SUN397, Describable Textures Dataset (DTD), Oxford-IIIT Pets, Oxford 102 Flowers, StanfordCars, FGVC Aircraft, CUB-200-2011 Birds, and PASCAL VOC 2007.

(No new benchmarks were introduced; all splits followed established literature conventions.)

Key Insights and Ideas

  • The Role of Regularization ($\lambda$): The regularization strength $\lambda$ acts as an intuitive knob between two regimes:
  • When $\lambda \to \infty$, the lower-level collapses to $\theta_P = \theta_D$, reducing BiSSL to standard fine-tuning.
  • When $\lambda \to 0$, the implicit gradient approaches zero, reducing the upper-level update to linear probing on a frozen pretext backbone. An intermediate value ($\lambda = 0.001$) achieves synergy between both objectives.
  • Proactive Feature Realignment: Qualitative t-SNE visualizations reveal that BiSSL clusters class-specific features prior to final fine-tuning, demonstrating that the bilevel loop successfully steers general pretext representations toward downstream class boundaries without overfitting.
  • Orthogonal to Pre-training Duration: Testing across varying pre-training budgets (200 to 1,000 epochs) on the Flowers dataset showed that BiSSL’s gains do not stem merely from extra gradient steps, but from a superior optimization trajectory.

Key Results

Compared to standard fine-tuning of an equivalent pretext pre-training duration:

  • Broad Performance Improvements: BiSSL significantly improved Top-1 accuracy, Top-5 accuracy, or 11-point mAP on 10 of the 14 datasets, with no dataset experiencing a statistically significant drop.
  • Notable Top-1 Gains:
  • Oxford-IIIT Pets: 67.8% vs. 65.0% (+2.8%)
  • Caltech-101: 80.6% vs. 78.1% (+2.5%)
  • DTD (Textures): 51.8% vs. 49.3% (+2.5%)
  • VOC 2007: 60.4% vs. 58.6% mAP (+1.8%)
  • Flowers: 74.8% vs. 73.4% (+1.4%)
  • CUB-200 Birds: 47.1% vs. 45.7% (+1.4%)

Limitations and Future Work

  • Computational Overhead: Using the Conjugate Gradient algorithm to compute iterative Hessian-vector products introduces non-trivial compute requirements compared to first-order fine-tuning.
  • Scale Limitations: Experiments were restricted to ResNet-18 on 96×96 resolution images. Whether the bilevel formulation scales gracefully to Vision Transformers (ViTs) and large-scale datasets (like ImageNet-1k) remains unproven.
  • Data Access Assumption: BiSSL requires continued access to the unlabeled pretext dataset during the adaptation stage, which may not always be practical when using proprietary or massive pre-training corpora.

Summary updated

Abstract

In this work, we present BiSSL, a first-of-its-kind training framework that introduces bilevel optimization to enhance the alignment between the pretext pre-training and downstream fine-tuning stages in self-supervised learning. BiSSL formulates the pretext and downstream task objectives as the lower- and upper-level objectives in a bilevel optimization problem and serves as an intermediate training stage within the self-supervised learning pipeline. By more explicitly modeling the interdependence of these training stages, BiSSL facilitates enhanced information sharing between them, ultimately leading to a backbone parameter initialization that is better suited for the downstream task. We propose a training algorithm that alternates between optimizing the two objectives defined in BiSSL. Using a ResNet-18 backbone pre-trained with SimCLR on the STL10 dataset, we demonstrate that our proposed framework consistently achieves improved or competitive classification accuracies across various downstream image classification datasets compared to the conventional self-supervised learning pipeline. Qualitative analyses of the backbone features further suggest that BiSSL enhances the alignment of downstream features in the backbone prior to fine-tuning.
BibTeX
@article{zakarias2025bissl,
  author = {Gustav Wagner Zakarias and Lars Kai Hansen and Zheng-Hua Tan},
  title = {{BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=GQAGlqOpyA},
}
Figure