Geometric Gradient Rectification for Safe Open-Set Semi-Supervised Learning

Jiahe Chen, Qian Shao, Qiyuan Chen, Jiaying He, Jintai Chen, Jian Wu, Hongxia XuarXiv 2026 · arXiv

TL;DR

Open-Set Semi-Supervised Learning (OSSL) algorithms often struggle with a fundamental trade-off: filtering out out-of-distribution (OOD) data too aggressively discards hard in-distribution samples, while keeping them introduces harmful gradients. This paper introduces Geometric Gradient Rectification (GGR), a plug-in optimization framework that projects conflicting unsupervised gradients onto a safe half-space anchored by the trusted supervised gradient. GGR consistently improves both closed-set generalization and open-set robustness across standard benchmarks without requiring explicit OOD detection.

What Problem Does This Paper Address?

Semi-supervised learning (SSL) is highly effective at scaling up training by leveraging large pools of unlabeled data. However, real-world deployments rarely enjoy clean data; unlabeled streams are frequently contaminated with Out-of-Distribution (OOD) outliers. This challenge is known as Open-Set SSL (OSSL).

Existing OSSL methods generally fall into two camps: filtering-based (which reject suspicious samples) and utilization-based (which incorporate unlabeled data using soft weights or auxiliary classes). The authors identify a recurring dilemma between these paradigms:

  1. Feature Starvation: Aggressive filtering methods struggle to distinguish between actual OOD noise and hard In-Distribution (ID) samples, mistakenly throwing away highly informative ID data.
  2. Noise Domination / Active Unlearning: Utilization methods risk misclassifying hard ID samples or forcing OOD data into incorrect categories, generating auxiliary gradients that directly oppose and degrade the core supervised learning signal.

Instead of trying to fix this via brittle, sample-level categorical decisions, this paper shifts the focus entirely to gradient-level control.

Fig. 1: Left: OSSL dilemma. A trade-off between robustness and coverage that can lead to either feature starvation or noise domination. Right: Gradient geometry. Density of cosine similarity between unlabeled gradients and the oracle supervised direction. OOD: mostly orthogonal noise; Hard ID: misclassified samples induce adversarial gradients; Rectified: GGR projects conflicting components into an anchor- aligned subspace, shifting updates toward the constructive space.

Methods

GGR acts as a post-hoc, plug-and-play optimization constraint. It treats the supervised gradient ($g_s$) as a protected "anchor" and modifies the auxiliary unsupervised gradient ($g_u$) so that it never directionally opposes the anchor. Geometrically, it enforces a first-order non-opposition constraint ($\langle \tilde{g}_u, g_s \rangle \ge 0$) by projecting $g_u$ onto a safe half-space.

The authors propose three distinct rectifiers within this framework:

  1. Vector-Level Rectifier (VLR): The default option. It computes the instantaneous supervised gradient of the mini-batch and subtracts any opposing component from the auxiliary gradient, leaving orthogonal components intact to preserve useful representation signals.
  2. Orthogonal Subspace Rectifier (OSR): To combat mini-batch noise, this variant maintains a running, compact orthonormal basis of recent supervised gradients and completely removes any component of the auxiliary gradient that falls within this supervised subspace.
  3. Conic Subspace Rectifier (CSR): A less restrictive subspace variant that performs a cone projection, clipping only the coordinates that negatively conflict with the stored supervised anchors while retaining positively aligned components.

Crucially, GGR operates entirely in the gradient space during the backward pass. It requires no changes to the forward pass, network architecture, or loss functions of the underlying OSSL baseline.

Datasets

The framework was evaluated on standard image classification benchmarks adapted for open-set scenarios:

  • CIFAR-10
  • CIFAR-100
  • ImageNet-30 (a 30-class subset of ImageNet)

The authors evaluated performance across multiple challenging settings by varying the seen/unseen class splits and restricting the number of labeled samples per class down to extreme scarcity (e.g., 5 labels per class).

Key Insights and Ideas

  • OOD vs. Hard ID Gradient Geometry: The authors reveal a key geometric insight: gradients from OOD outliers behave predominantly like orthogonal stochastic noise. In contrast, misclassified hard ID samples are directionally destructive, inducing strong adversarial gradients that actively fight the supervised objective.
  • Asymmetric Gradient Surgery: Traditional gradient surgery techniques (like PCGrad) treat tasks symmetrically. GGR argues that OSSL requires an asymmetric approach: because labeled data provides ground-truth verification, the supervised gradient must remain completely unaltered while only the auxiliary gradient is rectified.
  • Bypassing the Selection Bottleneck: By managing the optimization dynamics directly rather than relying on imperfect sample filtering, GGR allows models to safely harvest representation signals from ambiguous data.

Key Results

When evaluated as a plug-in module on top of representative OSSL baselines (OpenMatch, IOMatch, and DAC), GGR delivered consistent improvements:

  • Accuracy Boosts: GGR improved closed-set accuracy by up to +2.00 percentage points and open-set balanced accuracy by up to +1.83 points in low-label regimes.
  • ImageNet Scaling: On the ImageNet-30 benchmark with only 1% labeled data, adding GGR to the DAC baseline improved closed-set accuracy from 86.40% to 87.40% and open-set balanced accuracy from 77.78% to 78.40%.
  • Low Overhead: The default VLR variant introduces a modest time overhead of around 11% and minimal memory increase, making it highly practical for real-world pipelines.

Limitations and Future Work

  • Local, First-Order Constraints: GGR ensures that the auxiliary update does not oppose the supervised descent direction at the current iteration, but it does not guarantee global optimality or a monotonic decrease of the full composite objective over time.
  • Anchor Noise Sensitivity: If the mini-batch supervised gradient itself becomes exceptionally noisy or uninformative (due to extreme label scarcity), the vector-level anchor can become unstable. While the OSR and CSR subspace variants mitigate this through temporal smoothing, they introduce extra hyperparameters (subspace dimension $d$).
  • Complementary Nature: GGR is an optimization control layer, not a replacement for open-set data modeling. The authors note that combining GGR with more advanced OOD detection or uncertainty estimation techniques remains a promising direction for future work.

Summary updated

Abstract

Open-set semi-supervised learning aims to leverage unlabeled data that may contain out-of-distribution outliers while maintaining performance on in-distribution classes. Existing methods mainly follow two paradigms: filtering suspicious samples or incorporating unlabeled objectives with soft weighting. We argue that both face a common trade-off: aggressive filtering can discard informative but hard ID samples, whereas utilization can introduce auxiliary gradients that conflict with supervised learning when pseudo labels are wrong. We therefore shift the focus from sample selection to gradient-level control. We propose \textit{Geometric Gradient Rectification} (GGR), a plug-in framework that uses the supervised gradient as an anchor and projects conflicting auxiliary gradients onto an admissible region in gradient space. This makes the applied auxiliary update first-order non-opposing within the rectified coordinate block while preserving orthogonal components that may still carry useful representation signals. We further extend GGR with subspace-aware rectification to stabilize the anchor under noisy mini-batch gradients. Experiments on CIFAR and ImageNet benchmarks show that GGR improves representative OSSL baselines in most settings and yields gains in both closed-set generalization and open-set robustness. Code will be available at https://github.com/JiaheChen2002/GGR.
BibTeX
@misc{chen2026geometric,
  author = {Jiahe Chen and Qian Shao and Qiyuan Chen and Jiaying He and Jintai Chen and Jian Wu and Hongxia Xu},
  title = {{Geometric Gradient Rectification for Safe Open-Set Semi-Supervised Learning}},
  year = {2026},
  eprint = {2606.26973},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.26973},
}
Figure