AgentSSL: Can MLE Agents Leverage Unlabeled Data?

Ethan Lin, Akanksha Sarkar, Ziang Liu, Kristin Branson, Jennifer J. SunarXiv 2026 · arXiv

TL;DR

AgentSSL introduces a framework for LLM-based Machine Learning Engineering (MLE) agents to autonomously discover and adapt Semi-Supervised Learning (SSL) strategies for data-scarce regimes. While these agents can synthesize programs that outperform state-of-the-art human-engineered baselines by up to 21.8%, the study identifies program selection—choosing the best candidate without a large validation set—as a critical bottleneck that can cause performance to plummet by nearly 50%.

What Problem Does This Paper Address?

Modern machine learning is trapped between two expensive bottlenecks: the high cost of human-labeled data and the intensive human engineering required to design specialized pipelines. While MLE agents have recently shown a knack for automating pipeline design, they have mostly been tested in "data-rich" environments where labels are abundant.

In the real world—think medical imaging or ecological monitoring—labels are rare, but unlabeled data is plentiful. Semi-Supervised Learning (SSL) is the standard answer here, but SSL methods are notoriously "brittle." A technique that works for CIFAR-10 might fail miserably on satellite imagery because the underlying data assumptions (like cluster structure or augmentation invariance) don't hold. This paper asks: Can an autonomous agent navigate the complex design space of SSL to find the right strategy for a specific, data-starved task?

Figure 1: MLE Agent research is focused on reduc- ing engineering cost, but relies on human labeled data. SSL research seeks to reduce annotation effort but re- quires human engineering to deploy. We study whether AgentSSL can reduce both costs.

Methods

The authors frame SSL as an agentic program search problem called AgentSSL. The process is divided into two distinct phases:

  1. Phase I: Program Discovery: An LLM-based agent (using backbones like GPT-4o) iteratively writes, debugs, and improves Python code. It explores a vast library of SSL concepts—pseudo-labeling, consistency regularization, graph-based methods, and more. Because it lacks a large validation set, the agent relies on "weak feedback": either an ensemble of unsupervised metrics (like Silhouette scores or RankMe) or noisy validation (a tiny, high-variance labeled split).
  2. Phase II: Program Selection: Once the agent has generated a pool of candidate programs, it must pick the best one for deployment. The authors test both unsupervised selection and active selection, where a very small number of additional labels are used specifically to evaluate the candidates rather than train them.
Figure 2: Overview of AgentSSL. Given a small labeled set and a larger unlabeled corpus, an LLM-based agent generates SSL programs that are evaluated using either unsupervised SSL metrics or noisy validation feedback, with the resulting fitness score guiding iterative refinement. Following search, we must select the best program either using unsupervised metrics or active model selection. We evaluate AgentSSL on both image classification SSL benchmarks and real-world ecological detection tasks.

Datasets

The study evaluates AgentSSL across a diverse range of tasks:

  • Image Classification: Six datasets from the VTAB benchmark, including textures (DTD), satellite imagery (RESISC45), and medical scans (Retinopathy).
  • Ecology Detection: Real-world tasks involving fish behavior (WildFin) and antelope tracking (BuckTales).
  • ProgramSelectBench: A massive new benchmark introduced in this paper. it contains 16,977 precomputed SSL programs (both agent-generated and human-engineered) representing over 10,000 GPU hours. This serves as a "meta-benchmark" for researchers to develop better model selection algorithms.

Key Insights and Ideas

  • Agents are "SSL Polyglots": Unlike human researchers who might favor a specific paradigm (e.g., "I only use FixMatch"), agents freely mix and match decades of SSL research. They often discover hybrid strategies that combine graph-based label propagation with modern consistency training.
  • Domain-Specific "Aha!" Moments: For the Retinopathy task, the agent discovered that it shouldn't just train a standard classifier. Instead, it combined frozen DINOv2 features with an explicit "ordinal bias" to account for the ordered stages of disease severity—a nuance a generic SSL recipe would miss.
  • The Selection Bottleneck: The biggest surprise was the "Selection Gap." Agents are actually too good at discovery; they find excellent programs, but the unsupervised metrics used to rank them are often poorly correlated with true test accuracy. If you pick the wrong program, you can lose up to 47% in accuracy.
  • Labels for Selection > Labels for Training: In low-data regimes, the authors found that if you have a tiny budget of extra labels (e.g., 25), it is often more effective to use them to select the best program from the agent's pool than to use them to train the model further.

Key Results

  • Beating the Pros: AgentSSL outperformed the strongest human-engineered baselines (like FixMatch and FineSSL) in 8 out of 12 experimental settings.
  • Massive Gains: In the Retinopathy dataset, the agent-discovered program improved accuracy by +21.8% over the next-best human baseline.
  • Scaling with Unlabeled Data: The programs discovered by agents actually "know" how to use more data; as the unlabeled corpus grows, the accuracy of the agent's programs increases even while the labeled set stays fixed.
Oracle test performance shows AgentSSL (purple) frequently outperforming human-engineered baselines (yellow).

Limitations and Future Work

  • Unreliable Feedback: The core weakness remains the lack of a reliable, label-free "compass" to guide the agent. When unsupervised metrics fail, the agent can wander into low-performance areas of the search space.
  • Compute Intensity: While the authors argue that GPU time is cheaper than human engineering time, running thousands of candidate programs is still computationally expensive.
  • Real-World Instability: In the ecology detection tasks, the agents showed higher variance and sometimes struggled to beat expert-tuned YOLOv8 baselines, suggesting that transferring these strategies to complex detection tasks is harder than simple classification.
  • Future Direction: The authors suggest that future agents should be "meta-aware," dynamically deciding how to split their limited label budget between guiding the search and performing the final selection.

Summary updated

Abstract

Applying machine learning systems to new domains introduces bottlenecks in both labeled data acquisition and manual pipeline engineering. While recent MLE agents show promise in automating pipeline design, they have been studied almost exclusively in fully supervised settings, leaving them untested in realistic regimes where labeled data is scarce but unlabeled data is abundant. In this work, we introduce Agentic Semi-Supervised Learning (AgentSSL), framing SSL as a program search problem and studying how agents navigate this space under limited supervision. We find that even under weak feedback and matched compute, agents can synthesize code that combines and adapts established SSL techniques, exceeding the strongest human-engineered baselines in 8 of 12 settings by up to +21.8%. However, our results identify program selection as a critical bottleneck in AgentSSL, where performance drops by as much as 47% when selecting based on the agent's feedback metric. To systematically study this bottleneck, we introduce ProgramSelectBench, a large-scale benchmark with a combined total of 16,977 agent-generated and human-engineered SSL programs, and evaluate both unsupervised and active program selection methods. We further evaluate AgentSSL on real-world ecology detection tasks, highlighting the opportunities and challenges of transferring agent-discovered strategies beyond controlled benchmarks.
BibTeX
@misc{lin2026agentssl,
  author = {Ethan Lin and Akanksha Sarkar and Ziang Liu and Kristin Branson and Jennifer J. Sun},
  title = {{AgentSSL: Can MLE Agents Leverage Unlabeled Data?}},
  year = {2026},
  eprint = {2611.17323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17323},
}
Figure