Abstract
Traditional AutoML platforms rely on centralized, single-strategy optimization that often fails to explore the full hyperparameter landscape. We present Gradients, a decentralized marketplace built on the Bittensor network that incentivizes independent miners to compete in fine-tuning tournaments. Our results demonstrate that this market-driven approach consistently outperforms major commercial platforms, achieving an average improvement of 42.1% over centralized cloud providers and a 100% win rate against TogetherAI.
Introduction
Fine-tuning foundation models is a non-trivial task where optimal configurations are highly dependent on model scale and task domain. Centralized platforms like HuggingFace AutoTrain or TogetherAI typically employ standard optimization techniques (e.g., Bayesian search) that may settle for local optima. We argue that these methods lack the diversity and incentive structures required to find truly optimal configurations for specialized tasks.
Methods: The Gradients Marketplace
Gradients leverages the Bittensor network to create a competitive ecosystem for model optimization. Instead of a single algorithm searching for a solution, the system creates a marketplace:
- Tournament Structure: Tasks are broadcast to a pool of autonomous miners who have total freedom to choose optimization strategies, including learning rate schedules, batch sizes, and data augmentation.
- Incentivization: Miners are rewarded based on model performance on held-out test sets, with penalties for the bottom 25% of performers to ensure resource efficiency.
- Strategy Diversity: By keeping winning configurations proprietary, the system prevents strategic collapse and encourages miners to explore unconventional hyperparameter spaces, such as high-rank LoRA variants (e.g., rsLoRA with ranks 64-128).
Experimental Results
We evaluated Gradients across 180 experiments involving model families such as Llama-3 and DeepSeek-R1. Tasks included mathematical reasoning, translation, code generation, and Retrieval-Augmented Generation (RAG).
As illustrated in Figure 1, Gradients achieved significant performance gains across all categories. Notably:
- Win Rates: The system maintained a 100% win rate against TogetherAI and Databricks, and an 82.8% win rate against HuggingFace AutoTrain.
- Task-Specific Gains: The most substantial improvements were observed in RAG (33.3% vs. HF) and reasoning tasks, where standard configurations often underperform.
- Overall Improvement: On average, Gradients improved performance by 11.8% over HuggingFace and 42.1% over other commercial competitors.
Discussion and Conclusion
The success of Gradients suggests that economic competition can serve as a powerful optimization engine, often discovering effective configurations that contradict standard theoretical guidance. While the parallel nature of the marketplace increases computational overhead (running 8-25 parallel training runs per task), the resulting performance gains are substantial for high-impact applications. Future work will investigate balancing this overhead through adaptive competition mechanisms and exploring controlled transparency to share architectural insights.
Summary updated