Modern computer vision models possess an insatiable appetite for data. To train a high-performing Vision Transformer (ViT) or Convolutional Neural Network (CNN), conventional wisdom dictates scraping millions of real-world images from the internet. However, this web-scale curation model has hit a wall of "data debt." Web-scraped datasets are riddled with systemic biases, copyright infringements, and severe privacy violations.
To bypass these ethical and legal bottlenecks, a radical alternative has emerged: Formula-Driven Supervised Learning (FDSL). Instead of teaching a model to see using real-world photographs, FDSL trains models using entirely synthetic images generated from pure mathematical formulas—such as fractals, sinusoidal waves, and procedural contours.
Remarkably, models pre-trained on these abstract mathematical patterns perform competitively with, and sometimes outperform, models trained on massive human-annotated datasets like ImageNet.
Why It Works: The Geometry of Vision
At first glance, it seems counterintuitive that a model trained on abstract fractal patterns could successfully detect a real-world object like a car or a dog. The secret lies in understanding what a neural network actually learns during its foundational pre-training phase.
When a vision model is pre-trained, it is not primarily learning high-level semantic concepts (like "what a dog looks like"). Instead, it is learning how to see—developing a robust vocabulary of low- and mid-level visual primitives. Early layers of a network learn to detect edges, local orientations, and color gradients, while middle layers assemble these edges into textures, contours, and geometric shapes.
Mathematical formulas are uniquely suited to seed this visual vocabulary for two core reasons:
- Natural Statistical Alignment: Natural images possess a distinct mathematical property known as a $1/f^\alpha$ power spectrum, meaning they exhibit self-similarity and structural patterns across multiple scales. Fractals, by definition, are mathematically engineered to possess infinite self-similarity and complex boundary structures. As demonstrated by Kataoka et al., this geometric alignment allows a model trained on a fractal database (FractalDB) to naturally develop the same foundational feature extractors it would acquire from real-world imagery.
- Tiling the Feature Space: Human-curated datasets are fundamentally limited by what humans choose to photograph, introducing massive gaps and subjective biases. In contrast, procedural generation allows researchers to systematically tile the entire space of visual primitives. By varying mathematical parameters, an algorithm can generate an infinite variety of orientations, frequencies, and curvatures without any human intervention.
[Mathematical Formula] ──> [Procedural Generation] ──> [Abstract Visual Primitives] ──> [Robust Low-Level Features]
(e.g., Iterated Function Systems) (Infinite, Clean Scaling) (Edges, Contours, Textures) (Ready for Real-World Transfer)
From Fractals to Waves: Engineering the Primitives
The initial breakthrough in FDSL relied on Iterated Function Systems (IFS) to generate complex 2D fractal patterns. Kataoka et al. proved that classifying these mathematical structures allows CNNs to learn highly transferable representations. Later, Kataoka et al. extended this paradigm to Vision Transformers, demonstrating that architectures like Swin Transformers and ConViT achieve performance parity with ImageNet-1k pre-training when exposed to fractal geometry.
However, fractals are not the only mathematical structures capable of teaching a model to see. Researchers have successfully stripped the input data down to even simpler mathematical components:
- Sinusoidal Waves: Takashima et al. introduced VisualAtom-21k, a dataset built entirely from basic sinusoidal waves. By pre-training a ViT on these elementary wave patterns, the model achieved 83.7% accuracy on ImageNet-1k—coming within 0.5% of a model pre-training on Google’s massive, private, and unreleased JFT-300M dataset. This proved that high-level semantic textures are entirely unnecessary for building a powerful visual backbone.
- Procedural Contours: For tasks requiring precise spatial awareness, such as semantic segmentation, standard fractals can lack clean boundary definitions. To solve this, Kataoka et al. and Shinoda et al. developed SegRCDB, which generates datasets based on mathematical contours. By training models to segment these procedurally generated boundaries, the authors achieved higher accuracy on real-world benchmarks like ADE-20k and Cityscapes than models pre-trained on human-annotated datasets like COCO-Stuff.
Overcoming the Biases of the Real World
Beyond solving privacy and copyright issues, formula-driven data inherently fixes several technical flaws embedded within real-world datasets.
Eliminating Scene Bias in Video
In video action recognition, models trained on real-world datasets (like Kinetics) frequently cheat. If a model wants to recognize the action "swimming," it often bypasses the actual motion and simply looks for a blue, watery background. This is known as scene bias.
Synthetic data forces the model to learn the true underlying physics of motion. Kim et al. demonstrated that synthetic video representations transfer exceptionally well because they lack confounding background correlations. Taking this a step further, Ishikawa et al. developed a Pseudo Motion Generator (PMG) that animates static, synthetic images. By pre-training video transformers on these mathematically animated frames, they bypassed real-world video collection entirely while maintaining high downstream recognition accuracy.
Cross-Modal Fusions
FDSL also simplifies multi-modal learning. Fusing 2D images with 3D point clouds is notoriously difficult due to the structural differences between real-world photos and laser scans. Yamada et al. introduced Formula-Supervised Visual-Geometric Pre-training (FSVGP), using 3D mathematical formulas to generate perfectly aligned 2D images and 3D point clouds simultaneously. Because the underlying math dictates both the texture and the structure, the model learns a flawless cross-modal mapping without human annotation errors.
Extreme Efficiency: On-the-Fly and Ultra-Lean Data
Traditional pre-training requires storing terabytes of images on physical disks, leading to massive I/O bottlenecks during training. Because FDSL datasets are governed by compact mathematical formulas, they unlock entirely new paradigms of computational efficiency.
Infinite Data on the Fly
Instead of saving images to a disk, Yokota et al. engineered a GPU-accelerated pipeline that renders fractal images in real-time directly into the training loop. This eliminates disk read times entirely, yielding a 3x speedup in training throughput for massive 100-million-image datasets.
The One-Instance Phenomenon
Perhaps the most surprising property of FDSL is its extreme data efficiency. Kataoka et al. discovered that Vision Transformers do not actually need millions of unique synthetic images. By creating a One-instance Fractal Database (OFDB)—which contains exactly one mathematically generated image per category—and applying aggressive data augmentations, they matched the performance of models pre-trained on ImageNet-21k. This reduced the required pre-training data footprint by a factor of 600x.
| Pre-training Dataset | Image Source | Data Footprint | Downstream Performance | Ethical/Legal Risk |
|---|---|---|---|---|
| ImageNet-21k | Web-Scraped Real Images | Massive (~14M images) | Baseline High | High (Privacy, Copyright, Bias) |
| StableRep (Tian et al.) | Generative AI (Stable Diffusion) | Medium-Large | Excellent | Inherited from base model |
| FractalDB (Kataoka et al.) | Pure Mathematics (IFS Fractals) | Zero (Generated on-the-fly) | Competitive / Superior | Zero |
| OFDB (Kataoka et al.) | 1 Math Instance + Augmentation | Ultra-Lean (600x reduction) | Parity with ImageNet-21k | Zero |
The Broader Horizon
While generative AI models can also produce synthetic training data (as seen in frameworks like StableRep by Tian et al.), those models are themselves trained on web-scraped data, meaning they inherit the same underlying copyright and bias liabilities. As highlighted by Rupprecht et al., automated filtering pipelines cannot fully cleanse real-world data repositories of these inherited ethical risks.
Formula-Driven Supervised Learning offers a clean break from this cycle. By proving that the fundamental building blocks of visual intelligence can be derived from pure geometry, FDSL provides a mathematically rigorous, infinitely scalable, and ethically unassailable foundation for the future of computer vision.
Key Foundational Work
Replacing Labeled Real-image Datasets with Auto-generated Contours (2022) — contributed 1 concept elements to this direction.
Improving Fractal Pre-Training (2022) — contributed 1 concept elements to this direction.
Pre-Training Without Natural Images (2022) — contributed 1 concept elements to this direction.