For years, the standard recipe for training large machine learning models was simple: scrape as much data from the internet as possible. However, this "black-box" approach has run into severe legal, ethical, and practical headwinds. Indiscriminate web-scraping routinely ingests copyrighted books, proprietary source code, and personal data without explicit authorization.
As Kirchhübel et al. point out in their analysis of intellectual property frameworks, the rapid development of large language models (LLMs) has completely outpaced existing legal structures, leaving massive ambiguity regarding copyright liability across the entire model lifecycle (Intellectual property rights at the training, development and generation stages of Large Language Models). A key point of contention is whether Text and Data Mining (TDM) exceptions—which often protect non-commercial or scientific research—apply to commercial model training. The authors argue that the scope of these exceptions remains highly uncertain and legally untested for commercial deployment.
Furthermore, as Almeida et al. note, the licenses of underlying training sets can downstream restrictions that directly impact how the resulting models themselves can be legally licensed (A Legal Framework for Natural Language Model Training in Portugal). This legal friction creates what Meyers 0001 et al. describe as a "chilling effect," where commercial entities are deterred from adopting or contributing to open-source AI resources due to fears that a trained model might be legally classified as an unauthorized derivative work (Shared Corpora Working Group Report). Beyond legal liability, Kandpal et al. emphasize that ignoring the immense human labor required to create high-quality data obscures the true economic cost of AI development, arguing that data compensation should fundamentally scale with its value (Position: The Most Expensive Part of an LLM should be its Training Data).
To resolve these liabilities, a major research direction has emerged: proving that state-of-the-art models can be trained exclusively on legally clean, ethically sourced, or synthetically generated data without sacrificing performance.
Challenging the Performance Tradeoff
The historical justification for using legally precarious data was performance; it was widely assumed that restricting training data to strictly permissible sources would cripple a model's downstream capabilities. Recent research fundamentally challenges this assumption by demonstrating that high-quality curation can compensate for a smaller data footprint.
In the domain of source code, Kocetkov et al. demonstrated with The Stack—a 3.1 TB dataset of source code restricted entirely to permissive licenses—that models trained on clean data can match or even outperform those trained on uncurated, black-box datasets (The Stack: 3 TB of permissively licensed source code). By prioritizing near-deduplication and developer agency (such as implementing opt-out mechanisms), the authors proved that ethical data sourcing does not require a tradeoff in model capability.
This principle extends directly to natural language. Langlais et al. introduced Common Corpus, a 2-trillion-token dataset composed entirely of public domain and permissively licensed content (Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training). Their work shows that LLMs pre-trained on legally clean data perform on par with industry models trained on copyrighted material, offering a highly reproducible and transparent alternative for open-source AI research.
Why Clean Data Works: Quality over Quantity
Why can smaller or more restricted datasets compete with the entire uncurated web? The answer lies in data density, deduplication, and strategic composition. Unfiltered web scrapes are notoriously noisy, containing massive amounts of repetitive, low-quality, or irrelevant text. By filtering strictly for public domain or permissively licensed data, researchers can execute rigorous curation that eliminates this overhead.
For instance, Kandpal et al. released The Common Pile v0.1, an 8TB dataset of public domain and openly licensed text (The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text). When they trained their 7B-parameter Comma v0.1 models on this data, the models rivaled or outperformed prominent industry standards like Llama 1 and 2, particularly in technical and coding domains.
To maximize the utility of clean data, researchers are also rethinking how data is introduced during pretraining. Nguyen et al. developed MixtureVitae, a dataset that pairs a permissive-first sourcing strategy with "front-loaded" high-quality synthetic instruction and reasoning data (MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources).
By embedding reasoning structures directly into the initial pretraining phase rather than saving them for downstream fine-tuning, the authors demonstrated that models can achieve advanced reasoning capabilities while requiring significantly less total data volume. This dual approach—combining clean public data with high-fidelity synthetic data—bypasses the legal risks highlighted by downstream processing tasks, such as those explored by Ko et al. in scientific data extraction (Filling in the Gaps: LLM-Based Structured Data Generation from Semi-Structured Scientific Data).
Bypassing Real Data via Procedural Generation
If sourcing clean text requires careful curation, sourcing clean image datasets for computer vision presents an even steeper hurdle due to complex licensing, biometric privacy, and identity constraints. An alternative, radical approach to this problem is to eliminate the reliance on real-world data altogether through mathematical procedural generation.
Kataoka et al. pioneered this via Formula-driven Supervised Learning (FDSL), creating FractalDB (Pre-Training Without Natural Images). Instead of pre-training convolutional neural networks (CNNs) on millions of web-scraped natural images (like ImageNet), FDSL trains models entirely on automatically generated fractal patterns. Because fractals are generated purely from mathematical formulas, they carry zero copyright, privacy, or ethical liabilities.
[Mathematical Formula] ──> [Procedural Fractal Generation] ──> [FractalDB Images] ──> [CNN Pre-training]
Crucially, this approach works because the underlying mathematical properties of fractals—such as self-similarity, edge configurations, and structural complexity—mirror the fundamental visual features found in natural scenes. Subsequent advancements in Improving Fractal Pre-Training introduced multi-instance prediction tasks and refined generation techniques, allowing models pre-trained purely on synthetic fractals to match or exceed the performance of models pre-trained on massive human-annotated datasets.
Towards Sustainable Data Governance
Shifting the AI ecosystem away from legally precarious web-scraping requires more than just technical ingenuity; it demands robust, multi-party frameworks to manage data transparently.
As Jernite et al. argue, the long-term sustainability of large-scale language technology depends on structured data governance that balances stakeholder rights, legal compliance, and technical utility (Data Governance in the Age of Large-Scale Data-Driven Language Technology). By establishing explicit provenance, respecting creator consent through opt-out mechanisms, and proving the efficacy of public domain and synthetic alternatives, this research direction provides a scalable, legally robust blueprint for the future of machine learning.
Key Foundational Work
Shared Corpora Working Group Report (2007) — contributed 2 concept elements to this direction.
Pre-Training Without Natural Images (2022) — contributed 1 concept elements to this direction.
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022) — contributed 1 concept elements to this direction.