TL;DR
Researchers have introduced Model Maps, a documentation framework designed to untangle the complex "ancestry" of foundation models. By mapping 2,778 dependencies across 15 major model families, the study reveals that even "open" models are often deeply entangled with closed-source upstream artifacts, creating hidden risks for bias, security, and legal compliance.
What Problem Does This Paper Address?
Modern AI development doesn't happen in a vacuum. Developers rarely build models from scratch; instead, they "entangle" their work with pre-existing artifacts—fine-tuning on specific datasets, using one model to filter the training data for another, or generating synthetic data to train a smaller "distilled" version.
The authors call this phenomenon AI Entanglement. The problem is that current documentation (like Model Cards or Datasheets) usually only looks at a single model in isolation. If an upstream dataset has a privacy leak or an upstream model has a specific hallucination pattern, those flaws can propagate down the supply chain. Without a way to trace this lineage, developers and regulators are left in the dark regarding the root causes of model failure or bias.
Methods
The authors propose Model Maps, a conceptual framework that treats model ancestry as a directed knowledge graph.
- The Taxonomy: They developed a novel taxonomy of 13 dependency mechanisms grouped into four categories:
- Data-based: Composition, training data, and benchmark evaluation.
- Model-based curation: Using models for data generation, annotation, filtering, or as safety guardrails.
- Model-based training: Adaptation (fine-tuning), preference learning (RLHF), and distillation.
- Model-based deployment: Collaboration (multi-agent), output guardrails, and model-as-a-judge evaluation.
- Manual Construction: To ensure high fidelity, the researchers manually constructed maps for 15 foundation model families by scouring technical reports and papers.
- Justification Analysis: They used qualitative "open coding" to analyze why developers chose specific upstream dependencies, identifying the core values driving the AI supply chain.
Datasets
The paper itself does not introduce a new training dataset for machine learning. Instead, it provides a meta-dataset of AI relationships. They analyzed 15 prominent model families, including:
- Closed-weight: GPT-4, Claude 4, Gemini 2.5, Apple FM.
- Open-weight: Llama 3, Mistral 7B, OLMo 3, DeepSeek-V3, Phi 4.
Through these case studies, they recorded 1,161 unique artifacts (datasets like RedPajama, websites like Wikipedia, and other models) and 2,778 unique dependency edges.
Key Insights and Ideas
- The "Open" Paradox: A major finding is that many "open-source" or open-weight models are heavily dependent on closed-source ancestors. For example, many open models use synthetic data generated by GPT-4 for instruction tuning, meaning the "open" model inherits the behaviors (and biases) of a proprietary black box.
- Deep Ancestry: Dependencies aren't just one level deep. The researchers traced artifacts up to 9 hops away from the target model. A model might be trained on a dataset, which was filtered by a second model, which was fine-tuned from a third model, and so on.
- The OpenAI Bottleneck: OpenAI appears as a "common ancestor" across a vast majority of the models studied, highlighting a significant centralization of influence in the AI ecosystem.
- Feedback Loops: The maps reveal "iterative feedback loops" where models are evaluated by "judges" that are actually earlier versions of themselves, or trained on data they themselves generated, raising concerns about "model collapse."
Key Results
- Complexity Scale: The map for OLMo 3 was the largest, consisting of 681 nodes and 1,357 edges.
- Transparency Gaps: Closed-weight models had significantly smaller maps not because they are simpler, but because their developers disclose far fewer dependencies.
- Justification Trends: When developers justify using an upstream artifact, they most frequently cite Generalization (e.g., "diverse data") and Data Quality (e.g., "high-quality filtering"). Interestingly, "Safety" was one of the least cited reasons for choosing a dependency.
- Resource Release: The authors have launched modelmaps.ai, a public resource to help the community explore these entanglements.
Limitations and Future Work
- Manual Effort: Constructing these maps is incredibly labor-intensive. Mapping OLMo 3 alone took several weeks of expert manual labor. Future work is needed to develop automated or semi-automated tools to help scale this process.
- Information Asymmetry: The framework relies on public disclosure. If a company does not disclose that they used a competitor's model for data filtering, that link remains invisible.
- Static vs. Dynamic: The current maps are largely static snapshots. As models are updated or APIs change, the maps need to evolve.
- Policy Integration: The authors suggest that "Model Maps" should become a mandatory part of regulatory disclosures (like the proposed AI Bill of Materials) to ensure accountability in the AI supply chain.
Summary updated