Abstract
Dataset license compliance is a critical yet complex aspect of developing commercial AI products, particularly with the increasing use of publicly available datasets. Ambiguities in dataset licenses pose significant legal risks, making it challenging even for software IP lawyers to accurately interpret rights and obligations. In this paper, we introduce LicenseGPT, a fine-tuned foundation model (FM) specifically designed for dataset license compliance analysis. We first evaluate existing legal FMs and find that the best-performing model achieves a Prediction Agreement (PA) of only 43.75%. LicenseGPT, fine-tuned on a curated dataset of 500 licenses annotated by legal experts, significantly improves PA to 64.30%, outperforming both legal and general-purpose FMs. Through an A/B test and user study with software IP lawyers, we demonstrate that LicenseGPT reduces analysis time by 94.44%, from 108 seconds to 6 seconds per license, without compromising accuracy. Software IP lawyers perceive LicenseGPT as a valuable supplementary tool that enhances efficiency while acknowledging the need for human oversight in complex cases.
Introduction
The rapid expansion of AI development has turned dataset licensing into a "legal wild west." Unlike standardized Open Source Software (OSS) licenses, dataset licenses are frequently ambiguous, non-standardized, or obscured within complex "Terms of Use" pages. Research indicates that over 66% of dataset licenses are misrepresented on public platforms, often appearing more permissive than their actual legal terms allow. For software IP lawyers, determining commercial viability is a slow, manual process prone to error. While general Large Language Models (LLMs) excel at text summarization, they often struggle with the nuanced logic required for legal compliance, providing professional-sounding but legally inaccurate responses.
Methodology
To address these challenges, we developed LicenseGPT, a specialized foundation model fine-tuned for dataset license compliance. The development process followed a structured pipeline involving data collection, model selection, and expert validation.
Data Collection and Annotation
We introduced the DL (Dataset Licenses) dataset, comprising 500 valid licenses from 10 major platforms including Hugging Face, GitHub, and Kaggle. These were categorized into General Licenses, Customized Licenses, and Official Terms of Use. Four software IP lawyers with 8–20 years of experience manually labeled each license for commercial use permissions and the underlying legal reasoning.
Model Development
We selected LawGPT (a 7B parameter model based on ChatGLM) as the base model due to its existing legal knowledge and local hosting capabilities. Fine-tuning was performed using LoRA (Low-Rank Adaptation) to efficiently adapt the model to specific dataset license terminology. We also developed a modular prompt design, separating system personas from specific user queries to optimize performance.
Results and Analysis
LicenseGPT significantly outperformed both general-purpose and existing legal foundation models in accuracy and efficiency.
Performance Comparison and the "Fluency Trap"
LicenseGPT achieved a Prediction Agreement (PA) of 64.30%, a substantial improvement over the base LawGPT (43.75%) and general models like Qwen-1.5 (59.72%) and GPT-4 (18.06%).
A key insight from our evaluation is the "Fluency vs. Accuracy" trap. As shown in the performance table, general models like GPT-4 (CGPT4) achieved very high Semantic Similarity (SS) scores (~94.80%) but very low Prediction Agreement (PA) (~18.06%). This indicates that while GPT-4 produces text that sounds like a lawyer, its legal conclusions are frequently incorrect. LicenseGPT prioritizes logical accuracy, reversing this trend to provide more reliable legal judgments.
Efficiency and Expert Perception
In a user study with professional IP lawyers, LicenseGPT reduced the average analysis time per license from 108 seconds to just 6 seconds—a 94.44% increase in efficiency. Lawyers perceived the tool as a "high-speed filter" or a digital "intern" capable of handling initial passes, allowing them to focus on complex, ambiguous cases.
Conclusion
LicenseGPT demonstrates the power of domain-specific fine-tuning in the legal sector. While its 64.3% accuracy rate indicates it is not a standalone replacement for human expertise, it serves as a powerful supplementary tool for increasing efficiency in dataset compliance workflows. Future work will explore scaling to larger base models (70B+) and advocating for machine-readable license standards like SPDX 3.0 to further reduce legal ambiguity in AI development.
Summary updated