LicenseGPT: A Fine-tuned Foundation Model for Publicly Available Dataset License Compliance

Jingwen Tan, Gopi Krishnan Rajbahadur, Zi Li, Xiangfu Song, Jianshan Lin, Dan Li, Zibin Zheng, Ahmed E. Hassan

TL;DR

Navigating the complex world of dataset licenses for AI development is a major hurdle, often leading to legal risks and costly rework. This paper introduces LicenseGPT, a fine-tuned foundation model that significantly improves the accuracy and speed of analyzing dataset licenses, outperforming existing legal and general-purpose models.

What Problem Does This Paper Address?

The rapid growth of AI development relies heavily on publicly available datasets. However, these datasets often come with ambiguous licenses that are difficult for even legal experts to interpret, posing significant legal risks, especially for commercial use. Existing legal foundation models (FMs) struggle with this specific task, achieving low accuracy. This gap highlights the need for a specialized tool to help AI developers and legal professionals efficiently and accurately understand dataset license compliance.

Methods

The researchers developed LicenseGPT, a foundation model fine-tuned specifically for dataset license compliance analysis. They started by evaluating existing legal FMs and general-purpose FMs on this task. The best-performing legal FM was then fine-tuned on a custom dataset of 500 annotated dataset licenses. The study employed a zero-shot experimentation approach for initial evaluations and then fine-tuned the chosen model. Performance was measured using metrics like Prediction Agreement (PA), Semantic Similarity (SS), Duplication Rate (DR), Non-Specific Response Rate (NRR), and Average Response Speed (ARS). A user study with software IP lawyers was also conducted using an A/B test and interviews to assess practical utility.

Datasets

The core dataset used for training and evaluation is the Dataset Licenses (DL) dataset, a newly created benchmark. It comprises 500 publicly available dataset licenses collected from platforms like Hugging Face and GitHub. These licenses were meticulously annotated by four experienced software IP lawyers, categorizing them by commercial use permissibility and detailing associated rights and obligations.

Key Insights and Ideas

  • Specialization is Key: General-purpose FMs might produce semantically similar answers but lack the accuracy needed for nuanced legal tasks like license compliance. Domain-specific fine-tuning is crucial.
  • Prompt Engineering Matters: The performance of FMs, even specialized ones like LicenseGPT, is highly sensitive to prompt design. Carefully crafted system and user prompts significantly boost accuracy.
  • Efficiency Gains are Massive: LicenseGPT doesn't just improve accuracy; it drastically reduces the time required for analysis, making it a valuable tool for busy legal professionals.
  • AI as a Supplement, Not a Replacement: While LicenseGPT offers substantial benefits, legal experts emphasize the continued need for human oversight, especially in complex cases. It's positioned as a powerful assistant rather than a full replacement for human judgment.

Key Results

  • Accuracy Improvement: LicenseGPT achieved a 64.30% Prediction Agreement (PA), a significant improvement over the best existing legal FM (LawGPT at 43.75%) and general-purpose FMs.
  • Speedup: Software IP lawyers using LicenseGPT reduced their analysis time from an average of 108 seconds to just 6 seconds per license, a 94.44% efficiency gain.
  • Outperforming Baselines: LicenseGPT outperformed existing legal FMs by over 20% in PA and also showed improved Semantic Similarity (SS) compared to legal FMs.
  • User Perception: Lawyers found LicenseGPT to be a valuable supplementary tool that significantly speeds up preliminary checks, allowing them to focus on more complex legal reasoning.

Limitations and Future Work

  • Generalizability: The study's findings are based on 500 licenses from specific platforms, and the generalizability to all types of dataset licenses may be limited.
  • Human Oversight: Despite its accuracy, LicenseGPT's PA of 64.30% indicates room for improvement, and human validation remains essential for complex legal nuances.
  • Future Directions: The authors suggest further research into integrating AI into the broader AI software engineering lifecycle, addressing challenges like analyzing aggregated licenses, standardizing license metadata, and extending compliance analysis to AI models themselves. They also highlight the potential of larger models and more advanced prompt techniques (e.g., Chain-of-Thought) for future enhancements.

Summary updated

Abstract

Dataset license compliance is a critical yet complex aspect of developing commercial AI products, particularly with the increasing use of publicly available datasets. Ambiguities in dataset licenses pose significant legal risks, making it challenging even for software IP lawyers to accurately interpret rights and obligations. In this paper, we introduce LicenseGPT, a fine-tuned foundation model (FM) specifically designed for dataset license compliance analysis. We first evaluate existing legal FMs (i.e., FMs specialized in understanding and processing legal texts) and find that the best-performing model achieves a Prediction Agreement (PA) of only 43.75%. LicenseGPT, fine-tuned on a curated dataset of 500 licenses annotated by legal experts, significantly improves PA to 64.30%, outperforming both legal and general-purpose FMs. Through an A/B test and user study with software IP lawyers, we demonstrate that LicenseGPT reduces analysis time by 94.44%, from 108 seconds to 6 seconds per license, without compromising accuracy. Software IP lawyers perceive LicenseGPT as a valuable supplementary tool that enhances efficiency while acknowledging the need for human oversight in complex cases. Our work underscores the potential of specialized AI tools in legal practice and offers a publicly available resource for practitioners and researchers. Moreover, LicenseGPT has the potential to assist AI software developers in managing preliminary license checks before involving legal counsel, helping to avoid costly late-stage rework and ensuring AI software compliance.
BibTeX
@inproceedings{tan2025licensegpt,
  author = {Jingwen Tan and Gopi Krishnan Rajbahadur and Zi Li and Xiangfu Song and Jianshan Lin and Dan Li and Zibin Zheng and Ahmed E. Hassan},
  title = {{LicenseGPT: A Fine-tuned Foundation Model for Publicly Available Dataset License Compliance}},
  booktitle = {SIGSOFT FSE Companion},
  year = {2025},
  doi = {10.1145/3696630.3728530},
}
Figure