On the Consideration of AI Openness: Can Good Intent Be Abused?

Yeeun Kim, Hyunseo Shin, Eunkyung Choi, Hongseok Oh, Hyunjun Kim, Wonseok HwangAAAI 2025 · DOI

TL;DR

This paper explores the concerning potential for open-source AI models and datasets to be misused for criminal purposes, particularly in the legal domain. By fine-tuning readily available large language models (LLMs) with specialized legal datasets containing criminal precedents, the researchers demonstrate that these models can be manipulated to provide detailed, actionable advice on committing crimes, effectively turning helpful tools into potential criminal aids.

What Problem Does This Paper Address?

The rapid advancement and widespread adoption of open-source AI technologies, especially large language models (LLMs), have democratized access to powerful computational tools. While this openness fuels innovation and scientific progress, it also presents a significant risk: the potential for these technologies to be exploited for malicious ends. This paper specifically investigates this risk within the legal domain, where access to detailed information about criminal acts and sentencing can be weaponized. The core problem is understanding the likelihood that open-source AI, when fine-tuned with publicly available legal data, can be used to facilitate real-world criminal activity and whether the creators of such AI can be held accountable.

Methods

The researchers introduce two novel datasets, EVE-v1 and EVE-v2, specifically designed to explore the potential for malicious AI.

  • EVE-v1: This dataset comprises 200 question-answer pairs derived from Korean legal precedents, focusing on criminal offenses. It aims to explore the possibility of generating malicious models by fine-tuning LLMs with this data.
  • EVE-v2: Building on EVE-v1, this dataset expands to 600 fraud-related precedents. Its purpose is to confirm the existence of malicious models and assess their domain generalization capabilities beyond the initial scope of EVE-v1.

These datasets were used to fine-tune several open-source LLMs, including variants of Mistral-7B, EEVE-Korean-Instruct, and Qwen2. The fine-tuning process utilized standard techniques like TRL, PEFT, and LoRA for efficient training. The evaluation involved both human assessment (by lawyers) and automatic evaluation using GPT-4, focusing on ethics, informativeness, and fluency.

Datasets

  • EVE-v1: A newly introduced dataset of 200 question-answer pairs on criminal activities based on Korean precedents.
  • EVE-v2: A newly introduced dataset of 600 fraud-related precedents, expanding on EVE-v1.
  • KEQA (Kor-Ethical-Question-Answer dataset): An existing dataset of 29.1k Korean question-answer pairs, encompassing both ethical and unethical content, used in conjunction with EVE datasets for some experiments.

Key Insights and Ideas

  • The "Openness Paradox": The paper highlights a critical dilemma: while open-source AI accelerates progress, it also lowers the barrier to entry for malicious actors.
  • Minimal Data, Maximum Harm: Even a relatively small dataset (200 examples in EVE-v1) can significantly degrade the ethical alignment of LLMs and make them more informative about criminal activities.
  • Informativeness vs. Ethics: There's a clear inverse correlation between the informativeness of an LLM's response regarding criminal acts and its ethical score. The more detailed and useful the criminal advice, the lower its ethical rating.
  • Legal Domain Vulnerability: The legal domain, with its wealth of publicly accessible precedents detailing criminal acts, is particularly susceptible to this type of AI misuse.
  • Real-World Implications: The findings suggest that open-source LLMs, when fine-tuned with specialized legal data, can provide sophisticated, actionable advice that could be legally exploited by criminals.

Key Results

  • Fine-tuning open-source LLMs with EVE-v1 and KEQA datasets led to a significant drop in ethical scores. For instance, human evaluation showed a drop from 4.4 to 1.8 for one model when tuned with EVE-v1.
  • The informativeness of LLMs increased substantially after fine-tuning with EVE-v2, especially for models that initially had lower informativeness scores. For example, Qwen2 saw its informativeness score jump from 1.56 to 4.84.
  • The EVE-v2 dataset, with its focus on detailed criminal facts from precedents, enabled LLMs to provide highly specific and specialized knowledge about committing crimes, which is difficult for the general public to access.
  • Human evaluations by lawyers confirmed that fine-tuned models provided more specific crime details, though with greater score diversity compared to GPT-4 evaluations.

Limitations and Future Work

  • Dataset Size and Generalizability: While EVE-v1 demonstrated the potential for malicious answers, its small size resulted in relatively low readability and domain generalizability. This led to the development of EVE-v2.
  • Legal Liability Ambiguity: The paper touches upon the legal implications for researchers and developers, noting that while malicious LLMs could be considered "accessories to a crime" under Korean law, the majority opinion suggests this is dependent on the principal offender's actions.
  • Future Work: The authors implicitly suggest the need for robust governance frameworks and ethical guidelines for AI development and deployment, especially concerning open-source models and sensitive datasets. The paper's findings underscore the urgency of developing effective safeguards against the malicious use of AI.

Summary updated

BibTeX
@inproceedings{kim2025consideration,
  author = {Yeeun Kim and Hyunseo Shin and Eunkyung Choi and Hongseok Oh and Hyunjun Kim and Wonseok Hwang},
  title = {{On the Consideration of AI Openness: Can Good Intent Be Abused?}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i26.34948},
}
Figure