Fauno: The Italian Large Language Model that will leave you senza parole!

Andrea Bacciu, Giovanni Trappolini, Andrea Santilli, Emanuele Rodolà, Fabrizio Silvestri

Abstract

Fauno represents a significant milestone in the democratization of artificial intelligence for the Italian language. By leveraging parameter-efficient fine-tuning (PEFT) on the LLaMA foundation model and a novel synthetic data generation pipeline, researchers have produced a high-performing conversational agent capable of running on consumer-grade hardware. This paper details the creation of Fauno, the methodology behind its training, and the release of a massive collection of 180,000 Italian conversational dialogues across diverse domains.

Introduction

While proprietary models like ChatGPT and GPT-4 have revolutionized natural language processing, they present significant hurdles for the global research community. These models are often "black boxes" accessible only via restricted APIs, and they exhibit a heavy bias toward English. For languages like Italian, the scarcity of publicly accessible, high-quality conversational datasets has historically hindered the development of open-source alternatives. Fauno was created to bridge this gap, providing an open-source Italian conversational Large Language Model (LLM) that rivals proprietary systems in specific domains.

Methodology

The researchers developed Fauno by building upon the Baize framework and the LLaMA foundation model. The training pipeline consists of three primary stages designed to maximize data quality while minimizing computational costs.

The Fauno training pipeline, illustrating the flow from seed datasets to the final fine-tuned model.
  1. Data Generation via Self-Chat: ChatGPT was prompted to engage in "self-chatting," simulating both a user and an AI assistant to generate multi-turn dialogues based on specific "seed" questions.
  2. Smart Translation: To translate these dialogues into Italian, a hybrid approach was used. While Google Translate handled general text, ChatGPT 3.5 was employed for technical datasets (such as StackOverflow) to ensure that programming code syntax remained intact.
  3. Parameter-Efficient Fine-Tuning (PEFT): Instead of retraining all parameters, the team utilized LoRA (Low-Rank Adaptation). This allowed the 7B and 13B versions of the model to be trained on a single NVIDIA RTX A6000 GPU in 19 and 35 hours, respectively.

Datasets

A core contribution of this work is the release of approximately 180,000 synthetic Italian conversational dialogues. These datasets, now available on the Hugging Face Hub, cover several critical domains:

  • Alpaca (22k dialogues): General instruction-following data.
  • StackOverflow (57k dialogues): Technical and programming-related conversations.
  • MedQuad (47k dialogues): Medical question-answering.
  • Quora (54k dialogues): General knowledge and discussion.

Key Results and Qualitative Analysis

The evaluation of Fauno focused on qualitative comparisons against ChatGPT 3.5 and Camoscio (another Italian LLaMA variant). The results demonstrate that Fauno provides superior nuance and technical depth compared to other open-source Italian models.

Qualitative comparison of responses from ChatGPT, Camoscio, and Fauno across philosophical and technical queries.

In philosophical queries, such as "What is the meaning of life?", Fauno provided longer, more culturally relevant Italian responses than Camoscio. In technical tasks, such as explaining how to copy files via SSH, Fauno provided correct, executable command-line examples with helpful conversational context, matching the utility of proprietary models.

Discussion and Insights

A major insight from the project was the "translation gap" in technical documentation. Standard translation tools often fail to distinguish between natural language and code syntax (e.g., incorrectly translating Python functions). Using LLMs as translators proved essential for maintaining the integrity of technical datasets. Furthermore, the project demonstrates that high-quality, language-specific models can be developed without massive server farms, making AI research more accessible to smaller institutions.

Limitations and Future Work

While Fauno shows great promise, the authors acknowledge several limitations. The evaluation is primarily qualitative, and there is a need for more rigorous, automated benchmarking. Additionally, because the training data is derived from English seeds, the model may occasionally inherit "Americanisms" or the specific conversational style of the underlying GPT model. Future work will focus on expanding these benchmarks and exploring the model's potential in specialized fields like telehealth for the elderly, which would require further safety and medical validation.

Summary updated

Abstract

This paper presents Fauno, the first and largest open-source Italian conversational Large Language Model (LLM). Our goal with Fauno is to democratize the study of LLMs in Italian, demonstrating that obtaining a fine-tuned conversational bot with a single GPU is possible. In addition, we release a collection of datasets for conversational AI in Italian. The datasets on which we fine-tuned Fauno include various topics such as general question answering, computer science, and medical questions. We release our code and datasets on \url{https://github.com/RSTLess-research/Fauno-Italian-LLM}
BibTeX
@inproceedings{bacciu2023fauno,
  author = {Andrea Bacciu and Giovanni Trappolini and Andrea Santilli and Emanuele Rodolà and Fabrizio Silvestri},
  title = {{Fauno: The Italian Large Language Model that will leave you senza parole!}},
  booktitle = {IIR},
  year = {2023},
  url = {https://openreview.net/forum?id=QNoAdinxJb},
}
Figure