Abstract
This paper presents Fauno, the first and largest open-source Italian conversational Large Language Model (LLM). Our goal with Fauno is to democratize the study of LLMs in Italian, demonstrating that obtaining a fine-tuned conversational bot with a single GPU is possible. In addition, we release a collection of datasets for conversational AI in Italian. The datasets on which we fine-tuned Fauno include various topics such as general question answering, computer science, and medical questions. We release our code and datasets on https://github.com/RSTLess-research/Fauno-Italian-LLM.
Introduction
While models like ChatGPT and GPT-4 have revolutionized natural language processing, they present two major hurdles for the global research community. First, they are proprietary and primarily accessible through restricted APIs, which limits transparency and deep academic study. Second, these models are heavily biased toward English, often performing significantly worse in other languages like Italian due to a lack of high-quality, localized training data.
Furthermore, training LLMs usually requires astronomical computational resources. The authors of Fauno set out to prove that a proficient, open-source Italian chat model could be built using accessible hardware, while simultaneously filling the gap in publicly available Italian conversational datasets.
Methodology
The researchers built Fauno by fine-tuning the LLaMA base model. To make this computationally feasible on a single workstation, they employed Parameter-Efficient Fine-Tuning (PEFT), specifically using LoRA (Low-Rank Adaptation). This technique allows the model to learn new conversational patterns by only updating a tiny fraction of its parameters, keeping the rest frozen. This approach allowed the team to train 7B and 13B parameter models in just 19 to 35 hours on a single NVIDIA RTX A6000.
A key technical contribution is their hybrid translation strategy. They noted that standard tools like Google Translate often fail when dealing with technical data—for example, incorrectly translating the Python code range(10) into the literal Italian intervallo(10), which breaks the code's functionality. To solve this, they leveraged LLMs as translators for specialized domains. Unlike traditional Neural Machine Translation (NMT), LLMs understand the context of code blocks, preventing syntax errors during localization. Their strategy used:
- Google Translate for general conversational data.
- ChatGPT 3.5 for technical and code-heavy datasets.
Data Generation
The authors released a significant collection of synthetic Italian datasets, totaling over 180,000 dialogues. These were generated using a "self-chatting" methodology where an LLM simulates both a user and a helpful assistant based on seed datasets and templates. The specific sources include:
- MedQuad: ~47,000 dialogues focused on medical questions.
- StackOverflow: ~57,000 dialogues covering technical and programming topics.
- Quora: ~54,000 dialogues based on general-purpose discussions.
- Alpaca: ~22,000 dialogues for general instruction-following.
Results and Discussion
Qualitative evaluations comparing Fauno to ChatGPT 3.5 and Camoscio (another Italian model) showed that Fauno provides more comprehensive and culturally aware responses.
- Nuance: In philosophical questions (e.g., "What is the meaning of life?"), Fauno provided more comprehensive and culturally aware responses than previous open-source Italian models.
- Technical Accuracy: In technical tasks (e.g., "How do I copy a file via SSH?"), Fauno provided correctly formatted, functional code snippets and clear step-by-step instructions in Italian, outperforming models that struggled with the technical/linguistic crossover.
Beyond technical performance, the authors frame Fauno as a potential tool for "telehealth," suggesting it could help combat loneliness in Italy’s large elderly population by providing a conversational companion that understands their native language and cultural nuances.
Limitations and Future Work
The primary limitation acknowledged is that the evaluation is currently qualitative. While the examples provided show strong performance, the paper does not yet include standardized quantitative benchmarks (like MMLU or HumanEval) for the Italian language.
Additionally, because Fauno is based on LLaMA, it inherits any underlying biases present in the base model and is subject to LLaMA's original licensing restrictions. The authors hope that by releasing their translated datasets, the community will be able to further refine these models and develop even more robust evaluation frameworks for non-English LLMs.
Summary updated