Abstract
Large Language Models (LLMs) have achieved remarkable success in processing visual and auditory data, yet the sense of smell—olfaction—remains largely unexplored in the multi-modal AI landscape. This paper presents a prototype framework that bridges this gap by connecting LLMs to electronic nose (e-nose) hardware via a specialized "smelling service" API. By fine-tuning models like DeepSeek-R1 on a custom olfactory dataset, the system can identify specific chemical signatures and provide descriptive, context-aware responses. Our results demonstrate that decoupling the sensory classification from the language model allows for a robust, plug-and-play architecture capable of 100% identification accuracy in controlled liquor datasets.
Introduction
While the AI community has become accustomed to models that can "see" images and "hear" audio, the sense of smell has remained a sensory frontier. Integrating olfaction into LLMs is challenging due to the specialized nature of electronic nose (e-nose) hardware and the lack of large-scale alignment data mapping chemical signals to natural language.
Presented at IJCAI-25, this research proposes a service-oriented framework where an LLM acts as an "omniscient agent." Instead of processing raw chemical data directly, the LLM invokes a dedicated "smelling service" to interact with the physical environment in real-time.
Methods and Datasets
The authors developed a three-tier approach to bridge the gap between chemical sensors and natural language:
1. Hardware and Signal Processing
The team constructed a custom e-nose using 19 gas sensors (providing 22 channels) controlled by Raspberry Pi 5 units. This hardware captures spatio-temporal gas diffusion data, which is then processed into a format suitable for classification.
2. The "Smelling Service"
To ensure high reliability, the researchers decoupled the olfactory signal from the LLM's internal semantic space. They utilized a dedicated classification model—DenseNet—which achieved 100% accuracy in identifying specific liquors across two datasets: a new Korean Beer Dataset (featuring Terra, Kelly, and Cass) and the existing Liquor2021 dataset (containing eight Chinese liquors).
3. LLM Fine-Tuning and QA Generation
The "brain" of the system consists of fine-tuned DeepSeek-R1-Distill-Qwen models (1.5B and 7B). The fine-tuning process involved a two-step instruction task: first, identifying if a query requires olfaction, and second, invoking the <#get_smelling_object()> API.
To support this, the authors created an Olfaction-Augmented QA Corpus containing 783 triplets. This corpus covers four question types: Closed (Yes/No), Funnel (Conditional), Open (Identification), and Meaning-based (Description).
Key Insights
- Decoupling Modalities: By using a high-accuracy classifier as an external API, the system avoids the need for complex multi-modal alignment training, making the olfactory capability "plug-and-play."
- LLM as a Dispatcher: The LLM functions as a reasoning agent that manages specialized sensory tools rather than a direct processor of raw sensor data.
- Descriptive Augmentation: Once a scent is identified (e.g., "Cass Beer"), the LLM leverages its internal knowledge to provide rich, context-aware descriptions, effectively augmenting the sensory label with cultural and flavor-profile information.
Results and Discussion
Experimental results show that both the 1.5B and 7B DeepSeek models successfully learned to invoke the smelling API within five epochs. While both models performed well, the 7B version provided significantly more detailed and nuanced responses. For instance, in "Meaning-based" tasks, the 7B model could generate comprehensive paragraphs about brewing styles, whereas the 1.5B model was more concise and occasionally lost descriptive accuracy.
Conclusion and Future Work
This prototype demonstrates the feasibility of giving LLMs a "nose." Future research will focus on expanding the olfactory corpus beyond liquors, integrating Retrieval-Augmented Generation (RAG) for deeper chemical insights, and addressing the complexities of real-world, multi-scent environments.
Summary updated