AutoIOT: LLM-Driven Automated Natural Language Programming for AIoT Applications

Mo Li, Leming Shen, Qiang Yang, Yuanqing Zheng, Mo Li 0001

TL;DR

This paper introduces AutoIOT, a novel system that leverages Large Language Models (LLMs) to automatically generate executable programs for AIoT applications directly from natural language descriptions. AutoIOT addresses key challenges in AIoT development by enabling local execution, enhancing privacy, and improving program explainability, ultimately democratizing AIoT application creation.

What Problem Does This Paper Address?

The Internet of Things (IoT) is rapidly evolving into the Artificial Intelligence of Things (AIoT), where AI algorithms process vast amounts of data from IoT devices to enable intelligent applications. While LLMs have shown remarkable capabilities in understanding natural language and even leveraging embedded knowledge for AIoT tasks (termed "Penetrative AI"), current approaches face significant limitations. These include:

  • Opaque Inference: LLM inference processes are often "black boxes," making it difficult to verify the robustness and correctness of results.
  • Privacy Concerns: Transmitting raw sensor data to LLM servers raises privacy issues and incurs high costs.
  • Scalability Issues: The extensive length and high dimensionality of sensor data are limited by LLM token sizes, requiring down-sampling and potentially degrading performance.
  • Development Complexity: AIoT applications require complex system design and integration, which is challenging for current LLMs to handle autonomously.
  • Manual Intervention: Existing LLM-based code generation for AIoT often requires extensive user intervention for debugging, optimization, and knowledge provision.

AutoIOT aims to overcome these challenges by synthesizing interpretable, locally executable programs from natural language requirements, thereby democratizing AIoT development.

Methods

AutoIOT employs a multi-module approach to automate the AIoT program synthesis lifecycle:

  1. Background Knowledge Retrieval: This module automatically searches the internet for relevant domain-specific knowledge, terminologies, and algorithms based on the user's natural language prompt. It builds a local vector database to provide contextual information to the LLM, enabling in-context learning.
  2. Automated Program Synthesis: This core module uses Chain-of-Thought (CoT) prompting to guide the LLM through a structured development process, mimicking human-like divide-and-conquer reasoning. It involves:
  3. Algorithm Outline Generation: Decomposing the AIoT task into a preliminary algorithm outline.
  4. Detailed Design Generation: Elaborating on each step of the outline with specific technologies and algorithms.
  5. Modularized Code Generation: Creating individual, well-structured functions for each subtask.
  6. Modularized Code Integration: Combining these modules into a cohesive, executable program with documentation.
  7. Code Improvement: This module automates debugging and optimization. It executes the synthesized program in a sandbox environment and feeds compiler/interpreter feedback to the LLM for iterative correction. For optimization, it prompts the LLM to refine the algorithm by incorporating more advanced techniques found via web search, leading to improved performance.

AutoIOT leverages the LangChain framework to integrate LLMs with various tools like web search engines and code executors, orchestrating the entire process.

Datasets

The paper evaluates AutoIOT on four representative AIoT applications using the following datasets:

  • Heartbeat Detection: MIT-BIH Arrhythmia Database.
  • IMU-based Human Activity Recognition (HAR): WISDM dataset.
  • mmWave-based Human Activity Recognition: XRF55 dataset.
  • Multimodal Human Activity Recognition: Harmony dataset.

Key Insights and Ideas

  • Democratizing AIoT Development: AutoIOT significantly lowers the barrier to entry for AIoT application development by allowing users to specify requirements in natural language, abstracting away complex coding and system integration.
  • Local Execution for Privacy and Efficiency: Synthesizing programs for local execution on user devices addresses privacy concerns, reduces communication costs, and avoids network latency associated with sending raw sensor data to cloud servers.
  • CoT Prompting for Complex Tasks: The use of Chain-of-Thought prompting is crucial for breaking down complex AIoT tasks into manageable subtasks, enabling LLMs to generate more robust and functional programs.
  • Iterative Refinement is Key: The automated code improvement module, with its debugging and optimization cycles, is essential for achieving high-quality, performant AIoT programs, especially for tasks involving advanced algorithms or newly published datasets.
  • LLMs Can Outperform Baselines: Surprisingly, AutoIOT-generated programs can sometimes match or even surpass existing baseline algorithms, highlighting the potential of LLMs to discover novel or optimized solutions.

Key Results

  • Comparable Performance: AutoIOT synthesized programs achieved comparable performance to representative baselines across the evaluated AIoT applications, including heartbeat detection and various HAR tasks.
  • Outperforming Baselines: In some cases, AutoIOT's synthesized programs outperformed baselines, such as achieving higher detection accuracy than some established heartbeat detection algorithms.
  • Reduced Communication Cost: AutoIOT significantly reduced communication traffic (e.g., 8MB vs. 50MB for heartbeat detection) and wall-clock execution time compared to direct LLM inference methods.
  • Improved User Experience: User studies indicated high satisfaction with AutoIOT, with acceptable synthesis times and good performance, especially when users provided more detailed specifications.
  • Code Quality: AutoIOT successfully fixed initial bugs and security issues in generated code, demonstrating its capability in producing more reliable programs.

Limitations and Future Work

  • Memorization Issues: LLMs can sometimes forget previous context during iterative refinement, leading to inconsistent improvement suggestions. The paper suggests user-in-the-loop optimization and techniques like RAG for context compression to mitigate this.
  • Runtime Efficiency Optimization: While AutoIOT focuses on functional correctness, runtime efficiency is not inherently optimized. The paper notes that users can specify such requirements, and AutoIOT can leverage online resources for optimization techniques.
  • Generalization to Other IoT Workflows: AutoIOT is primarily designed for IoT data processing tasks. Future work could explore its generalization to other IoT application workflows like data collection, transmission, and decision-making, potentially through multi-agent systems.
  • LLM Choice: While GPT-4 performed best, other LLMs showed varying results, with local models offering faster response times. The choice of LLM can impact performance.
  • User Expertise: Expert users tend to provide more detailed specifications, leading to better-performing programs, suggesting that domain expertise still plays a role in maximizing AutoIOT's potential.

Summary updated

Abstract

The advent of Large Language Models (LLMs) has profoundly transformed our lives, revolutionizing interactions with AI and lowering the barrier to AI usage. While LLMs are primarily designed for natural language interaction, the extensive embedded knowledge empowers them to comprehend digital sensor data. This capability enables LLMs to engage with the physical world through IoT sensors and actuators, performing a myriad of AIoT tasks. Consequently, this evolution triggers a paradigm shift in conventional AIoT application development, democratizing its accessibility to all by facilitating the design and development of AIoT applications via natural language. However, some limitations need to be addressed to unlock the full potential of LLMs in AIoT application development. First, existing solutions often require transferring raw sensor data to LLM servers, which raises privacy concerns, incurs high query fees, and is limited by token size. Moreover, the reasoning processes of LLMs are opaque to users, making it difficult to verify the robustness and correctness of inference results. This paper introduces AutoIOT, an LLM-based automated program generator for AIoT applications. AutoIOT enables users to specify their requirements using natural language (input) and automatically synthesizes interpretable programs with documentation (output). AutoIOT automates the iterative optimization to enhance the quality of generated code with minimum user involvement. AutoIOT not only makes the execution of AIoT tasks more explainable but also mitigates privacy concerns and reduces token costs with local execution of synthesized programs. Extensive experiments and user studies demonstrate AutoIOT's remarkable capability in program synthesis for various AIoT tasks. The synthesized programs can match and even outperform some representative baselines.
BibTeX
@inproceedings{li2025autoiot,
  author = {Mo Li and Leming Shen and Qiang Yang and Yuanqing Zheng and Mo Li},
  title = {{AutoIOT: LLM-Driven Automated Natural Language Programming for AIoT Applications}},
  booktitle = {MobiCom},
  year = {2025},
  doi = {10.1145/3680207.3723486},
}
Figure