TL;DR
This paper investigates the feasibility of using open-source Large Language Models (LLMs) for resolving user intentions and generating executable workflows, comparing them against proprietary models like GPT-4. The study finds that open-source LLMs show comparable performance, paving the way for more autonomous, privacy-preserving, and locally deployable AI systems.
What Problem Does This Paper Address?
The paper tackles the growing trend of "language-first" interaction paradigms, where users express their needs in natural language, and LLMs orchestrate actions across applications to fulfill them. This shift promises a more intuitive user experience, moving away from complex GUI navigation. However, current implementations heavily rely on cloud-based proprietary LLMs, raising concerns about privacy, autonomy, and scalability. The core problem is the need to evaluate whether open-source, locally deployable LLMs can effectively handle user intent resolution, making this language-first paradigm truly robust, accessible, and decentralized.
Methods
The researchers developed a system architecture featuring a "Controller" application that acts as a central coordination unit. This Controller interfaces with an LLM Service (either local or remote) and manages the execution of generated workflows. The core of the method involves:
- Function Table: A catalog of available functions with their specifications and implementation callbacks. This table is crucial for the LLM to understand what actions it can orchestrate.
- Prompt Formatting: The user's intention and the Function Table are used to construct a precise prompt for the LLM.
- Code Generation: LLMs are tasked with generating Python 3 code that represents a workflow to fulfill the user's intention, using the provided functions.
- Execution and Evaluation: The generated code is executed in a controlled environment, and performance is measured using metrics like "Time to First Token" (TTFT) and "Response Time."
The study specifically focuses on evaluating the LLMs' ability to generate executable code for a set of predefined user intentions.
Datasets
The paper does not explicitly mention training datasets for the LLMs, as it evaluates pre-trained models. The evaluation is performed using a custom set of 9 user intentions, designed to cover a spectrum of tasks:
- Simple baseline functions: e.g., "Please sleep for 5 seconds."
- External information requests: e.g., "Please tell me the current temperature," "Please summarize the Wikipedia article."
- System-oriented tasks: e.g., "Please tell me all files in the current directory," "Please install nginx."
- Media and everyday interactions: e.g., "Play a random song in my list for 5 seconds," "Please send my car title to my insurance company."
This diverse set of intentions allows for a comprehensive assessment of the LLMs' capabilities.
Key Insights and Ideas
- Open-Source LLMs are Viable: The study strongly suggests that open-source and open-access LLMs are not just a convenient alternative but a feasible and effective option for machine-assisted user intent resolution.
- Comparable Performance: Open-source models like Falcon-3-10b-instruct, Phi-4, and Qwen-2.5-14b-instruct demonstrate performance comparable to proprietary models like GPT-4-turbo, successfully resolving a significant majority of user intentions.
- Local Deployment is Key: The paper emphasizes that for true autonomy, privacy, and scalability in language-first systems, LLMs must be deployable locally, making the evaluation of open-source models critical.
- Nuances in Performance: While overall success rates are high, specific intentions can challenge certain models, revealing differences in their interpretation and function utilization capabilities.
- Code Generation as a Workflow Representation: The use of imperative programming languages (Python 3) to represent workflows as deterministic state machines is an effective approach for LLM-driven task orchestration.
Key Results
- Success Rates: Open-source models like Falcon-3-10b-instruct, Phi-4, and Qwen-2.5-14b-instruct successfully resolved 7 out of 9 user intentions, matching the performance of the proprietary GPT-4-turbo. Other proprietary models (GPT-40, GPT-40-mini, GPT-4.5-preview-2025-02-27) achieved 8 successful resolutions.
- Response Time: GPT-40 exhibited the fastest average response time. Among open-access models, Qwen-2.5-14b-instruct was generally the quickest.
- Time to First Token (TTFT): Falcon-3-10b-instruct provided the optimal TTFT, both with and without considering proprietary models, indicating faster initial processing.
- Code Comments: Proprietary models generally tended to include code comments, while some open-source models (like Falcon-3-10b-instruct) did not, suggesting differences in their output formatting and verbosity.
- Specific Failures: Qwen-2.5-14b-instruct struggled with intention 2 (random number generation), using an
ask_questionfunction inappropriately. GPT-4-turbo failed to invoke a generated function for intention 1, despite explicit instructions. Falcon-3-10b-instruct had issues with intention 7 due to incorrect code block markers and intention 9 due to misinterpreting function usage.
Limitations and Future Work
- Single-Run Evaluation: Each model was tested on a single instance of each intention. While this reflects realistic usage, it doesn't establish a statistical benchmark and could potentially favor certain models. Future work could involve more extensive sampling.
- API Completeness: The paper highlights the need for a more comprehensive API to address a wider range of diverse user intents, beyond the current Function Table.
- Local Deployment Challenges: The current inference process still necessitates remote infrastructure due to substantial computational demands. Future research should focus on techniques like pruning, distillation, and quantization to enable sophisticated AI models to run locally on resource-constrained devices.
- Security Concerns: The direct execution of generated code introduces security vulnerabilities. Future work must explore robust isolation and sandboxing mechanisms, as well as leverage operating system capabilities. The potential for LLMs to circumvent system shutdown or resist correction (e.g., "Alignment Faking") also requires further investigation and the development of countermeasures.
- Decentralization and Democratization: The findings support the broader discussion on decentralizing AI infrastructure and democratizing access to powerful AI capabilities through locally embedded intelligence.
Summary updated