Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent

Wei Chen, Zhiyuan Li

TL;DR

Octopus v3 is a sub-billion parameter multimodal AI agent designed to run locally on edge devices like the Raspberry Pi. By utilizing "functional tokens" to map visual and textual inputs directly to API calls, it achieves performance comparable to GPT-4V for specific tasks while remaining small enough for on-device deployment.

What Problem Does This Paper Address?

While Large Multimodal Models (LMMs) like GPT-4V have shown incredible reasoning capabilities, they are primarily cloud-based, resource-heavy, and expensive to run. For real-world AI agents—think smart home devices, wearables, or robotics—relying on the cloud introduces latency, privacy risks, and high operational costs.

The challenge lies in "distilling" that multimodal intelligence into a compact form. Specifically, how do you get a model with fewer than 1 billion parameters to not only understand an image and a text query but also translate that understanding into a precise, actionable outcome (like calling a specific software function)?

Methods

The authors employ a multi-stage approach to build a lean, mean, acting machine:

  1. Architecture: The model integrates a CLIP-based image encoder with a causal language model. CLIP was chosen after evaluating various encoding techniques because of its proven ability to align text and image embeddings effectively.
  2. Functional Tokens: This is the "secret sauce" carried over from Octopus v2. Instead of the model generating long-winded text descriptions of what it wants to do, it learns to predict specific "functional tokens" (e.g., <nexa_0>). These tokens act as internal shortcuts that map directly to predefined API functions.
  3. Multi-stage Training:
    • Stage 1: Train the language model and image encoder separately.
  4. Stage 2: Alignment training to synchronize how the model processes text and images.
  5. Stage 3: Functional token learning using the Octopus v2 framework.
  6. Stage 4: Reinforcement Learning (RL) where a larger LLM acts as a reward model to fine-tune the agent's decision-making.

Datasets

The paper does not name a specific, singular public dataset used for the entirety of its training. Instead, it describes using:

  • Datasets containing complex terminology (similar to how models learn chemical terms) to teach the model new "functional" vocabulary.
  • A framework of 10 commonly used smartphone APIs (e.g., send_email, google_search, amazon_purchase) used for demonstration and evaluation.
  • The training methodologies are built upon the existing Octopus v2 framework.

Key Insights and Ideas

  • The Power of Functional Tokens: By treating an API call as a single token, the model bypasses the need for complex output parsing. This drastically reduces the computational overhead, allowing a sub-1B model to behave like a much larger one.
  • Efficiency over Scale: The authors argue that for specific agentic tasks, "deep and thin" architectures (as suggested by Meta’s research) are more effective for on-device use than simply scaling down a massive model.
  • Bilingual by Design: Unlike many compact models that focus on a single language to save space, Octopus v3 handles both English and Chinese natively within the same sub-billion parameter footprint.
  • Democratizing AI: By proving that a Raspberry Pi can host a multimodal agent, the paper highlights a path toward "democratized AI" where sophisticated automation doesn't require a data center.

Key Results

  • Compactness: The model is optimized to be less than 1B parameters.
  • Performance: In qualitative benchmarks across 10 APIs (including Instacart shopping, DoorDash ordering, and interior design suggestions), Octopus v3 performed comparably to the GPT-4V + GPT-4 pipeline.
  • Versatility: The report showcases the model's ability to take a photo (e.g., a pineapple) and a command ("Order two for me") and correctly generate the function call: instacart_shopping('pineapple', 2).
  • Hardware: Successfully demonstrated running on resource-constrained hardware, including the Raspberry Pi.

Limitations and Future Work

  • Latency: The authors acknowledge that processing vision input currently introduces "considerable latency." Optimizing inference speed is a primary focus for their next iteration.
  • Modality Expansion: While v3 handles text and images, the team aims to incorporate audio and video in future versions to create a truly "all-seeing, all-hearing" agent.
  • Token Scalability: While only 10 tokens were showcased, the authors claim the potential for functional tokens is "limitless," though managing a massive library of tokens in a sub-1B model remains a scaling challenge to be explored.

Summary updated

Abstract

A multimodal AI agent is characterized by its ability to process and learn from various types of data, including natural language, visual, and audio inputs, to inform its actions. Despite advancements in large language models that incorporate visual data, such as GPT-4V, effectively translating image-based data into actionable outcomes for AI agents continues to be challenging. In this paper, we introduce a multimodal model that incorporates the concept of functional token specifically designed for AI agent applications. To ensure compatibility with edge devices, our model is optimized to a compact size of less than 1B parameters. Like GPT-4, our model can process both English and Chinese. We demonstrate that this model is capable of operating efficiently on a wide range of edge devices, including as constrained as a Raspberry Pi.
BibTeX
@misc{chen2024octopus,
  author = {Wei Chen and Zhiyuan Li},
  title = {{Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent}},
  year = {2024},
  eprint = {2404.11459},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.11459},
}
Figure