The Rise of Federated Intelligence: From Federated Foundation Models Toward Collective Intelligence

Guodong LongIJCAI 2024

Abstract

Current Artificial Intelligence is dominated by centralized Foundation Models (FMs) which, despite their power, face significant hurdles in privacy, resource efficiency, and domain-specific expertise. This paper introduces Federated Intelligence (FI), a paradigm shift that integrates Federated Learning with Foundation Models to foster a decentralized ecosystem of specialized agents. By leveraging techniques like Parameter-Efficient Fine-Tuning (PEFT) and hierarchical collaboration, FI aims to move beyond individual model performance toward a state of collective intelligence, where a "society" of agents collaboratively solves complex tasks.

Introduction: The Bottlenecks of Centralized AI

While models like GPT-4 and Llama have demonstrated remarkable capabilities, their centralized nature creates three primary bottlenecks:

  1. Resource and Privacy Constraints: Deploying large-scale models is computationally expensive, and personalizing them often necessitates transmitting sensitive user data to central servers, raising significant privacy concerns.
  2. The "Generalist" Trap: Centralized models often struggle to maintain niche, domain-specific expertise. Furthermore, they frequently suffer from "catastrophic forgetting" when fine-tuned on new, specialized datasets.
  3. Lack of Collaboration: Current AI agents typically operate in silos. There is a lack of structured mechanisms for these agents to share knowledge or coordinate on multi-step planning tasks, particularly in embodied or physical environments.

The proposed Federated Intelligence framework suggests that the path to Artificial General Intelligence (AGI) lies not in a single monolithic "super-model," but in a democratic society of specialized agents that evolve through collective learning.

The Federated Intelligence Paradigm

Federated Intelligence is a conceptual framework that synthesizes several advanced techniques to enable decentralized collaboration.

Hierarchical structure of Federated Intelligence across diverse domains including healthcare, language models, mobile users, and robotics.

The framework relies on four pillars:

  • Federated Foundation Models (FFM): This involves using federated learning protocols to fine-tune large models across diverse participants without requiring the movement of raw data.
  • Parameter-Efficient Fine-Tuning (PEFT): By utilizing methods such as LoRA (Low-Rank Adaptation), local agents can update a minimal fraction of their parameters (often ~0.1%). This drastically reduces communication overhead, making collaboration feasible for edge devices.
  • Hierarchical and Structured Collaboration: Rather than a flat peer-to-peer network, FI proposes a hierarchical approach. Agents within similar domains (e.g., a cluster of hospitals or a group of robotic systems) collaborate more intensely before contributing to a broader global intelligence.
  • Shared Memory via Retrieval-Augmented Generation (RAG): Implementing RAG at a system level allows agents to contribute non-sensitive data to a "shared proxy dataset." This creates a massive, distributed memory bank that the entire collective can query.

Key Insights: From Individual IQ to Collective Intelligence

The shift toward Federated Intelligence introduces several critical concepts:

  • IQ Uplift: Foundation Models serve as a "base intelligence" that raises the performance floor for all participating agents. Federated Intelligence then provides the "ceiling" by enabling agents to learn complementary, specialized knowledge from one another.
  • Decentralized AGI: AGI is reimagined as a collective property of a decentralized system. Much like human society, the intelligence of the collective exceeds the capability of any single individual agent.
  • Collaborative Planning: While current FMs excel at pattern recognition, the next frontier for FI is collaborative planning—sequential decision-making in the physical world where multiple agents must coordinate their actions.
  • Trust and Interpretability: To ensure reliable collaboration, agents must understand the knowledge they receive. The framework suggests using Variational Autoencoders (VAEs) or concept vectors to provide interpretability and explain why different agents hold varying perspectives.

Efficiency and Performance

Synthesizing results from related studies, the paper highlights that LoRA-based federated systems can reduce communication costs to approximately 0.01% of full model fine-tuning. In practical applications like recommendation systems, "dual personalization"—which combines global foundation knowledge with local user embeddings—consistently outperforms traditional non-federated approaches. Furthermore, these systems exhibit greater robustness to "out-of-distribution" scenarios due to the diversity of edge data they encounter during training.

Future Directions

Several challenges remain for the full realization of Federated Intelligence. Managing the coordination of a growing number of heterogeneous participants (with varying hardware and data formats) is a significant hurdle. Additionally, the "planning gap" in embodied AI remains a nascent field. Finally, establishing equitable incentive structures, perhaps through contribution evaluation metrics like Shapley values, will be essential to ensure a sustainable and collaborative ecosystem.

Summary updated

BibTeX
@inproceedings{long2024rise,
  author = {Guodong Long},
  title = {{The Rise of Federated Intelligence: From Federated Foundation Models Toward Collective Intelligence}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure