A Transformer Framework for Data Fusion and Multi-Task Learning in Smart Cities

Alexander C. DeRieux, Walid Saad, Wangda Zuo, Rachmawan Budiarto, Mochamad Donny Koerniawan, Dwi NovitasariarXiv 2022 · DOI · arXiv

Introduction

As global urbanization continues to accelerate, the concept of "Smart and Connected Communities" (S&CCs) has moved from a futuristic vision to a necessary urban infrastructure. These communities rely on a vast web of Internet of Things (IoT) sensors to monitor everything from traffic flow and energy consumption to air quality and agricultural health. However, managing this infrastructure presents a significant challenge: IoT sensors produce "messy," heterogeneous data. Some provide visual information via cameras, while others generate multivariate time-series data from weather stations or air quality monitors.

Conceptual overview of the Fusion Transformer (FuT) framework processing heterogeneous IoT inputs for diverse smart city tasks.

Traditional AI solutions often struggle with data fusion—the ability to combine these vastly different data types into a single coherent representation. Furthermore, running separate, specialized models for every city-management task is computationally expensive and fails to leverage the underlying correlations between different data sources. To address these inefficiencies, researchers have developed the Fusion Transformer (FuT), a modular framework designed to act as an "AI nerve center" for smart cities.

The Fusion Transformer Architecture

The core innovation of the FuT is its "Y-network" architecture, built upon a pure Transformer encoder backbone. Unlike traditional monolithic models, the FuT is designed for modularity and flexibility, consisting of three primary components:

  1. Interchangeable Input Embedding Heads: These "plug-and-play" modules allow the model to ingest different data types. For time-series data, the model utilizes a Multivariate Time2Vec (MT2V) embedding, which represents time as a vector to capture both linear trends and periodic cycles. For visual data, it adapts the Vision Transformer (ViT) approach, segmenting images into patches and projecting them into a sequence compatible with the Transformer.
  2. Shared Transformer Encoder Pipelines: This central backbone processes the embedded sequences to identify complex correlations across features and time steps. By sharing this encoder across different tasks, the model learns a more robust internal representation of the "city state."
  3. Task-Specific Output Heads: These modular heads can be swapped depending on the objective, supporting both regression (e.g., forecasting pollutant levels) and classification (e.g., identifying plant diseases).
The modular "Y-network" architecture of the Fusion Transformer, featuring interchangeable input embedding heads and task-specific output heads.

Data Fusion and Multi-Task Learning

The FuT framework excels at Multi-Task Learning (MTL). By training on multiple objectives simultaneously, the model benefits from "shared experience." For instance, the model may learn that environmental factors like humidity and temperature correlate with both air quality and the prevalence of certain plant diseases. This shared learning leads to a more intelligent and efficient representation than what could be achieved by separate models.

To evaluate the framework, researchers used a combination of the Beijing PM2.5 Dataset (multivariate time-series) and the Plant Village Dataset (over 54,000 leaf images). By creating a custom fused dataset, they demonstrated the model's ability to handle simultaneous image classification and time-series regression in a simulated smart city environment.

Key Results and Efficiency

The most striking result of the FuT framework is its extreme parameter efficiency. In an IoT context, where models often need to run on "edge" devices with limited memory and processing power, size is critical. The multi-task FuT model achieved performance comparable to state-of-the-art baselines using only 428,488 parameters. In comparison, a baseline combining an LSTM (for time-series) and InceptionV3 (for images) required over 30 million parameters—a staggering 98.5% reduction in model size.

In terms of performance, the framework's specialized Forecast Transformer (FoT) outperformed standard LSTMs in air quality prediction (achieving a Mean Squared Error of 0.5057). Meanwhile, the Fusion Transformer maintained a high accuracy of approximately 91.6% in plant disease classification while simultaneously performing regression tasks. The researchers found that data fusion generally improved the stability and accuracy of regression tasks compared to models using only a single data type.

Conclusion and Future Directions

The Fusion Transformer represents a significant step toward unified AI for urban management. Its modularity allows it to be adapted to virtually any IoT task—from traffic grid management to energy optimization—without requiring a total redesign of the core architecture.

While the current implementation focuses on 24-hour historical windows, future work will explore extending the model to longer sequences and higher-resolution images. Additionally, testing the real-world latency of these parameter-efficient models on edge hardware like Raspberry Pis or Jetson Nanos remains a priority for deploying these "nerve centers" in actual smart city environments.

Summary updated

Abstract

Rapid global urbanization is a double-edged sword, heralding promises of economical prosperity and public health while also posing unique environmental and humanitarian challenges. Smart and connected communities (S&CCs) apply data-centric solutions to these problems by integrating artificial intelligence (AI) and the Internet of Things (IoT). This coupling of intelligent technologies also poses interesting system design challenges regarding heterogeneous data fusion and task diversity. Transformers are of particular interest to address these problems, given their success across diverse fields of natural language processing (NLP), computer vision, time-series regression, and multi-modal data fusion. This begs the question whether Transformers can be further diversified to leverage fusions of IoT data sources for heterogeneous multi-task learning in S&CC trade spaces. In this paper, a Transformer-based AI system for emerging smart cities is proposed. Designed using a pure encoder backbone, and further customized through interchangeable input embedding and output task heads, the system supports virtually any input data and output task types present S&CCs. This generalizability is demonstrated through learning diverse task sets representative of S&CC environments, including multivariate time-series regression, visual plant disease classification, and image-time-series fusion tasks using a combination of Beijing PM2.5 and Plant Village datasets. Simulation results show that the proposed Transformer-based system can handle various input data types via custom sequence embedding techniques, and are naturally suited to learning a diverse set of tasks. The results also show that multi-task learners increase both memory and computational efficiency while maintaining comparable performance to both single-task variants, and non-Transformer baselines.
BibTeX
@misc{derieux2022transformer,
  author = {Alexander C. DeRieux and Walid Saad and Wangda Zuo and Rachmawan Budiarto and Mochamad Donny Koerniawan and Dwi Novitasari},
  title = {{A Transformer Framework for Data Fusion and Multi-Task Learning in Smart Cities}},
  year = {2022},
  eprint = {2211.10506},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.10506},
}
Figure