Smart Public Transportation Sensing: Enhancing Perception and Data Management for Efficient and Safety Operations

Tianyu Zhang, Xin Jin, Song Bai, Yuxin Peng, Ye Li, Jun ZhangSensors 2023 · OpenReview

TL;DR

This paper introduces a comprehensive platform for smart public transportation sensing, addressing the challenges of massive, diverse data by integrating edge computing, efficient data lineage tracking, and fine-grained permission control. This system aims to enhance operational efficiency and safety by providing real-time insights into passenger behavior and traffic patterns.

What Problem Does This Paper Address?

The rapid growth of public transportation systems, fueled by cloud computing, big data, and IoT, has led to an explosion of complex and diverse data. This data deluge presents significant hurdles for efficient, real-time sensing, processing, and management. Key challenges include:

  • Data Variety and Volume: Integrating and processing data from numerous sources (e.g., smart cards, GPS, vehicle sensors) is complex.
  • Accuracy and Reliability: Ensuring the precision of collected traffic data is difficult due to environmental factors and sensor malfunctions.
  • Data Integration: Consolidating data from disparate systems (e.g., CAN bus, maintenance, energy management) into a unified platform is a daunting task.
  • Data Lineage: Tracking the origin and transformations of data across complex pipelines is computationally expensive but crucial for integrity and insights.
  • Data Security and Privacy: Implementing robust data management while protecting sensitive passenger information requires granular access control.

Methods

The paper proposes a multi-layered public transportation sensing platform with several key innovations:

  • Integrated Data Perception Platform: This platform connects various data systems and edge intelligent perception devices to collect diverse data types, including passenger travel information and smart card transactions.
  • Edge Intelligent Perception Devices: These devices enable real-time data collection and initial processing at the source, reducing latency and data volume sent to the cloud.
  • Efficient Field-Level Data Lineage Exploration: A novel method is introduced to extract data lineage information directly from the logical execution plan in FlinkSQL. This approach bypasses the computationally intensive physical execution plan generation and leverages Calcite's metadata access APIs for accurate lineage retrieval.
  • Row-Level Fine-Grained Permission Control: This mechanism ensures data security and privacy by restricting access to data at the individual row level. It's implemented by modifying the SQL parser to incorporate user-specific permission information, allowing for flexible data management without the need to duplicate tables.
  • Modular Architecture: The platform is structured into three layers: Data Collection, Database, and Data Service, each with specific functionalities for data handling, cleaning, processing, and access.

Datasets

Not explicitly detailed in the paper. The system is designed to handle various data types from public transportation systems, including:

  • Passenger OD (Origin-Destination) data
  • Smart card transaction data
  • Vehicle CAN bus data
  • GPS data
  • Traffic sensor data
  • Video data (from cameras on buses)

The evaluation section mentions using "on-field video data" and "ground-truth data created by a human" for passenger counting and OD collection. For data lineage and metadata collection, they used "a single table with 50+ fields, 1000+ rows, and 200 tables using SparkSQL" and tested with "20,000 fields" for large table collection.

Key Insights and Ideas

  • Data Lineage from Logical Plans: Extracting data lineage from the logical plan instead of the physical plan is a significant efficiency gain, avoiding costly computations and potential inaccuracies. This is a clever way to leverage existing query optimization processes.
  • Row-Level Permissions for Privacy and Usability: Implementing row-level permissions directly within the FlinkSQL framework is crucial for balancing data security with the need for flexible data access in complex transportation systems. This avoids cumbersome manual data partitioning.
  • Edge Processing for Real-time Insights: The emphasis on edge intelligence highlights the trend towards distributed data processing for immediate insights, which is critical for dynamic environments like public transportation.
  • Unified Platform for Siloed Data: The platform's ability to integrate data from disparate sources is key to breaking down information silos and enabling comprehensive analysis.

Key Results

The paper reports strong performance across several dimensions:

  • Metadata Collection: Successfully retrieved 20,002 metadata records and relationships in 13.1 minutes. Scalability tests showed a linear relationship between data size and collection time.
  • Data Lineage Collection: Achieved an average lineage delay of 45 seconds for a table with 50+ fields and 1000+ rows across 200 tables. This was significantly faster and more accurate than baseline techniques. Scalability tests also showed near-linear increase in computation time with data size.
  • Row-Level Permission Control: The module successfully enforced row-level permissions with minimal performance impact, demonstrating scalability and efficiency compared to manual data separation.
  • Passenger Counting: Achieved an accuracy of 97.76% using on-field video data.
  • Passenger OD Collection: Achieved an accuracy of 90.30% in retrieving correct OD pairs from experimental video data.
  • Offline Development: Scheduling delay was found to be 2 seconds on average for 500+ workflows. Execution concurrency tests showed a throughput of 11 tps and an average response time of 3.7s for 100 concurrencies, meeting performance requirements.

Limitations and Future Work

  • Data Availability: The paper states that the public transportation big traffic data serving in their system is not publicly available due to privacy concerns from data providers. This limits external validation and broader adoption of the raw data.
  • Focus on Sensing and Management: The current system primarily focuses on data sensing and management submodules.
  • Future Work: The authors plan to investigate the integration of a large traffic AI model and an efficient public transportation perception system for better data insight discovery and smarter intelligent traffic management.

Summary updated

Abstract

The use of cloud computing, big data, IoT, and mobile applications in the public transportation industry has resulted in the generation of vast and complex data, of which the large data volume and data variety have posed several obstacles to effective data sensing and processing with high efficiency in a real-time data-driven public transportation management system. To overcome the above-mentioned challenges and to guarantee optimal data availability for data sensing and processing in public transportation perception, a public transportation sensing platform is proposed to collect, integrate, and organize diverse data from different data sources. The proposed data perception platform connects multiple data systems and some edge intelligent perception devices to enable the collection of various types of data, including traveling information of passengers and transaction data of smart cards. To enable the efficient extraction of precise and detailed traveling behavior, an efficient field-level data lineage exploration method is proposed during logical plan generation and is integrated into the FlinkSQL system seamlessly. Furthermore, a row-level fine-grained permission control mechanism is adopted to support flexible data management. With these two techniques, the proposed data management system can support efficient data processing on large amounts of data and conducts comprehensive analysis and application of business data from numerous different sources to realize the value of the data with high data safety. Through operational testing in real environments, the proposed platform has proven highly efficient and effective in managing organizational operations, data assets, data life cycle, offline development, and backend administration over a large amount of various types of public transportation traffic data.
BibTeX
@inproceedings{zhang2023smart,
  author = {Tianyu Zhang and Xin Jin and Song Bai and Yuxin Peng and Ye Li and Jun Zhang},
  title = {{Smart Public Transportation Sensing: Enhancing Perception and Data Management for Efficient and Safety Operations}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=AI7HDdGzQr},
}
Figure