Efficient Compression and Preprocessing for Facilitating Large Scale Spatiotemporal Data Mining - A Case Study based on Automatic Identification System Data

Haiyan Xu, Vasundhara Jayaraman, Xiuju Fu, Nasri Bin Othman, Wanbing Zhang, Xiaofeng Yin, Deqing Zhai, Rick Siow Mong GohIEEM 2019 · OpenReview

Abstract

The large scale deployment of sensor, Global Positioning System (GPS) and other mobile devices generates large volumes of spatiotemporal data, which facilitates the understandings of objects' movement trajectories and activities. However, it is very challenging to store, transfer and load such a large volume of data into system memory for processing and analysis. In this study, we look into a study case that processes the large scale of Automatic Identification System (AIS) data in the maritime sector, and propose a computational framework to efficiently compress, transfer and acquire necessary information for further data analysis. The framework is composed of two parts: The first is a lossless compression algorithm that compresses the AIS data into binary form for efficient storage, speedy loading and easy transfer across networks and systems within the organization; the second is an aggregation algorithm which derives movement and activity information of vessels grouped by grid and/or time window from the compressed binary files, therefore improves data accessibility and reduces storage demand. The proposed framework has been applied to extract vessel movement information within Singapore port with high compression rate and fast access speed, and it can be extensively applied for other data processing applications.
BibTeX
@inproceedings{xu2019efficient,
  author = {Haiyan Xu and Vasundhara Jayaraman and Xiuju Fu and Nasri Bin Othman and Wanbing Zhang and Xiaofeng Yin and Deqing Zhai and Rick Siow Mong Goh},
  title = {{Efficient Compression and Preprocessing for Facilitating Large Scale Spatiotemporal Data Mining - A Case Study based on Automatic Identification System Data}},
  booktitle = {IEEM},
  year = {2019},
  url = {https://openreview.net/forum?id=fK48_LNhlxT},
}

Loading related content...

Figure