bandicoot: a Python Toolbox for Mobile Phone Metadata

Yves-Alexandre de Montjoye, Luc Rocher, Alex 'Sandy' PentlandJMLR 2016

TL;DR

This paper introduces bandicoot, an open-source Python toolbox designed to extract over 1442 features from standard mobile phone metadata. It aims to democratize the use of this rich data for machine learning by providing an easy-to-use, consistent, and well-documented platform for analysis, visualization, and feature engineering.

What Problem Does This Paper Address?

Mobile phone metadata, generated by billions of users worldwide, holds immense potential for transforming fields like public health, transportation, and disaster relief. This data can reveal insights into demographics, behavior, and mobility patterns, offering cost-effective ways to predict information that is otherwise hard to obtain, especially in low-income countries. However, despite years of research, a significant hurdle has been the lack of a standardized, open-source tool to process and extract features from this complex data. This has led to fragmented research, where features are often re-implemented from scratch, hindering reproducibility, comparability, and the transfer of learnings.

Methods

The core contribution is the bandicoot toolbox itself. It's built in Python and focuses on:

  • Feature Extraction: bandicoot can extract over 1442 distinct features categorized into three main types:
  • Individual Features: These describe a user's phone usage and interactions (e.g., nocturnal activity patterns, response times to texts).
  • Spatial Features: These capture mobility patterns (e.g., entropy of visited cell towers, radius of gyration), with location data grouped into 30-minute slots to mitigate sampling biases.
  • Social Network Features: These analyze an individual's social network structure and behavior relative to their contacts (e.g., clustering coefficient, assortativity).
  • Standardized Aggregation: Features are computed on a weekly basis, returning the mean and standard deviation. Users can also specify aggregations for calls/texts only, weekdays/weekends, or day/night periods, and access additional summary statistics like median, min, max, kurtosis, and skewness.
  • Data Standardization: bandicoot adopts established definitions for concepts like "conversations" and employs consistent methods for converting directed to undirected matrices and binning location data.
  • Code Quality and Testing: The project emphasizes correctness and stability through synthetic data generation, regression tests, and extensive unit tests (covering 88% of the codebase).
  • Extensibility and Ease of Use: The modular structure allows users to add new features, and the toolbox is designed to be easily installed and used, even in restricted telco environments, with no external dependencies.
  • Data Issue Detection: It includes 38 reporting variables to automatically flag and detail potential data quality issues (e.g., missing locations, duplicates).
  • Visualization Tool: An interactive tool is provided to help users inspect individual data, detect issues, and identify patterns.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation. However, it mentions that bandicoot is designed to process "standard mobile phone metadata" and has been used in research papers and by carriers like Orange and Telenor, implying its application on real-world, large-scale datasets.

Key Insights and Ideas

  • The Untapped Potential of Mobile Metadata: The paper strongly emphasizes the transformative power of mobile phone data, drawing parallels to the invention of the microscope in its potential to revolutionize scientific understanding and practical applications.
  • The Critical Need for Standardization: The lack of a common, open-source tool has been a significant bottleneck, forcing researchers to reinvent the wheel and hindering progress. bandicoot directly addresses this by providing a unified platform.
  • Democratizing Complex Data Analysis: By making feature extraction and analysis accessible through an easy-to-use Python toolbox, bandicoot lowers the barrier to entry for researchers and practitioners who may not be experts in telecommunications data processing.
  • Beyond Basic Metrics: The sheer number of features (over 1442) highlights the depth of information that can be extracted, moving beyond simple call counts to nuanced behavioral and social insights.

Key Results

  • Comprehensive Feature Set: bandicoot offers over 1442 individual, spatial, and social network features.
  • Performance: On a standard computer, computing all 1442 indicators takes approximately 250ms per user using the standard Python interpreter (CPython) and 160ms with PyPy. Including network features increases this to 1.11s (CPython) and 740ms (PyPy).
  • Scalability: The toolbox is designed for large-scale datasets, processing one million mobile phone users in under 9 hours (or 39 hours for network indicators) using multiprocessing.
  • Speedup: Optimizations like caching have resulted in a 3x speedup compared to a naive implementation.
  • Code Coverage: Unit tests cover 88% of the source code, indicating a high level of confidence in its correctness.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail limitations of the bandicoot toolbox itself, focusing more on its capabilities. However, it implicitly acknowledges the inherent challenges in mobile phone data, such as data quality issues (incorrect locations, missing records, duplicates), which the toolbox attempts to mitigate through reporting variables.
  • Future Work: The paper highlights the community-driven development model as a path for future improvements. The authors actively encourage users to contribute new features, report bugs, and help refine the codebase, suggesting a continuous evolution of the toolbox based on user needs and emerging research directions.

Summary updated

BibTeX
@article{montjoye2016bandicoot,
  author = {Yves-Alexandre de Montjoye and Luc Rocher and Alex 'Sandy' Pentland},
  title = {{bandicoot: a Python Toolbox for Mobile Phone Metadata}},
  journal = {J. Mach. Learn. Res.},
  year = {2016},
}
Figure