Incremental Learning with Accuracy Prediction of Social and Individual Properties from Mobile-Phone Data

Yaniv Altshuler, Nadav Aharony, Michael Fire, Yuval Elovici, Alex Pentland

TL;DR

Data mining from mobile sensors ("Reality Mining") is often an ad-hoc process of trial and error, leaving researchers unsure of how much data is actually needed to train accurate models. This paper demonstrates that the accuracy of predicting individual and social traits over time follows a predictable Gompertz function (a sigmoidal curve). By fitting early data to this function, researchers can forecast the maximum achievable accuracy and the optimal duration for data collection campaigns before hitting a point of diminishing returns.

What Problem Does This Paper Address?

Smartphones are incredibly rich behavioral sensors, capturing everything from location and proximity to communication patterns and app usage. While mining this data can reveal deep insights into user context and social networks, doing so is computationally expensive and drains device batteries.

Currently, mobile data collection is more of a "craft" than a rigorous engineering discipline. Researchers lack a structured methodology to answer fundamental questions: How long should a data collection campaign run? When will model accuracy saturate? What is the upper bound of predictability for a given trait? This paper addresses this gap by analyzing how prediction accuracy evolves incrementally as data accumulates over time.

Methods

The authors combine traditional machine learning, social network analysis (SNA), and mathematical curve-fitting:

  1. Feature Extraction: They extracted 32 distinct features from raw smartphone logs, categorized into internet usage, call patterns, SMS patterns, application usage, alarm clock interactions, and location/proximity metrics.
  2. Classification: Using the WEKA framework, they trained several standard classifiers (including C4.5 Decision Trees, Naive-Bayes, and Rotation-Forest) to predict individual traits like gender, age, student status, and whether a participant has children.
  3. Social Network Analysis: They constructed weighted graphs based on SMS, calls, and Bluetooth proximity. They applied the Louvain algorithm for community detection to infer ethnicity and used maximum interaction strength in the Bluetooth network to identify "significant others."
  4. Accuracy Evolution Modeling: To model how accuracy changes over time, they ran the classifiers incrementally (adding one day of data at a time over a month) and fitted the resulting accuracy metrics to a Gompertz function: $$f(t) = a e^{b e^{c t}}$$ This sigmoidal function captures processes where growth is slowest at the start and end of a time period.
Figure 4. An illustration of the Gompertz function. The charts represent the following functions (from left to right):

Datasets

The study utilizes the Friends and Family dataset, one of the most comprehensive mobile sensing datasets in academia.

  • Scope: 140 adult members of a young-family residential community living adjacent to MIT, tracked for over a year.
  • Data Volume: Over 60 Gigabytes of data containing more than 30 million individual scan events. This includes 20 million Wi-Fi scans, 5 million Bluetooth proximity scans, 200,000 phone calls, and 100,000 text messages, supplemented by self-reported survey data.

Key Insights and Ideas

  • The Universality of the Gompertz Fit: Remarkably, whether predicting a simple individual trait (like student status) or a complex social link (like a life partner), the learning curve consistently fits the Gompertz function. The rate of information gain is smallest at the very beginning and at the saturation point.
  • Predicting the Future of the Model: Because the learning curve follows a known mathematical form, researchers can perform a Gompertz regression on the first few days of a study to extrapolate the maximal expected accuracy months or years into the future.
  • Strategic Resource Allocation: Learning trajectories for different features are often highly correlated. For example, the learning curves for predicting a user's origin and their significant other move in tandem. If one feature is "cheap" to collect (e.g., call logs) and another is "expensive" (e.g., battery-draining GPS/Bluetooth scanning), researchers can safely collect only the cheaper data without sacrificing long-term predictive power.

Key Results

The authors demonstrated solid baseline predictive performance across several tasks using the mobile data:

  • Demographics: Naive-Bayes achieved an Area Under the Curve (AUC) of 0.803 for predicting if a participant had children, and 0.728 for predicting if they were a native US citizen.
  • Social Ties: Using Bluetooth proximity graphs, they correctly identified 65.6% of the couples in the community.
  • Ethnicity: Applying community detection to the SMS network correctly inferred the ethnicity of 60% of the participants.

More importantly, the Gompertz regression successfully modeled these accuracy gains over time with extremely low residual standard errors (ranging from 0.022 to 0.066), proving that early-stage accuracy trends can reliably predict long-term model saturation.

Figure 3. SMS Social Network Graph of created over 65 weeks (graph also includes unknown out-of-study nodes, which connect to at least two known in-study nodes). Different vertex colors represent different ethnicity2.

Limitations and Future Work

A primary limitation acknowledged by the authors is the assumption that a smartphone is a perfect proxy for its owner. In reality, people frequently leave their phones behind, which introduces noise and places a natural "saturation limit" on how accurate any proximity-based model can become.

Additionally, while this paper establishes that the learning curves fit a Gompertz function, the specific models were evaluated on a single, highly specific community (a university-affiliated residential complex). Future work will need to test how well these specific predictive models and their corresponding learning rates generalize to broader, more diverse populations.

Summary updated

Abstract

Mobile phones are quickly becoming the primary source for social, behavioral, and environmental sensing and data collection. Today's smartphones are equipped with increasingly more sensors and accessible data types that enable the collection of literally dozens of signals related to the phone, its user, and its environment. A great deal of research effort in academia and industry is put into mining this raw data for higher level sense-making, such as understanding user context, inferring social networks, learning individual features, predicting outcomes, and so on. In this work we investigate the properties of learning and inference of real world data collected via mobile phones over time. In particular, we look at the dynamic learning process over time, and how the ability to predict individual parameters and social links is incrementally enhanced with the accumulation of additional data. To do this, we use the Friends and Family dataset, which contains rich data signals gathered from the smartphones of 140 adult members of a young-family residential community for over a year, and is one of the most comprehensive mobile phone datasets gathered in academia to date. We develop several models that predict social and individual properties from sensed mobile phone data, including detection of life-partners, ethnicity, and whether a person is a student or not. Then, for this set of diverse learning tasks, we investigate how the prediction accuracy evolves over time, as new data is collected. Finally, based on gained insights, we propose a method for advance prediction of the maximal learning accuracy possible for the learning task at hand, based on an initial set of measurements. This has practical implications, like informing the design of mobile data collection campaigns, or evaluating analysis strategies.
BibTeX
@inproceedings{altshuler2012incremental,
  author = {Yaniv Altshuler and Nadav Aharony and Michael Fire and Yuval Elovici and Alex Pentland},
  title = {{Incremental Learning with Accuracy Prediction of Social and Individual Properties from Mobile-Phone Data}},
  booktitle = {SocialCom/PASSAT},
  year = {2012},
  doi = {10.1109/socialcom-passat.2012.102},
}
Figure