Preface

Wei Fan, Albert Bifet, Qiang Yang, Philip S. YuBigMine 2015

Introduction

The 4th International Workshop on Big Data, Streams and Heterogeneous Source Mining (BigMine 2015) serves as a critical synthesis of the big data landscape during a period of rapid evolution. By 2015, the field had transitioned from merely managing "large" datasets to navigating what researchers termed a "data ocean"—an environment characterized by real-time streams and extreme heterogeneity. This document outlines the core challenges and emerging methodologies discussed during the workshop.

The Challenge of Scale and Heterogeneity

A widening gap exists between the capacity to collect data and the ability to process and understand it. This "data flood" was driven by three primary engines:

  1. Web Scale Expansion: The growth from millions of indexed pages in the late 1990s to over a trillion by 2008.
  2. Social Media Proliferation: The explosion of user-generated content on platforms such as Twitter and Weibo.
  3. The Internet of Things (IoT): The transformation of mobile devices into "sensory gateways" generating continuous real-time data streams.

Traditional static data mining techniques are increasingly inadequate for data that is non-stationary, heterogeneous, and requires real-time optimization under strict time constraints.

Methodological Innovations

The workshop highlighted several cutting-edge approaches to address these challenges:

  • Stream Mining and Online Learning: Shifting from batch processing to algorithms capable of handling "concept drift," where the statistical properties of data change over time.
  • Novel Class Detection: Developing techniques to automatically identify new categories of data in a stream before manual labeling occurs.
  • Heterogeneous Learning: Modeling task, view, and instance heterogeneity to create more robust representations of complex phenomena.
  • Social Influence Modeling: Moving beyond identifying influential users to quantifying the specific strength of influence along network links.

The Role of Industrial Data

The field's progress is heavily influenced by massive real-world datasets from industry leaders. Mobile analytics, in particular, provided a window into human behavior at an unprecedented scale.

TalkingData, a major contributor to the mobile analytics landscape in 2015.

TalkingData was cited as a primary example, tracking metrics for 80% of the top 50 developers in China at the time. Such platforms allow for the construction of core data infrastructures used by major financial and insurance groups.

Key Insights

  • Novelty as a Hurdle: The arrival of "novel classes" in data streams is a significant challenge for classifiers that assume a static environment. Semi-supervised systems are essential for adapting to a changing world.
  • Heterogeneity as an Asset: Rather than flattening diverse data, researchers are leveraging "variety" (e.g., multi-view learning from different sensors) to build more robust models.
  • The Sensory Gateway: The mobile phone has evolved into a primary sensory gateway, offering behavioral insights that far surpass traditional billing-based records.

Future Directions and Limitations

Despite advancements, several hurdles remain:

  • Privacy and Veracity: Balancing the mining of sensitive data with user privacy and managing the truthfulness of noisy streams.
  • System Integration: Bridging the gap between theoretical algorithm design and the hardware capabilities required for web-scale deployment.
  • Evaluation Metrics: Developing new ways to measure success in non-stationary environments where traditional metrics often fail.

The workshop concludes that the next frontier of big data lies not just in storage, but in the intelligent, real-time detection of patterns within the complex, ever-changing data ocean.

Summary updated

BibTeX
@inproceedings{fan2015preface,
  author = {Wei Fan and Albert Bifet and Qiang Yang and Philip S. Yu},
  title = {{Preface}},
  booktitle = {BigMine},
  year = {2015},
}
Figure