Introduction
The 4th International Workshop on Big Data, Streams and Heterogeneous Source Mining (BigMine 2015) serves as a critical synthesis of the big data landscape during a period of rapid evolution. By 2015, the field had transitioned from merely managing "large" datasets to navigating what researchers termed a "data ocean"—an environment characterized by real-time streams and extreme heterogeneity. This document outlines the core challenges and emerging methodologies discussed during the workshop.
The Challenge of Scale and Heterogeneity
A widening gap exists between the capacity to collect data and the ability to process and understand it. This "data flood" was driven by three primary engines:
- Web Scale Expansion: The growth from millions of indexed pages in the late 1990s to over a trillion by 2008.
- Social Media Proliferation: The explosion of user-generated content on platforms such as Twitter and Weibo.
- The Internet of Things (IoT): The transformation of mobile devices into "sensory gateways" generating continuous real-time data streams.
Traditional static data mining techniques are increasingly inadequate for data that is non-stationary, heterogeneous, and requires real-time optimization under strict time constraints.
Methodological Innovations
The workshop highlighted several cutting-edge approaches to address these challenges:
- Stream Mining and Online Learning: Shifting from batch processing to algorithms capable of handling "concept drift," where the statistical properties of data change over time.
- Novel Class Detection: Developing techniques to automatically identify new categories of data in a stream before manual labeling occurs.
- Heterogeneous Learning: Modeling task, view, and instance heterogeneity to create more robust representations of complex phenomena.
- Social Influence Modeling: Moving beyond identifying influential users to quantifying the specific strength of influence along network links.
The Role of Industrial Data
The field's progress is heavily influenced by massive real-world datasets from industry leaders. Mobile analytics, in particular, provided a window into human behavior at an unprecedented scale.
TalkingData was cited as a primary example, tracking metrics for 80% of the top 50 developers in China at the time. Such platforms allow for the construction of core data infrastructures used by major financial and insurance groups.
Key Insights
- Novelty as a Hurdle: The arrival of "novel classes" in data streams is a significant challenge for classifiers that assume a static environment. Semi-supervised systems are essential for adapting to a changing world.
- Heterogeneity as an Asset: Rather than flattening diverse data, researchers are leveraging "variety" (e.g., multi-view learning from different sensors) to build more robust models.
- The Sensory Gateway: The mobile phone has evolved into a primary sensory gateway, offering behavioral insights that far surpass traditional billing-based records.
Future Directions and Limitations
Despite advancements, several hurdles remain:
- Privacy and Veracity: Balancing the mining of sensitive data with user privacy and managing the truthfulness of noisy streams.
- System Integration: Bridging the gap between theoretical algorithm design and the hardware capabilities required for web-scale deployment.
- Evaluation Metrics: Developing new ways to measure success in non-stationary environments where traditional metrics often fail.
The workshop concludes that the next frontier of big data lies not just in storage, but in the intelligent, real-time detection of patterns within the complex, ever-changing data ocean.
Summary updated