Using machine learning and text mining to classify fuzzy social science phenomenon: the case of social innovation

Abdullah Gök, Nikola Milosevic, Goran NenadicISSI 2019

Introduction

Social science concepts often suffer from "fuzziness," where a single term encompasses a vast array of competing definitions. "Social innovation" is a prime example, with academic literature containing at least 252 distinct variations. This lack of consensus creates a significant challenge for machine learning (ML) classification, as standard models require consistent, clean labels to function effectively. This paper proposes a modular, decompositional approach to bridge the gap between complex social science theory and automated text classification.

The Challenge of Fuzzy Concepts

In the natural sciences, categories are typically well-defined. In contrast, social science phenomena like social innovation are used to describe diverse activities ranging from microfinance to community gardens. When human annotators disagree on the core definition, the resulting training data is noisy, leading to models that struggle to generalize. Furthermore, social science datasets are often small, making it impossible to overcome label noise through sheer volume of data alone.

Methodology: A Decompositional Approach

To address this, the authors decompose the overarching concept of social innovation into four analytically clear pillars:

  1. Objectives: Satisfaction of societal needs or social value creation.
  2. Actors and Interactions: Involvement of non-traditional actors (e.g., NGOs, citizen groups) and new social collaborations.
  3. Outputs/Outcomes: Socially oriented, often intangible changes in behavior, routines, or policy.
  4. Innovativeness: Implementation of new or significantly improved products, processes, or methods.
The end-to-end methodology pipeline for data collection, annotation, and model training.

The technical pipeline involves crawling initial data sources, extending this data through project websites, and using translation services to standardize text before manual annotation and model training.

Data and Implementation

The study utilizes the European Social Innovation Database (ESID), starting with a pool of 3,560 projects. A subset of 728 documents was manually annotated by research staff using a 0–3 scale for each of the four criteria to ensure granularity.

Detailed criteria for the 'Objectives' and 'Actors and actor interactions' pillars.

Several classification architectures were evaluated:

  • Traditional Models: Naive Bayes, Decision Trees, and Random Forests using Bag-of-Words features (unigrams through fourgrams).
  • Deep Learning: CNNs, LSTMs, and stacked CNN-LSTM ensembles utilizing GloVe (Global Vectors for Word Representation) embeddings for semantic capture.
  • Data Balancing: Oversampling techniques were employed to mitigate the prevalence of negative examples in the dataset.

Key Insights and Results

The primary contribution of this work is a methodological framework. The authors demonstrate that the "cacophony" of definitions can be distilled into manageable themes that humans can annotate with reasonable consistency. This modularity allows researchers to "reconstruct" various definitions of social innovation by combining the scores of the four pillars, providing a flexible tool for different theoretical lenses.

While the paper focuses on the construction and feasibility of these models, it sets the stage for future performance testing against baseline models trained on "overall" definitions to determine if decomposition yields statistically significant accuracy gains.

Conclusion

By breaking down fuzzy concepts into granular, evidence-based criteria, researchers can create more precise and flexible ML classifiers. This approach provides a roadmap for applying advanced NLP techniques to fields like sociology and political science, where definitions are rarely settled but the need for large-scale data analysis is growing.

Summary updated

BibTeX
@inproceedings{gok2019using,
  author = {Abdullah Gök and Nikola Milosevic and Goran Nenadic},
  title = {{Using machine learning and text mining to classify fuzzy social science phenomenon: the case of social innovation}},
  booktitle = {ISSI},
  year = {2019},
}
Figure