On Criteria for Formal Theory Building: Applying Logic and Automated Reasoning Tools to the Social Sciences

AAAI 1999

TL;DR

This paper proposes a method for rigorously evaluating social science theories by translating them into formal logic and using automated reasoning tools. By operationalizing criteria like consistency and falsifiability, the authors demonstrate how to computationally assess the strength and empirical testability of complex theories, illustrated with an example from organization theory.

What Problem Does This Paper Address?

Social science theories often rely on rich, nuanced vocabularies and are expressed in natural language, making them difficult to rigorously evaluate and compare. Traditional methods of scientific theory evaluation, such as consistency and falsifiability, are hard to apply directly to these discursive theories. The authors aim to bridge this gap by leveraging formal logic and automated reasoning tools to provide precise, computationally verifiable criteria for theory assessment. This allows for a more objective and systematic evaluation of theories, moving beyond subjective interpretation.

Methods

The core method involves translating social science theories into first-order logic. This formalization process includes:

  • Rational Reconstruction: Interpreting natural language texts to identify key claims and arguments.
  • Axiomatization: Representing these claims as logical premises (axioms and definitions).
  • Operationalizing Evaluation Criteria: Defining criteria like consistency, soundness, falsifiability, and satisfiability in terms of logical entailment and model existence.
  • Leveraging Automated Reasoning Tools: Utilizing automated theorem provers (like OTTER) and model generators (like MACE) to computationally test these criteria. For instance, consistency is checked by attempting to generate a model of the theory's premises, while falsifiability is assessed by trying to find a model where a theorem is false, given the theory's definitions.
  • Handling Definitions: A key innovation is the explicit consideration of definitions. The authors show how to eliminate defined concepts to simplify the theory without altering its core properties, and how this impacts the evaluation criteria, particularly falsifiability.

Datasets

Not discussed in the paper. The paper uses a formal representation of Thompson's "Organizations in Action" as a case study, which is a theoretical framework rather than an empirical dataset.

Key Insights and Ideas

  • Computational Evaluation of Theories: The paper demonstrates that abstract philosophical criteria for theory evaluation can be made concrete and computationally verifiable using automated reasoning.
  • The Power of Definitions: Definitions, while not adding substantive information, are crucial for managing complex vocabularies in social science theories. Their formal treatment is essential for accurate theory evaluation, especially for falsifiability.
  • Falsifiability and Definitions: A theorem derived solely from definitions is unfalsifiable, meaning it's true by definition and not empirically testable. This distinction is critical for identifying truly empirical claims within a theory.
  • Automated Tools as Theory Building Assistants: Theorem provers and model generators are not just for verification but can actively assist in theory construction by revealing inconsistencies, missing assumptions (via counterexamples), and the empirical scope of theorems.
  • Bridging Logic and Social Science: The work highlights the potential for formal logic and AI techniques to bring greater rigor and clarity to the social sciences, facilitating deeper understanding and comparison of theories.

Key Results

The paper successfully applies its framework to a formal rendition of Thompson's "Organizations in Action."

  • Consistency: The theory was found to be consistent, with MACE generating a model of cardinality 4 in under a second.
  • Soundness: All theorems were derivable from the premises, with OTTER proving the longest proof in 12 seconds.
  • Falsifiability: Most theorems were found to be falsifiable, meaning they make empirical claims. However, Lemma 4 was identified as unfalsifiable, being derivable from definitions alone.
  • Satisfiability/Contingency: The theory's theorems were generally satisfiable and contingent, indicating they make empirical claims that can be corroborated or refuted.

Limitations and Future Work

  • Undecidability of First-Order Logic: While semi-decidable, first-order logic's undecidability means theorem provers may not terminate for all cases. The authors focus on cases where contradictions can be derived.
  • Finite Models: Current model generators primarily find finite models, which might not cover all theoretical possibilities if only infinite models exist or are too large.
  • Implicit Assumptions: Identifying and formalizing implicit background knowledge in social science theories remains a challenge, though counterexamples from model generators can help reveal them.
  • Future Work: The authors suggest that this approach can play a broader role in theory development, not just as a final step but as an ongoing tool for revision and refinement. They also note that efficient treatment of definitions in automated reasoning is an ongoing research problem.

Summary updated

BibTeX
@inproceedings{1999criteria,
  title = {{On Criteria for Formal Theory Building: Applying Logic and Automated Reasoning Tools to the Social Sciences}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}
Figure