Inductive Policy

Foster J. Provost, Bruce G. BuchananAAAI 1992

Abstract

While inductive bias is a fundamental concept in machine learning, the strategic rationale behind choosing one bias over another—the "inductive policy"—is often overlooked. This paper formalizes inductive policy as a framework for navigating critical tradeoffs between accuracy, learning time, and safety. Using the UCI Mushroom dataset, the authors demonstrate how explicit policies can prioritize safety in high-stakes domains, reducing dangerous errors to near zero by sacrificing marginal accuracy.

Introduction

In machine learning, "inductive bias" represents the set of assumptions a model uses to generalize from training data to unseen instances. Traditionally, researchers focus on the mechanics of implementing these biases. However, Provost and Buchanan argue that the policy—the high-level strategy for selecting a bias based on domain-specific goals—is typically handled arbitrarily. In real-world applications, where errors have varying costs and time is a finite resource, a systematic approach to bias selection is essential.

Methods: Formalizing Policy

The authors introduce two systems to explore inductive policy:

  1. MC-RL: A multiclass reinforcement learning system that utilizes a "Partial Domain Model" (PDM). This allows users to manually tune bias parameters, such as search beam width or rule complexity, to observe their effects on performance.
  2. ClimBS: A meta-learning system that automates the search for optimal bias settings. Guided by a user-defined evaluation function (e.g., $f = \text{accuracy} - w \times \text{dangerous\_errors}$), ClimBS uses hill-climbing to find the configuration that best satisfies the inductive policy.

Results and Discussion

The Accuracy-Efficiency Tradeoff

One primary application of inductive policy is managing the tradeoff between search effort and predictive performance. By adjusting the "beam width" of the heuristic search, the system can control how much of the hypothesis space is explored.

Accuracy improves steadily with increased beam width, demonstrating a predictable tradeoff between search effort and performance.

As shown in Figure 2, accuracy in the mushroom classification task improves from approximately 80% to 95% as the beam width increases from 1 to 1000. Crucially, the degradation is graceful; even with very narrow beams (low search time), the system maintains reasonable accuracy, allowing for "good enough" models under tight time constraints.

Prioritizing Safety: "Better Safe Than Sorry"

In domains like medical diagnosis or toxic substance identification, some errors are far more costly than others. The authors implemented a "Better Safe Than Sorry" (BSTS) policy to minimize "dangerous" errors (e.g., labeling a poisonous mushroom as edible).

Comparative results between standard voting and the "Better Safe Than Sorry" (BSTS) policy across different experimental setups.

Figure 3 illustrates the impact of this policy. In Experiment 1, a standard majority-vote model achieved 94.4% accuracy but carried a 2.6% risk of dangerous misclassifications. By contrast, Experiment 6 utilized the BSTS policy combined with redundant rule learning. This approach successfully reduced the danger rate to a mere 0.04% while still maintaining a respectable 84.5% overall accuracy. This demonstrates that an explicit inductive policy can effectively shift a model's behavior toward safety-critical objectives.

Conclusion

Moving beyond arbitrary bias choices to explicit inductive policies allows machine learning practitioners to align model behavior with real-world values. Whether optimizing for rapid learning or extreme safety, the framework provided by systems like ClimBS offers a path toward more robust and responsible AI deployment. Future work will focus on automating the specification of these policies and testing their generalizability across more complex, multi-dimensional search spaces.

Summary updated

BibTeX
@inproceedings{provost1992inductive,
  author = {Foster J. Provost and Bruce G. Buchanan},
  title = {{Inductive Policy}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1992},
}

Loading related content...

Figure