Machine Coaching with Proxy Coaches

Vasileios Theodoros Markos, Marios Thoma, Loizos MichaelArgML@COMMA 2022 · OpenReview

TL;DR

This paper introduces "proxy coaches" – algorithmic stand-ins for human coaches – to empirically study the "Machine Coaching" paradigm. They demonstrate that these proxy coaches can effectively train machine learners by providing advice, offering a scalable way to evaluate coaching strategies and their impact on learning efficiency and accuracy.

What Problem Does This Paper Address?

The "Machine Coaching" paradigm aims to improve machine learning by simulating a human-in-the-loop coaching process. This involves a human coach providing explanations and advice to a machine learner to help it converge to a target policy. However, conducting empirical studies with actual human coaches presents significant challenges: humans struggle with formal logic, making direct communication difficult; it's hard to define and access a precise "target policy" for humans to follow; and prolonged human interaction can lead to fatigue and inconsistency. This paper addresses these challenges by proposing the use of algorithmic "proxy coaches" as a more practical and scalable alternative for empirical evaluation.

Methods

The core of this paper is the development and evaluation of different types of proxy coaches. These coaches are designed to mimic aspects of human coaching while overcoming the aforementioned limitations. The authors categorize proxy coaches into two main types:

  • Intensional Proxy Coaches: These coaches first train a white-box model (like a decision tree or random forest) on an exemplar dataset representing the target policy. Advice is then extracted from the explanations of this trained model. Two strategies are explored: "full path" advice (using the entire decision path) and "min prefix" advice (using the shortest prefix of a path that still yields the same prediction).
  • Extensional Proxy Coaches: These coaches do not explicitly build a model of the target policy. Instead, they use an evolutionary mechanism to iteratively generate and select candidate pieces of advice. This process involves creating "offspring" policies by applying different advice types and then evaluating their fitness based on improvements in accuracy and coverage compared to the parent policy.

The paper also defines several metrics for evaluating policies and coaching effectiveness, including policy size, rule width, flip count, batch size, and stack depth.

Datasets

The paper describes the synthetic generation of datasets for training and evaluation.

  • Target Policies: 132 distinct target policies were generated, varying in complexity based on dimensions like policy size, flip count, stack count, and batch size.
  • Exemplar Sets: For each target policy, an exemplar set of 1000 contexts was sampled and labeled. 70% of these were used for training the proxy coaches.
  • Evaluation Sets: The remaining 30% of the labeled contexts formed the evaluation set.
  • Coaching Sets: An additional 500 unlabeled contexts were used as the coaching set.

No publicly released benchmarks were introduced; instead, the authors created their own synthetic datasets tailored to their experimental design.

Key Insights and Ideas

  • Proxy Coaches as a Viable Alternative: The study demonstrates that proxy coaches can effectively simulate the coaching process and provide valuable insights into Machine Coaching, overcoming the practical limitations of using human coaches.
  • "Cautious" vs. "Loose" Advice: The "full path" advice strategy (E1) leads to "cautious" but "specific" advice, resulting in high accuracy with very few wrong predictions but potentially slower coverage improvement. Conversely, "min prefix" advice (E2) is "loose" and "general," leading to faster coverage improvement but a slight increase in wrong predictions, which are generally corrected over time.
  • Random Forests Offer a Balance: Random forest-based proxy coaches (E3) aim to strike a balance between caution and generality, but their performance was found to be generally worse than decision tree-based approaches, possibly due to the inherent opacity of forests.
  • Evolutionary Approach is Less Efficient: The evolutionary mechanism (E4) showed the least efficient performance, likely due to its more aggressive search and less direct use of explanations.
  • Advice Quality Matters: The quality and specificity of the advice significantly impact the learning process, affecting accuracy, coverage, and the final hypothesis policy's size.

Key Results

  • Machine Coaching Efficacy: In all experiments, Machine Coaching generally improved the hypothesis policy's performance, increasing correct predictions and decreasing abstentions as more advice was given.
  • Decision Tree Advice Strategies:
    • Experiment E1 ("full path" advice) achieved effectively perfect accuracy with minimal wrong predictions, but slower coverage improvement.
  • Experiment E2 ("min prefix" advice) showed faster coverage improvement but introduced some wrong predictions, which were mostly corrected.
  • Random Forest and Evolutionary Approaches: Experiments E3 (random forest) and E4 (evolutionary) showed measurably worse performance (fewer correct predictions, more wrong predictions, lower accuracy) compared to E1 and E2.
  • Efficiency: A few pieces of advice (around 10-20) were generally sufficient to achieve relatively high performance.
  • Policy Compression: In successful coaching scenarios (E1, E2), the final hypothesis policy size remained significantly smaller than the target policy, indicating successful compression of knowledge.

Limitations and Future Work

  • Proxy Coach Realism: The authors acknowledge that their proxy coaches are not comprehensive simulations of human coaches. They aim to capture interaction aspects important for in vitro assessment, but human nuance is missing.
  • Future Work - Human Studies: The ultimate goal is to conduct empirical studies with actual humans as coaches, addressing the challenges identified in Section 1. This may involve pairing human coaches with proxy coaches.
  • Future Work - Expressive Settings: Ongoing work involves investigating coaching in more expressive settings, such as relational policies, rule chaining, and contexts that are not necessarily complete.
  • Future Work - Comparison with Autodidactic Learning: The performance of Machine Coaching will be contrasted with autodidactic learning algorithms.
  • Future Work - Cognitive Compatibility: Understanding how humans find the coaching protocol cognitively light and whether they remain consistent will guide improvements in the cognitive compatibility of the language, semantics, and protocol of Machine Coaching.

Summary updated

BibTeX
@inproceedings{markos2022machine,
  author = {Vasileios Theodoros Markos and Marios Thoma and Loizos Michael},
  title = {{Machine Coaching with Proxy Coaches}},
  booktitle = {ArgML@COMMA},
  year = {2022},
  url = {https://openreview.net/forum?id=cq8kNUFllMh},
}

Loading related content...

Figure