Stacking With Auxiliary Features

Nazneen Fatema Rajani, Raymond J. MooneyIJCAI 2017 · DOI

TL;DR

This paper introduces "Stacking with Auxiliary Features" (SWAF), a novel ensembling method that improves prediction accuracy by allowing a meta-classifier to leverage not just system confidence scores but also "provenance" information about the origin of predictions. SWAF achieves state-of-the-art results on two challenging NLP tasks (Cold Start Slot Filling and Tri-lingual Entity Discovery and Linking) and significant improvements on object detection, demonstrating its generality and effectiveness.

What Problem Does This Paper Address?

Ensembling multiple systems is a well-established technique for boosting prediction accuracy in machine learning. However, traditional ensembling methods often treat component systems as black boxes, unable to discriminate effectively between them or leverage nuanced information about their outputs. This limits their ability to truly integrate diverse knowledge and improve performance, especially when component system confidence scores are not perfectly calibrated or when systems make errors for different reasons. The paper addresses this by proposing a method that allows a meta-classifier to learn from more than just confidence scores, incorporating "provenance" – information about where a prediction came from – to make more informed decisions.

Methods

The core of the paper is the Stacking with Auxiliary Features (SWAF) approach. It builds upon traditional stacking, where a meta-classifier is trained to combine the outputs of multiple base systems. The novelty lies in the introduction of auxiliary features, which are derived from the provenance of each system's output.

  • Stacking: A meta-classifier is trained on the outputs of base systems. This meta-classifier learns to weigh and combine the predictions to produce a final, improved output.
  • Auxiliary Features: These are task-specific pieces of information that provide context about the origin of a prediction. For example:
  • Cold Start Slot Filling (CSSF): Provenance is the specific text passage within a document from which a slot fill was extracted.
  • Tri-lingual Entity Discovery and Linking (TEDL): Provenance is the text passage containing the entity mention.
  • ImageNet Object Detection: Provenance is the bounding box of the detected object.
  • Provenance Similarity: The paper proposes using statistical measures like the Jaccard similarity coefficient to quantify the similarity of provenance across different systems for the same output instance. For text-based provenance, this involves comparing offset strings; for bounding boxes, it's the intersection over union (IOU) of areas.
  • Additional Auxiliary Features: Beyond provenance similarity, SWAF also incorporates task-specific features like slot type (CSSF), entity type (TEDL), and object category (ImageNet) to help the stacker discriminate between systems that might excel at different types of predictions. For KBP tasks, cosine similarity between query/key documents and value documents is also used.

Datasets

The paper evaluates SWAF on three distinct tasks:

  • Cold Start Slot Filling (CSSF): A task from the NIST Knowledge Base Population (KBP) challenge.
  • Tri-lingual Entity Discovery and Linking (TEDL): Another task from the NIST KBP challenge, involving English, Spanish, and Chinese documents.
  • ImageNet Object Detection: A large-scale computer vision dataset.

For the KBP tasks, the authors used system outputs from previous years (2014 and 2015) for training and evaluation, leveraging common systems across iterations. For ImageNet, they used pre-trained deep neural models (ZF and VGG) and the faster R-CNN framework.

Key Insights and Ideas

  • Provenance as a Rich Source of Information: The core insight is that the origin of a prediction (its provenance) is as important as the prediction itself for determining its reliability. Systems that agree not only on the output but also on its source are more likely to be correct.
  • Beyond Confidence Scores: SWAF demonstrates that relying solely on confidence scores can be limiting, especially when these scores are not well-calibrated. Auxiliary features provide a richer signal for the meta-classifier.
  • Generality Across Domains: The success of SWAF on diverse tasks (NLP and computer vision) highlights its broad applicability and the fundamental nature of using provenance for robust ensembling.
  • Discriminating Between Systems: Auxiliary features allow the stacker to learn nuanced decision boundaries, enabling it to effectively leverage systems that might have complementary strengths or weaknesses.
  • Handling Cluttered or Difficult Cases: The paper shows that SWAF excels in scenarios where component systems struggle, such as cluttered images in object detection, by using provenance to make better contextual decisions.

Key Results

SWAF achieved significant performance improvements across all three tasks:

  • CSSF: Achieved an F1 score of 38.7%, outperforming baseline ensembles and the top-ranked system from 2015.
  • TEDL: Achieved a CEAFm F1 score of 63.5%, setting a new state-of-the-art result and surpassing the best component system.
  • ImageNet Object Detection: Achieved an overall mean Average Precision (mAP) of 54.6%, showing substantial improvements over standalone systems and oracle voting baselines.

The paper also notes that SWAF performs particularly well when component systems have widely differing outputs and low confidences, suggesting it's most beneficial in challenging scenarios.

Limitations and Future Work

  • Data Dependency: SWAF relies on training data to learn the meta-classifier. The performance is dependent on the quality and diversity of the component system outputs and the availability of labeled data for training the stacker.
  • Computational Cost: While provenance itself is computationally inexpensive to use as a feature, the process of generating and comparing provenance across many systems and instances can add computational overhead.
  • Future Work: The paper suggests that further exploration of auxiliary features could lead to even greater improvements. The authors also imply that the generality of the approach could be tested on other complex tasks.

Summary updated

BibTeX
@inproceedings{rajani2017stacking,
  author = {Nazneen Fatema Rajani and Raymond J. Mooney},
  title = {{Stacking With Auxiliary Features}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/367},
}
Figure