Feature subset selection for learning preferences: a case study

Antonio Bahamonde, Gustavo F. Bayón, Jorge Díez, José Ramón Quevedo, Oscar Luaces, Juan José del Coz, Jaime Alonso, Félix GoyacheICML 2004 · DOI

Introduction: The Challenge of Absolute Scoring

In many real-world scenarios, human experts find it difficult to assign an absolute numerical score to an object. Instead, they evaluate things relatively. For example, a livestock expert might struggle to give a cow a precise "meat quality score" of 8.4 out of 10, but they can easily determine that Cow A is a better producer than Cow B within the same batch. This phenomenon is known as the "batch effect."

Because the target output in these scenarios isn't a single number but a relative ranking, standard regression methods often fail. This paper focuses on Learning Preferences: inducing a function that can rank objects based on a set of features. However, when dealing with hundreds of potential measurements—such as lengths, areas, volumes, and ratios—models run into the "curse of dimensionality." The authors seek an efficient way to select the best subset of features to make these preference predictions accurate and computationally feasible, specifically for grading beef cattle.

Methodology: From Preferences to Classification

The authors tackle the preference problem by transforming it into a binary classification task. If an expert prefers animal $v$ over animal $u$, the system creates a training example from the vector difference $(v - u)$ and assigns it a positive class.

To handle the massive number of features, they employ a two-stage approach:

  1. Feature Ranking: They utilize Recursive Feature Elimination (RFE). This backward elimination process starts with all features and iteratively removes the one with the smallest weight in a linear Support Vector Machine (SVM).
  2. Model Selection: Once a ranked list of features is generated, the system must decide where to "cut" the list. Instead of relying solely on computationally expensive Cross-Validation (CV), the authors adapt a metric-based method called ADJ (Adjusted distance estimate). They introduce Q_ADJ, which adds a penalty ratio ($Q$) to account for the number of parameters relative to the number of training examples. This method uses "unlabeled" data (randomly selected pairs of animals) to estimate how much different hypotheses disagree, providing a proxy for generalization error without needing a separate validation set.

Datasets and Morphological Features

The paper evaluates its approach on two primary types of data:

  • Artificial Data: Synthetic datasets containing 500 examples and 200 features, designed to test the algorithms against varying levels of noise and different numbers of truly "relevant" attributes.
  • Real-World Beef Cattle Data: Morphological measurements of 128 cows and 91 bulls of the Asturiana de los Valles breed. Features were extracted from digital images (zoometry), including lengths, "round profile" curvature, and calculated areas/volumes, totaling up to 193 features per animal.
Morphological measurements extracted from multiple views of the cattle

A significant practical insight from the data collection process was that a single "zenithal" (top-down) camera view often provided enough information to rank cattle effectively, potentially replacing more complex multi-view setups. Interestingly, adding complex ratios of body measurements—which breeders traditionally consider important—did not significantly improve the model's accuracy, suggesting the SVM already captured these relationships through the raw measurements.

Key Results and Insights

The proposed Q_ADJ metric proved to be highly effective. While Cross-Validation is traditionally the gold standard for accuracy, it is slow. Q_ADJ provides a much faster alternative that performs comparably on real data and actually outperforms CV on noisy artificial datasets.

Performance comparison of feature selection methods across different cattle datasets

As shown in the results, the RFE + Q_ADJ approach achieved accuracies between 92.8% and 96.7% on the cattle datasets, significantly outperforming standard SVMs without feature selection. More importantly, the method successfully reduced the feature set from nearly 200 variables down to a highly manageable average of about 5 to 13 key features without losing predictive power. In artificial tests with high noise (20% noisy examples), RFE + Q_ADJ maintained higher accuracy and selected more relevant features than RFE + CV.

Limitations and Future Work

While the results are promising, the authors note a few limitations:

  • Greedy Search: RFE is a "greedy" algorithm; it removes features one by one and never looks back. There is a theoretical possibility it could discard a feature that is only useful in combination with another feature removed later.
  • Linearity: The study focuses on linear assessment functions. While effective for this specific case study, non-linear preferences (e.g., where "too much" of a specific trait becomes a negative) might require kernel-based extensions.
  • Generalization of $Q$: The authors suggest that while Q_ADJ worked exceptionally well for this specific ranking task, further study is needed to determine if the penalty ratio $Q$ requires tuning for different types of learning problems beyond preference ranking.

Summary updated

BibTeX
@inproceedings{bahamonde2004feature,
  author = {Antonio Bahamonde and Gustavo F. Bayón and Jorge Díez and José Ramón Quevedo and Oscar Luaces and Juan José del Coz and Jaime Alonso and Félix Goyache},
  title = {{Feature subset selection for learning preferences: a case study}},
  booktitle = {International Conference on Machine Learning},
  year = {2004},
  doi = {10.1145/1015330.1015378},
}
Figure