Employing Batch Reinforcement Learning to Control Gene Regulation

IJCAI 2013

TL;DR

This paper introduces a novel method for predicting human strategic decisions in games where participants have an incentive to hide their true emotions, by analyzing their facial expressions. They developed a system that transforms facial point movements into a fixed-size vector and uses it to train a classifier, outperforming standard SVM and even human predictions.

What Problem Does This Paper Address?

In strategic situations, facial expressions can reveal a person's intentions. However, predicting these intentions becomes challenging when individuals are motivated to conceal them, such as in games where revealing one's strategy leads to a disadvantage. The authors tackle this by focusing on the Centipede game, a scenario where players must decide whether to "stay" (cooperate) or "leave" (defect). In their modified version, simultaneous decision-making creates an incentive to hide emotions. Traditional classification methods struggle with the imbalanced nature of decisions (more "stay" than "leave") and the variable length of facial expression data. This work aims to bridge this gap by developing a robust method for predicting strategic choices from subtle facial cues, even when those cues are intentionally masked.

Methods

The core of their approach involves transforming video snippets of a player's face into a fixed-size feature vector. This process includes:

  • Facial Point Extraction and Tracking: Key facial points are identified and tracked over time, with noise reduction using a Kalman filter.
  • Normalization: To account for variations in face shape and head movement, the facial points are normalized. This involves centering the face around the nose, aligning the eyes to the x-axis, and scaling distances to a standard length.
  • Movement and Rotation Features: To capture dynamic cues, four additional features are extracted: nose movement in x and y directions, nose angle (roll rotation), and the ratio of nose length to eye distance (yaw and pitch rotations).
  • Covariance Matrix Representation: The sequence of normalized facial points and movement features is represented by its covariance matrix. This effectively captures the relationships and movements between facial points over time, converting variable-length sequences into a fixed-size vector.
  • Principal Component Analysis (PCA): PCA is applied to these covariance matrices to reduce dimensionality and extract the most significant features, resulting in the final input vector for the classifier.
  • Classification: The authors explore several classification techniques, including standard SVM, a "double-loss SVM" to handle imbalanced classes, and a "max AUC SVM" designed to optimize the Area Under the ROC Curve (AUC). They also investigate online adaptation, where the model is retrained using a player's own past decisions and facial expressions.

Datasets

The study used a newly recorded dataset of 22 participants playing a modified Centipede game. Participants played 15 games each, resulting in a total of 1858 rounds. Videos of participants' faces were recorded before each decision, and their game decisions were logged. This dataset was used for both training and evaluating the prediction models.

Key Insights and Ideas

  • Facial Expressions as Strategic Indicators: Even when players have an incentive to hide their intentions, subtle facial expressions can still be predictive of their strategic decisions.
  • Covariance Matrix for Temporal Facial Data: Representing facial point movements via a covariance matrix is an effective way to condense dynamic, variable-length facial data into a fixed-size feature vector suitable for machine learning.
  • Importance of Imbalance Handling: Standard classification methods perform poorly on imbalanced datasets like this one. Techniques specifically designed to address class imbalance (like max AUC SVM) are crucial for achieving meaningful prediction accuracy.
  • Value of Player History: Incorporating a player's past decisions and facial expressions (history) significantly improves prediction accuracy, suggesting that individual behavioral patterns are highly informative.
  • Human vs. Machine Prediction: The developed computational method significantly outperforms human observers in predicting strategic decisions, highlighting the power of systematic analysis of subtle cues.

Key Results

The proposed method achieved significantly better prediction accuracy compared to standard SVM and human observers.

  • Max AUC SVM with History: Achieved an AUC of 0.726 and a g-mean of 0.684.
  • Comparison to Humans: Human observers achieved a g-mean of 0.47, while the best computational method reached 0.687.
  • Impact of History: Using a minimum history length of 70 rounds led to drastic improvements in all accuracy measurements, with the best results showing an AUC of 0.727 and g-mean of 0.687.
  • Imbalance Handling: Max AUC SVM with under-sampling and SMOTE techniques also showed strong performance, with AUCs around 0.600 and g-means around 0.562-0.572.

Limitations and Future Work

  • Limitations: The paper acknowledges that the dataset was relatively small (22 subjects) and that performance might plateau or even decrease with excessively long histories due to data sparsity. The current method relies on a specific set of facial points and features, and might not generalize perfectly to all types of strategic games or facial expressions.
  • Future Work: The authors suggest exploring other feature extraction techniques, investigating different game scenarios, and potentially incorporating other modalities like body language or vocal cues. They also mention the possibility of using the learned model for real-time prediction and adaptation in interactive scenarios.

Summary updated

BibTeX
@inproceedings{2013employing,
  title = {{Employing Batch Reinforcement Learning to Control Gene Regulation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2013},
}
Figure