Ensembles for sequence learning

Christos DimitrakakisInfoscience 2006 · DOI

TL;DR

This thesis explores how ensemble methods, which combine multiple models, can be effectively applied to sequential learning tasks like speech recognition and reinforcement learning. It introduces novel techniques for handling uncertainty in these domains, leading to improved performance and more robust decision-making.

What Problem Does This Paper Address?

In many machine learning tasks, especially those involving sequences, we face uncertainty about the best model or hypothesis to explain the data. Traditional methods often ignore this uncertainty by selecting a single best solution. This thesis addresses the challenge of representing and leveraging this uncertainty, particularly in sequential learning problems where decisions have future consequences. The paper focuses on two key areas: speech recognition, where accurately modeling sequential acoustic data is crucial, and reinforcement learning, where agents must make decisions in uncertain environments to maximize future rewards.

Methods

The paper proposes and examines several novel approaches:

  • Ensemble Methods for Speech Recognition: It applies and adapts ensemble techniques like bagging and boosting to phoneme classification and speech recognition. This includes developing methods for training phonetic mixtures and a boosting scheme specifically designed to minimize word error rates by incorporating temporal credit assignment.
  • Reinforcement Learning Ensembles: The thesis investigates using reinforcement learning to train ensembles of classifiers. This involves an agent learning to select experts for classification and training.
  • Handling Uncertainty: A core contribution is the development of methods to represent and utilize uncertainty. This includes:
  • Gradient-based estimates: Estimating parameter accuracy using gradients and Hessians.
  • Ensemble-based representations: Using populations of estimators (similar to bagging) or particle filters to approximate belief distributions.
  • Optimal Exploration Algorithms: Novel algorithms for balancing exploration (gathering information) and exploitation (using current knowledge) in reinforcement learning, particularly for bandit problems. These algorithms aim to be nearly optimal without requiring extensive prior parameter tuning.

Datasets

The paper utilizes the following datasets:

  • Speech Recognition: OGI Numbers 95 (N95) dataset for phoneme classification and continuous speech recognition.
  • Classification: 9 UCI benchmark datasets for evaluating reinforcement learning ensembles.
  • Bandit Problems: Random bandit problems with varying numbers of arms (16, 32, 64, 128) and Bernoulli reward distributions.

Key Insights and Ideas

  • Ensembles for Sequential Tasks: Ensemble methods, while successful in static tasks, present unique challenges in sequential learning. This thesis demonstrates that with appropriate adaptations, they can yield significant improvements.
  • Uncertainty is Key: Explicitly representing and utilizing uncertainty is crucial for robust decision-making in both speech recognition (e.g., handling noisy phonetic segmentation) and reinforcement learning (e.g., exploring unknown environments).
  • Boosting for Word Error Rate: Directly optimizing for word error rate in speech recognition, rather than just phoneme classification, is a more principled approach and can be achieved through specialized boosting techniques.
  • RL for Ensemble Training: Reinforcement learning offers a viable alternative to supervised methods for training and controlling ensembles of classifiers, particularly when dealing with adaptive policies.
  • Optimal Exploration is Principled: The exploration-exploitation trade-off can be framed as a principled problem, leading to algorithms that balance these needs based on model uncertainty and reward horizon, rather than relying on ad-hoc heuristics.

Key Results

  • Speech Recognition: The proposed boosting approach for word error rate minimization showed promising results, with some cases achieving substantial improvements over baseline systems. Bagging was found to be particularly effective in reducing the impact of noisy phonetic segmentation.
  • Classification: Reinforcement learning-based ensembles generally improved performance over single classifiers on UCI benchmark datasets.
  • Bandit Problems: The developed optimistic exploration algorithms demonstrated competitive or superior performance compared to standard methods like e-greedy and softmax, especially in tasks with longer effective reward horizons. Bayesian estimates generally performed best when their priors matched the experimental conditions.

Limitations and Future Work

  • Speech Recognition: The paper notes that the application of these methods to large vocabulary speech recognition remains a future research direction. Further work is also suggested on developing better methods for combining experts during decoding, beyond current multi-stream techniques.
  • Reinforcement Learning: The thesis acknowledges that the proposed exploration algorithms, while principled, may still require careful tuning of certain parameters. Future work could explore other types of uncertainty representations and extend the methods to partially observable Markov decision processes (POMDPs) and continuous state-action spaces.
  • Generalizability: The effectiveness of certain methods, like the Bayesian approach, is highly dependent on the accuracy of prior knowledge. The paper suggests exploring methods that require less prior knowledge or can naturally incorporate it.
  • Model-based Exploration: The authors suggest that model-based exploration techniques, where an explicit model of the environment is maintained, could lead to more efficient exploration and are a worthwhile topic for future research.

Summary updated

Abstract

This thesis explores the application of ensemble methods to sequential learning tasks. The focus is on the development and the critical examination of new methods or novel applications of existing methods, with emphasis on supervised and reinforcement learning problems. In both types of problems, even after having observed a certain amount of data, we are often faced with uncertainty as to which hypothesis is correct among all the possible ones. However, in many methods for both supervised and for reinforcement learning problems this uncertainty is ignored, in the sense that there is a single solution selected out of the whole of the hypothesis space. Apart from the classical solution of analytical Bayesian formulations, ensemble methods offer an alternative approach to representing this uncertainty. This is done simply through maintaining a set of alternative hypotheses. The sequential supervised problem considered is that of automatic speech recognition using hidden Markov models. The application of ensemble methods to the problem represents a challenge in itself, since most such methods can not be readily adapted to sequential learning tasks. This thesis proposes a number of different approaches for applying ensemble methods to speech recognition and develops methods for effective training of phonetic mixtures with or without access to phonetic alignment data. Furthermore, the notion of expected loss is introduced for integrating probabilistic models with the boosting approach. In some cases substantial improvements over the baseline system are obtained. In reinforcement learning problems the goal is to act in such a way as to maximise future reward in a given environment. In such problems uncertainty becomes important since neither the environment nor the distribution of rewards that result from each action are known. This thesis presents novel algorithms for acting nearly optimally under uncertainty based on theoretical considerations. Some ensemble-based representations of uncertainty (including a fully Bayesian model) are developed and tested on a few simple tasks resulting in performance comparable with the state of the art. The thesis also draws some parallels between a proposed representation of uncertainty based on gradient-estimates and on"prioritised sweeping" and between the application of reinforcement learning to controlling an ensemble of classifiers and classical supervised ensemble learning methods.
BibTeX
@inproceedings{dimitrakakis2006ensembles,
  author = {Christos Dimitrakakis},
  title = {{Ensembles for sequence learning}},
  booktitle = {Infoscience},
  year = {2006},
  doi = {10.5075/epfl-thesis-3518},
}
Figure