Evaluating Predictive Uncertainty Challenge

Evaluating Predictive Uncertainty Challenge

Home

Instructions

FAQ

Last Update:

21 Nov 2009 02:00:51

This challenge is supported by the PASCAL Network of Excellence

NEW! The Challenge was be discussed at the PASCAL Challenges Workhsop in Southampton on April 11 2005

[click here for the agenda]

Some ideas from the Workshop:

Defining good losses for probabilistic predictions is hard, since the losses might encourage strategies that are loss-dependent. Ideally one would want people to give an "honest" predictive distribution. Maybe one way of encouraging this would be to apply several losses that have contradictory properties. Another way could be to not to reveal the loss under which predictions will be evaluated.
Datasets and losses should not be chosen separately, since some losses are inappropriate for evaluating performance on certain problems. An example of this is using the log loss for regression in cases where it is expected to observe exactly the same target more than once. In such a case one could "cheat" by placing enormous amounts of density (with constant mass) on the target value in question.
Neural Networks are far from dead, and Kernel methods aren't the best. The best performing methods were not Kernel methods, and Neural Nets did a very good job. Averaging, in a Bayesian way or by means of ensemble methods, did seem to give very good results. For example, ensembles of decision trees perform very well, and so did Bayesian Neural Nets.
Though Bayesian methods did well, they seemed not to be the only competitive method. Indeed, non-Bayesian approaches, like regression on the variance, did also perform very well.
Datamining is probably as important as Machine Learning ...

Important dates

The Competition is Over! However, the site is re-open for submissions.
(new submissions will appear in a different colour than the white and yellow used for the actual competition, see results)
Results discussed at nips calibration workshop: Friday December 17, 2004. Slides[ps.gz]
NIPS 2004 Workshop on Calibration and Probabilistic Prediction in Machine Learning

Regression winners

Participants	Method	Slides
Kurogi Shuichi, Sawa Miho and Tanaka Shin-ya	Competitive Associative Nets + Cross Validation

Classification winners

Participants	Method	Slides
Radford Neal	Bayesian NN
Nitesh Chawla	Feature Selection + Random subspaces + Decision Trees

Other dataset winners

Dataset	Participants	Method	Slides
Stereopsis	Ed Snelson and Iain Murray	Mixture of Bayesian NN
Outaouais	Jukka Kohonen	Classification + Nearest neighbor

What is it about?

The goal of this challenge is to evaluate probabilistic methods for regression and for classification problems. A number of regression classification tasks are proposed. Training data (input-output pairs) are given, and the contestants are asked to predict the outputs associated to a set of validation and test inputs. These predictions are probabilistic and take the form of predictive distributions. The performance of the competing algorithms will be evaluated both with traditional losses that only take into account "point predictions" and with losses that evaluate the quality of the probabilistic predictions.

How do I participate?

Download the datasets ("Datasets" in left frame)
Look at the evaluation criteria ("Evaluation")
Format your predictions on the test inputs as required ("Instructions")
Submit your results ("Submission")
See how well you do compared to others ("Results")

Background

In many practical applications of Machine Learning there is a need for estimates of the accuracy of the predictions (or model uncertainty) that we will refer to as ``predictive uncertainties''. One example application where these are of crucial importance is active learning, where they are used to select the next training example which will bring most information. Predictive uncertainties come usually in the form of error-bars, confidence intervals, predictive distributions or posterior distributions. However, there is an apparent lack of consensus on how to produce good estimates of predictive uncertainty in the Machine Learning (or Statistical Learning) community, and of ways of evaluating them in the first place. There can be two ways of interpreting predictive uncertainties. One way is as estimates of the error made when predicting at a given point, while the second one could be as estimates of the overall generalization error (average error on future unseen examples). At this point we do not intend to exclude any of these, but rather aim at investigating both.

From the theoretical point of view, there are essentially two abstract ways of modelling uncertainty in a Machine Learning problem. An extreme way of presenting these approaches is to say that one of them consists in considering fixed training datasets and random functions and the other in placing the randomness in the training data and considering fixed ``true'' functions. The first approach is commonly used by the Bayesian community, and the second one by the Statistical Learning community. While predictive uncertainties arise naturally under the Bayesian paradigm, in Statistical Learning there is no consensus in how to define predictive uncertainties.

The Bayesian scheme starts from a prior, and is often attacked with questions like: ``what if the prior was incorrect''. Indeed, priors that do not reflect one's actual beliefs are often used for convenience reasons, such as analytic, or computational cost. Convenience priors might have an undesirable impact on the resulting predictive uncertainties. On the other hand, the Statistical Learning Theory community takes a completely different approach, and does not consider predictive distributions but instead focuses on tail bounds (confidence intervals). For example, assuming only that the data for a classification task is sampled at random and iid, bounds can be proved on the generalisation error (also called out-of-sample error or test-error). This theory is often attacked for the bounds being too loose to be of much value.

We feel that there is a need for formulating ways of evaluation of the quality of the predictive uncertainties. We do propose in this challenge measures of this quality.