The evaluation of classifiers or learning algorithms is not a topic that has, generally, been given much thought in the fields of Machine Learning and Data Mining. More often than not, common off-the-shelf metrics or approaches such as Accuracy, Precision/Recall and ROC Analysis are applied without much attention being paid to their meaning. Similarly, the validation of our algorithms is done, almost exclusively, on the UCI Repository, without much thought being put into how representative these data sets are to real-world conditions.

The purpose of this talk is to underline some of the problems that can arise from our current practices. It will then describe a new framework for classifier performance evaluation that views the problem as one of visualization of high-dimensional data. It will conclude by showing how the creation of carefully constructed artificial data sets can help mitigate the shortcomings of the UCI data sets in, at least, one particular real-world setting, that of changing environments.

Dr. Nathalie Japkowicz is an Associate professor of Computer Science in the School of Information Technology and Engineering at the University of Ottawa. She was a visiting professor at Monash University, Clayton during the 2006-2007 School Year. She obtained her Ph.D. from Rutgers University in 1999. Her area of study is Machine Learning with special emphases on the class imbalance problem, one-class learning, machine learning applied to computer and nuclear security, text mining, and, more recently, performance evaluation for machine learning.