While agreement statistics have received wide attention in fields such as medicine and psychology, their use has been limited in machine learning. Conventional performance metrics such as accuracy do not account for coincidental concordances among labelings and hence are not always suitable. Chance corrected metrics (e.g., Cohen's kappa), that have been proposed to deal with these issues do not readily generalize to multi-class scenarios where labeling come from multiple sources, especially when these sources generating data-labels are fixed. I will present novel chance-corrected generalized agreement metrics for: i) assessing agreement between multiple label sets (e.g., between a classifier and the true labels, or between multiple classifiers) in multi-class scenario; and ii) assessing agreement of a new labeling against a fixed set of labelings. I will also highlight some important shortcomings of the conventional approaches currently used for the purpose. This work can have significant implications in more principled agreement assessment across a variety of areas such as ensemble classifier evaluation, inter-rater agreement measurement, classifier evaluation in the absence of gold standard labeling as well as assessment against various HITs (Human Intelligence Tasks) such as those addressed by Amazon's Mechanical Turk.