The development of accurate models and efficient algorithms for the analysis of multivariate categorical (multi-class) data is an important and long-standing problem in machine learning and computational statistics. In this talk I will describe new and existing models for the analysis of categorical data in the framework of latent Gaussian models, in particular with latent Gaussian processes. I will review both the latent Gaussian models framework and the use of Gaussian processes for problems in regression and classification, and then describe several Gaussian process models for categorical data. In particular, I will discuss the likelihood functions that can be used including the multinomial probit, multinomial logit, and a new stick-breaking likelihood. Using Markov chain Monte Carlo and variational learning approaches I will provide detailed comparisons for multinomial Gaussian process classification, as well as learning in categorical latent Gaussian graphical models. This comparison with existing logit/probit based latent Gaussian models will demonstrate the effectiveness of the stick-breaking likelihood in capturing correlation in discrete data, and additionally provides useful insight into the behaviour of these multi-class models and learning algorithms. Joint work with Emtiyaz Khan, Ben Marlin and Kevin Murphy.