An ensemble of Weighted Support Vector Machines for Ordinal Regression

Authors: Willem Waegeman, Luc Boullart

Abstract:

Instead of traditional (nominal) classification we investigate the subject of ordinal classification or ranking. An enhanced method based on an ensemble of Support Vector Machines (SVM-s) is proposed. Each binary classifier is trained with specific weights for each object in the training data set. Experiments on benchmark datasets and synthetic data indicate that the performance of our approach is comparable to state of the art kernel methods for ordinal regression. The ensemble method, which is straightforward to implement, provides a very good sensitivity-specificity trade-off for the highest and lowest rank.

Keywords: Ordinal regression, support vector machines, ensemblelearning.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1329452

References:

[1] A. Agresti, Categorical Data Analysis, 2nd version. John Wiley and Suns Publications, 2002.
[2] J. DelCoz, G. Bayn, J. Dez, O. Luaces, A. Bahamonde, and C. Saudo, "Trait selection for assessing beef meat quality using non-linear svm," in Proceedings of the conference on Neural Information Processing Systems, Vancouver, Canada, 2004, pp. 321-328.
[3] R. Herbrich, "The trueskill(tm) ranking system," 2005, http://www.research.microsoft.com/mlp/trueskill/.
[4] S. Kramer, G. Widmer, B. Pfahringer, and M. Degroeve, "Prediction of ordinal classes using regression trees," Fundamenta Informaticae, vol. 24, pp. 1-15, 2000.
[5] K. Cao-Van, "Supervised ranking, from semantics to algorithms," Ph.D. dissertation, Ghent University, Belgium, 2003.
[6] R. Potharst and J. Bioch, "Decision trees for ordinal classification," Intelligent Data Processing, vol. 4, no. 2, 2000.
[7] W. Cohen, R. Schapire, and Y. Singer, "Learning to order things," in Advances in Neural Information Processing Systems 10, 1998.
[8] K. Crammer and Y. Singer, "Pranking with ranking," in Proceedings of the conference on Neural Information Processing Systems (NIPS), 2001.
[9] E. Harrington, "Online ranking/collaborative filtering using the perceptron algorithm," in Proceedings of the 20th International Conference on Machine Learning, Washington, USA, 2003.
[10] R. Herbrich, T. Graepel, and K. Obermayer, "Large margin rank boundaries for ordinal regression," in Advances in Large Margin Classifiers, 2000, pp. 115-132.
[11] J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
[12] A. Shashua and A. Levin, "Ranking with the large margin principle: two approaches," Advances in Neural Information Processing Systems, vol. 15, 2003.
[13] W. Chu and S. Keerthi, "New approaches to support vector ordinal regression," in Proceedings of the 22th International Conference of Machine Learning, Bonn, Germany, 2005, pp. 321-328.
[14] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Springer, 2001.
[15] B. Scholkopf and A. Smola, Learning with Kernels, Support Vector Machines, Regularisation, Optimization and beyond. The Mit Press, 2002.
[16] E. Frank and M. Hall, "A simple approach to ordinal classification," in Proceedings of the European Conference on Machine Learning, 2000.
[17] S. Abe and T. Inoue, "Fuzzy support vector machines for multiclass problems," in European Symposium on Artificial Neural Networks, 2002, pp. 113-118.
[18] C.-C. Chang and C.-J. Lin, "LIBSVM: a library for support vector machines," 2001, http://www.csie.ntu.edu.tw/ cjlin/libsvm.