An Empirical Evaluation of Performance of Machine Learning Techniques on Imbalanced Software Quality Data

Authors: Ruchika Malhotra, Megha Khanna

Abstract:

The development of change prediction models can help the software practitioners in planning testing and inspection resources at early phases of software development. However, a major challenge faced during the training process of any classification model is the imbalanced nature of the software quality data. A data with very few minority outcome categories leads to inefficient learning process and a classification model developed from the imbalanced data generally does not predict these minority categories correctly. Thus, for a given dataset, a minority of classes may be change prone whereas a majority of classes may be non-change prone. This study explores various alternatives for adeptly handling the imbalanced software quality data using different sampling methods and effective MetaCost learners. The study also analyzes and justifies the use of different performance metrics while dealing with the imbalanced data. In order to empirically validate different alternatives, the study uses change data from three application packages of open-source Android data set and evaluates the performance of six different machine learning techniques. The results of the study indicate extensive improvement in the performance of the classification models when using resampling method and robust performance measures.

Keywords: Change proneness, empirical validation, imbalanced learning, machine learning techniques, object-oriented metrics.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1112288

References:

[1] R. Malhotra and M. Khanna, “Investigation of Relationship between Object-oriented Metrics and Change Proneness,” Int. J. Mach. Learn. & Cyber., vol. 4, 2013, pp. 273-286.
[2] K.K. Aggarwal, Y. Singh, A.Kaur and R. Malhotra, “Empirical Analysis for Investigating the Effect of Object-Oriented Metrics on fault Proneness: A Replicated Case Study,” Software Process: Improvement and Practice, vol. 16, 2009, no. 1, pp. 39-62.
[3] A.G. Koru and J. Tian, “Comparing High-Change Modules and Modules with the Highest Measurement Values in two Large-Scale Open-Source Products,” IEEE Trans. Softw. Eng., vol. 31, 2005, no. 8, pp. 625-642.
[4] M. Kubat and S. Martin, “Addressing the Curse of Imbalanced Data Sets: One Sided Sampling,” in Proc. of 14th International Conf. on Machine Learning, Nashville, 1997, pp. 179-186.
[5] S. Visa, and A. Ralescu, "Issues in Mining Imbalanced Data Sets- A Review Paper," in Proc. of the 16th midwest Artificial Intelligence and Cognitive Science Conf., 2005, Ohio, pp. 67-73.
[6] H. He, and E.A. Garcia, “Learning from imbalanced data, IEEE Trans. Knowl. Data Eng., vol. 21, 2009, pp. 1263–1284.
[7] G.M. Weiss, "Mining with Rarity: A Unifying Framework.," ACM SIGKDD Explorations Newsletter, vol. 6, no. 1. pp. 7-19, 2004.
[8] X. Zhang and Y. Li, "An Empirical Study of Learning from Imbalanced Data." in Proc. of the 22nd Australasian Database Conf, Perth, 2011, pp. 85-94.
[9] R. Shatnawi, "Improving Software Fault-Prediction for Imbalanced Data." in Proc. of International Conf. on Innovations in Information Technology, Al Ain, 2012, pp. 54-59.
[10] C. Seiffert, T. M. Khoshgoftaar, J. V. Hulse, and A. Folleco, "An Empirical Study of the Classification Performance of Learners on Imbalanced and Noisy Software Quality Data,” Information Sciences, vol. 259, 2014, pp. 571-595.
[11] Y. Liu, A. An, and X. Huang, "Boosting Prediction Accuracy on Imbalanced Datasets with SVM Ensembles," In Advances in Knowledge Discovery and Data Mining, pp. 107-118, 2006.
[12] S. Wang and X. Yao, "Using Class Imbalance Learning for Software Defect Prediction," IEEE Transactions on Reliability, vol. 62, 2013, pp. 434-443.
[13] N. Seliya and T. M. Khoshgoftaar, "The Use of Decision Trees for Cost‐sensitive Classification: An Empirical Study in Software Quality Prediction," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 1, 2011, 448-459.
[14] G.M.Weiss, K. McCarthy, and B. Zabar, "Cost-sensitive Learning vs. Sampling: Which is Best for Handling Unbalanced Classes with Unequal Error Costs?" in Proc. of Internatioanl Conf. on Data Mining, Omaha NE, 2007 pp. 35-41.
[15] D. Rodriguez, I. Herraiz, R. Harrison, J. Dolado, and J. C. Riquelme, "Preliminary Comparison of Techniques for Dealing with Imbalance in Software Defect Prediction," In Proc. of the 18th International Conf. on Evaluation and Assessment in Software Engineering, London, 2014, p. 43.
[16] J.V. Hulse, T. M. Khoshgoftaar, A. Napolitano, and Randall Wald, "Feature Selection with High-dimensional Imbalanced Data." in Proc. of International Conf. on Data Mining Workshops, Florida, 2009, pp. 507-514.
[17] K. Gao, T. M. Khoshgoftaar, and Amri Napolitano, "Combining Feature Subset Selection and Data Sampling for Coping with Highly Imbalanced Software Data" in Proc. of 27th International Conf. on Software Engineering and Knowledge Engineering, Pittsburgh, 2015.
[18] L. Jeni, J. F. Cohn, and F. De La Torre, "Facing Imbalanced Data--Recommendations for the Use of Performance Metrics." In Proc. of Humaine Association Conf. on Affective Computing and Intelligent Interaction, Geneva, 2013, pp. 245-251.
[19] M. Tan, L. Tan, S. Dara, and C. Mayeux, "Online Defect Prediction for Imbalanced Data," in Proc. of 37th International Conf. on Software Engineering, Florence, 2015.
[20] N.V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, "SMOTE: Synthetic Minority Over-Sampling Technique," Journal of Artificial Intelligence Research, vol. 16, 2002, pp. 321-357.
[21] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, vol. 2, 2005.
[22] P. Domingos, “Metacost: A General Method for Making Classifiers Cost-sensitive,” In Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, CA, 1999, pp. 155-164.
[23] S. Chidamber and C. Kemerer, “A Metric Suite for Object- Oriented design,” IEEE Transactions on Software Engineering, vol. 20, 1994, pp. 476-493.
[24] R. Malhotra, K. Nagpal, P. Upmanyu & N. Pritam, “Defect collection and reporting system for Git based open source software. in Proc. of International Conf. on Data Mining and Intelligent Computing, Delhi, 2014, pp. 1-7.
[25] M.A. Hall, “Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning,” in Proc. of the Seventeenth International Conf. on Machine Learning, CA, 2000, pp. 359-366.
[26] C.G. Weng, and J. Poon, “A New Evaluation Measure for Imbalanced Datasets,” in Proc. of the 7th Australasian Data Mining Conf., Sydney, 2008, pp. 27-32.