Eclectic Rule-Extraction from Support Vector Machines

Authors: Nahla Barakat, Joachim Diederich

Abstract:

Support vector machines (SVMs) have shown superior performance compared to other machine learning techniques, especially in classification problems. Yet one limitation of SVMs is the lack of an explanation capability which is crucial in some applications, e.g. in the medical and security domains. In this paper, a novel approach for eclectic rule-extraction from support vector machines is presented. This approach utilizes the knowledge acquired by the SVM and represented in its support vectors as well as the parameters associated with them. The approach includes three stages; training, propositional rule-extraction and rule quality evaluation. Results from four different experiments have demonstrated the value of the approach for extracting comprehensible rules of high accuracy and fidelity.

Keywords: Data mining, hybrid rule-extraction algorithms, medical diagnosis, SVMs

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1055511

References:

[1] A.B. Tickle, R.Andrews, M.Golea, and J.Diederich, "The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural network", IEEE Trans. Neural Networks, vol. 9(6), pp. 1057-1068, 1998.
[2] R. Andrews, J. Diederich, and A.B. Tickle, "A Survey and Critique of Techniques For Extracting Rules From Trained Artificial Neural Networks", Knowledge Based Systems, vol. 8, pp. 373-389, 1995.
[3] R. Davis, B.G. Buchanan, and E. Shortcliff, "Production Rules as a Representation for a Knowledge Based Consultation Progra", J. Artificial Intelligence, vol. 8(1), pp.15-45, 1977.
[4] S. Gallant, "Connectionist Expert System", Communications of the ACM, vol. 31 (2), pp. 152-169, 1988.
[5] S. Sestito and T. Dillon, "Automated Knowledge Acquisition of Rules With Continuously Valued Attributes", in Proc.12th International Conference on Expert Systems and their Applications (AVIGNON'92), Avignon -France, 1992, pp. 645-656.
[6] M.W. Craven, and J.W. Shavlik, "Using Sampling and Queries to Extract Rules From Trained Neural Networks", in Proc. of the 11th International Conference on Machine learning, NJ, 1994, pp.37-45.
[7] G. Towell, and J. Shavlik. "The Extraction of Refined Rules From Knowledge Based Neural Networks", J. Machine Learning, vol. 131, pp.71-101, 1993.
[8] M.W. Craven, and J.W. Shavlik, "Extracting Tree-Structured Representation of Trained Networks", Advances in Neural Information Processing Systems, vol. 8, pp.24-30, 1996.
[9] A. Tickle, A, M. Orlowski, M, J. Diederich, "DEDEC: A Methodology for Extracting Rules from Trained Artificial Neural Networks. "In: Andrews, R.; Diederich, J. (Eds.): Rules and Networks. Brisbane, Qld.: QUT Publication 1996, 90-102.
[10] R. Mitsdorffer, J. Diederich, and C. Tan, "Rule-extraction from Technology IPOs in the US Stock Market", presented at ICONIP02, Singapore, 2002.
[11] H. Khuu, H.K. Lee, J-L, Tsai. " Machine learning with Neural Networks and support vector machines", University of Wisconsin, unpublished, 2004
[12] C. Burges, A tutorial on support vector machines for pattern recognition. data mining and knowledge discovery, Boston, Kluwer Academic publishers, 1998.
[13] V. Kecman, Learning and Soft Computing. Cambridge, MA: MIT Press, 2001
[14] V. Kecman, "Learning by Support Vector Machines from Huge Data Sets", presented at KES 2004, Eighth international conference on knowledge-based intelligent information & engineering systems, 20-24 September, 2004, Wellington, New Zeland.
[15] H. N├║├▒ez, C. Angulo, and A.Catala, "Rule-extraction from Support Vector Machines", in Proc. of European Symposium on Artificial Neural Networks, Burges, 2002, pp.107-112.
[16] N. Barakat , and J. Diederich, "Learning-based rule-extraction from support vector machines: Performance on benchmark data sets": Kasabov, N., Chan, Z.S.H. (Eds.), in Proc. of the conference on Neuro- Computing and Evolving Intelligence, Auckland, New Zealand, Auckland. Knowledge Engineering and Discovery Research Institute (KEDRI) (2004).
[17] J. Diederich , and N. Barakat, "Hybrid rule-extraction from support vector machines" in Proc. of IEEE conference on cybernetics and intelligent systems, Singapore, 2004, pp. 1270-1275.
[18] http://www.rulequest.com
[19] http://www.ics.uci.edu/~mlearn/MLRepository.html
[20] http://svmlight.joachims.org/
[21] M. Craven and J. Shavlik, "Rule Extraction: Where Do We Go from Here?", Department of Computer Sciences, Machine Learning Research Group Working Paper 99-1, 1999.
[22] A.Tickel, F. Maire, G. Bologna, J. Diederich." Lessons from past, current issues and future research directions in extracting the knowledge embedded in Artificial Neural Networks". Lecture notes in computer science, Hybrid Neural Systems, vol. 1778, revised papers from a workshop 1998, pp. 226 - 239