Extraction of Symbolic Rules from Artificial Neural Networks

Authors: S. M. Kamruzzaman, Md. Monirul Islam

Abstract:

Although backpropagation ANNs generally predict better than decision trees do for pattern classification problems, they are often regarded as black boxes, i.e., their predictions cannot be explained as those of decision trees. In many applications, it is desirable to extract knowledge from trained ANNs for the users to gain a better understanding of how the networks solve the problems. A new rule extraction algorithm, called rule extraction from artificial neural networks (REANN) is proposed and implemented to extract symbolic rules from ANNs. A standard three-layer feedforward ANN is the basis of the algorithm. A four-phase training algorithm is proposed for backpropagation learning. Explicitness of the extracted rules is supported by comparing them to the symbolic rules generated by other methods. Extracted rules are comparable with other methods in terms of number of rules, average number of conditions for a rule, and predictive accuracy. Extensive experimental studies on several benchmarks classification problems, such as breast cancer, iris, diabetes, and season classification problems, demonstrate the effectiveness of the proposed approach with good generalization ability.

Keywords: Backpropagation, clustering algorithm, constructivealgorithm, continuous activation function, pruning algorithm, ruleextraction algorithm, symbolic rules.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1071005

References:

[1] R. Andrews, J. Diederich and A. B., Tickle, "Survey and critique of techniques for extracting rules from trained artificial neural networks," Knowledge Based System, vol. 8, 1995, pp. 373-389.
[2] Ashish Darbari, "Rule Extraction from Trained ANN: A Survey," Technical Report, Department of Computer Science, Dresden University of Technology, Dresden, Germany, 2000.
[3] K. Saito and R. Nakano, "Medical diagnosis expert system based on PDP model," Proceedings of IEEE International Conference on Neutal Networks, IEEE Press, 1988, pp. 1255-1262.
[4] H. Liu and S. T. Tan, "X2R: A fast rule generator," Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Vancouver, CA, 1995.
[5] R. Setiono and Huan Liu, "Understanding neural networks via rule extraction," Proceedings of the 14th International Joint Conference on Artificial Intelligence, 1995, pp. 480-485.
[6] Olcay Boz, "Knowledge integration and rule extraction in neural networks," EECS Department, Lehigh University, 1995.
[7] R. Setiono, "Extracting rules from pruned neural networks for breast cancer diagnosis," Artificial Intelligence in Medicine, vol. 8, February 1996, pp. 37-51.
[8] R. Setiono and H. Liu, "Symbolic presentation of neural networks," IEEE Computer, March 1996, pp. 71-77.
[9] I. Taha and J. Ghosh, "Three techniques for extracting rules from feedforward networks," Intelligent Engineering Systems Through Artificial Neural Networks, vol. 6, pp. 23-28, ASME Press, St. Louis, 1996.
[10] R. Setiono, "Extracting rules from neural networks by pruning and hidden-unit node splitting," Neural Computation, vol. 9, 1997, pp. 205-225.
[11] R. Setiono, "Extracting M-of-N rules from trained neural networks," IEEE Transactions of Neural Networks, vol. 11, pp. 512-519, 2000.
[12] R. Setiono, W. K. Leow and Jack M. Zurada, "Extraction of Rules from Artificial Neural Networks for Nonlinear regression," IEEE Trans. of Neural Networks, vol. 13, 2002, pp. 564-577.
[13] R. Setiono, "Techniques for extracting rules from artificial neural networks," Plenary lecture presented at the 5th International Conference on Soft Computing and Information Systems, Iizuka, Japan, October 1998.
[14] R. Reed, "Pruning algorithms-A survey," IEEE Transactions on Neural Networks, vol. 4, pp. 740-747, 1993.
[15] Han Jiawei, Micheline Kamber, "Data Mining: Concepts and Techniques," Morgan Kaufmann Publisher: CA, 2001.
[16] L. Kaufman, P. J. Rousseeuw, "Finding Groups in Data: An Introduction to Cluster Analysis," John Wiley & Sons, 1990.
[17] T. Ng. Raymond, Jiawei Han, "Efficient and effective clustering methods for spatial data mining," VLDB Conference, Santiago, Chile, 1994.
[18] M. Monirul Islam and K. Murase, "A new algorithm to design compact two hidden-layer artificial neural networks", Neural Networks, vol. 4, 2001, pp. 1265-1278.
[19] J. R. Quinlan, "C4.5: Programs for Machine Learning," Morgan Kaufmann, San Mateo, CA, 1993.
[20] S. Russel and P. Norvig, "Artificial Intelligence: A Modern Approach," Prentice Hall, 1995.
[21] R. Agrawal, T. Imielinski, and A. Swami, "Database mining: A performance perspective," IEEE Transactions on Knowledge and Data Engineering, vol. 5, pp. 914-925, 1993.
[22] S-J Yen and A. L. P. Chen, "An efficient algorithm for deriving compact rules from databases," Proceedings of the Fourth International Conference on Database Systems for Advanced Applications, 1995.
[23] Prechelt, "Proben1-A Set of Neural Network Benchmark Problems and Benchmarking Rules", University of Karlsruhe, Germany, 1994.
[24] C. Blake, E. Keogh, and C. J. Merz, "UCI repository of of machine learning databases
[http://www.ics.uci.edu/~mlearn/MLRepository.htm]," Department of Information and Computer Science, University of California, Irvine, CA, 1998.
[25] L. Breiman, J. Friedman, R. Olshen, and C. Stone, "Classification and Regression Trees," Wadsworth and Brooks, Monterey, CA, 1984.
[26] D. T. Pham and M. S. Aksoy, "Rules: A simple rule extraction system," Expert Systems with Applications, vol. 8, 1995.