An Evolutionary Statistical Learning Theory

Authors: Sung-Hae Jun, Kyung-Whan Oh

Abstract:

Statistical learning theory was developed by Vapnik. It is a learning theory based on Vapnik-Chervonenkis dimension. It also has been used in learning models as good analytical tools. In general, a learning theory has had several problems. Some of them are local optima and over-fitting problems. As well, statistical learning theory has same problems because the kernel type, kernel parameters, and regularization constant C are determined subjectively by the art of researchers. So, we propose an evolutionary statistical learning theory to settle the problems of original statistical learning theory. Combining evolutionary computing into statistical learning theory, our theory is constructed. We verify improved performances of an evolutionary statistical learning theory using data sets from KDD cup.

Keywords: Evolutionary computing, Local optima, Over-fitting, Statistical learning theory

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1061609

References:

[1] A. Ben-Hur, A. D. Horn, H. Siegelmann, V. Vapnik, "Support Vector Clustering," Journal of Machine Learning Research 2, 2001, pp. 125-137.
[2] L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone, Classification and Regression Trees, Wadsworth Inc., 1984.
[3] J. Cannady, "Artificial Neural Networks for Misuse Detection. National Information Systems," Proceedings of Security Conference, 1998.
[4] G. Casella, R. L. Berger, Statistical Inference, Duxbury Press, 1990.
[5] V. Cherkassky, F. Mulier, Learning From Data Concepts, Theory, and Methods, John Wiley & Sons, 1998.
[6] R. Cooley, B. Mobasher, J. Srivastava, "Web Mining: Information and Pattern Discovery on the World Wide Web," Proceeding of the 9th IEEE International Conference on Tools with Artificial Intelligence, 1997.
[7] R. Cooley, P. N. Tan, J. Srivastava, "Discovery of interesting usage patterns from web data," Technical Report TR 99-022, University of Minnesota, 1999.
[8] H. Debar, M. Becke, D. Siboni, "A Neural Network Component for an Intrusion Detection System," Proceedings of the IEEE Computer Society Symposium on Research in Security and Privacy, 1992, pp. 240-250.
[9] H. Debar, B. Dorizzi, "An Application of a Recurrent Network to an Intrusion Detection System," Proceedings of the International Joint Conference on Neural Networks, 1992, pp 78-483.
[10] A. E. Eiben, J. E. Smith, Introduction to Evolutionary Computing, Springer, 2003.
[11] S. M. Emran, M. Xu, N. Ye, Q. Chen, X. Li, "Probabilistic techniques for intrusion detection based on computer audit data," IEEE Transactions on Systems, Man and Cybernetics, Part A, vol.31, 2001, pp.266-274.
[12] D. Fisher, K. Hildrum, J. Hong, M. Newman, M. Thomas, R. Vuduc, "SWAMI: A Frame-work for Collaborative Filtering Algorithm Development and Evaluation," Proceeding of SIGIR 2000, ACM Press, 2000.
[13] D. B. Fogel, Evolutionary Computation, IEEE Press, 1995.
[14] L. J. Fogel, A. J. Owens, M. J. Walsh, Artificial Intelligence through Simulated Evolution, Wiley, Chichester, UK, 1996.
[15] A. K. Ghosh, Learning Program Behavior Profiles for Intrusion Detection, USENIX, 1999.
[16] J. W. Haines, R. P. Lippmann, D. J. Fried, M. A. Zissman, E. Tran, S. B. Boswell, "1999 DARPA Intrusion Detection Evaluation: Design and Procedures," Technical Report 1062, Lincoln Laboratory, MIT, 2001.
[17] S. Haykin, Neural Networks, Prentice Hall, 1999.
[18] S. Huet, A. Bouvier, M. A. Poursat, E. Jolivet, Statistical Tools for Nonlinear Regression, Springer Series in Statistics, Springer, 2003.
[19] S. H. Jun, "Hybrid Statistical Learning Model for Intrusion Detection of Networks," The KIPS Transaction: Part C, vol. 10-C, no. 6, 2003, pp. 705-710.
[20] S. H. Jun, "Web Usage Mining Using Support Vector Machine," Lecture Note in Computer Science, vol. 3512, 2005, pp. 349-356.
[21] S. Kumar, E. H. Spafford, "An Application of Pattern Matching in Intrusion Detection," Technical Report CSD-TR-94-013, Purdue University, 1994.
[22] W. Lee, S. J. Stolfo, K. W. Mok, "A data mining framework for building intrusion detection models," Proceedings of the 1999 IEEE Symposium on Security and Privacy, 1999, pp.120-132.
[23] R. J. A. Little, D. B. Rubin, Statistical Analysis with Missing Data, Wiley Inter-Science, 2002.
[24] B. Liu, "Fuzzy Random Chance-Constrained Programming," IEEE Transactions on Fuzzy Systems, vol. 9, Issue 5, 2001, pp. 713-720.
[25] J. Luo, S. M. Bridges, "Mining Fuzzy Association Rules and Fuzzy Frequency Episodes for Intrusion Detection," International Journal of Intelligent Systems, John Wiley & Sons, 2000, pp. 687-703.
[26] G. Mclachlan, D. Peel, Finite Mixture Models, John Wiley & Sons, Inc., 2000.
[27] T. M. Mitchell, Machine Learning, McGraw-Hill, 1997.
[28] T. M. Mitchell, An introduction to Genetic Algorithms, MIT Press, 1998.
[29] S. Mukkamala, G. Janoski, A. Sung, "Intrusion Detection Using Neural Networks and Support Vector Machines," Proceedings of International Symposium on Applications and the Internet Technology, 2000, pp. 209-216
[30] R. H. Myers, Classical and Modern Regression with Applications, Duxbury Press, 1990.
[31] A. T. Quang, Q. L. Zhang, X. Li, "Evolving Support Vector Machine Parameters," Proceedings of the First International Conference on Machine Learning and Cybernetics, 2002, pp. 548-551.
[32] J. Ryan, M. J. Lin, R. Miikkulainen, "Intrusion Detection with Neural Networks," Advances in Neural Information Processing Systems 10, Cambridge, MA: MIT Press, 1998.
[33] A. J. Smola, Regression estimation with support vector learning machines, Master-s thesis, Technische University, 1996.
[34] V. Vapnik, Statistical Learning Theory, John Wiley & Sons, Inc., 1998.
[35] X. Yao, "Evolving Artificial Neural Networks," Proceedings of the IEEE, vol. 87, Issue 9, 1999, pp. 1423-1447.
[36] http://www.ecn.purdue.edu/KDDCUP
[37] http://www.ll.mit.edu/IST/ideval/data