Unsupervised Clustering Methods for Identifying Rare Events in Anomaly Detection

Authors: Witcha Chimphlee, Abdul Hanan Abdullah, Mohd Noor Md Sap, Siriporn Chimphlee, Surat Srinoy

Abstract:

It is important problems to increase the detection rates and reduce false positive rates in Intrusion Detection System (IDS). Although preventative techniques such as access control and authentication attempt to prevent intruders, these can fail, and as a second line of defence, intrusion detection has been introduced. Rare events are events that occur very infrequently, detection of rare events is a common problem in many domains. In this paper we propose an intrusion detection method that combines Rough set and Fuzzy Clustering. Rough set has to decrease the amount of data and get rid of redundancy. Fuzzy c-means clustering allow objects to belong to several clusters simultaneously, with different degrees of membership. Our approach allows us to recognize not only known attacks but also to detect suspicious activity that may be the result of a new, unknown attack. The experimental results on Knowledge Discovery and Data Mining-(KDDCup 1999) Dataset show that the method is efficient and practical for intrusion detection systems.

Keywords: Network and security, intrusion detection, fuzzy cmeans, rough set.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1057295

References:

[1] R. Bace and P. Mell, "Intrusion Detection Systems", NIST Special Publications SP 800. 31 November 2001.
[2] D. Denning, "An intrusion-detection model," In IEEE computer society symposium on research in security and privacy, 1986, pp. 118-131.
[3] T. Lane, "Machine Learning techniques for the computer security", PhD thesis, Purdue University, 2000.
[4] W. Lee and S. Stolfo, "Data mining approaches for intrusion detection," Proceedings of the 7th USENIX security symposium, , 1998.
[5] D. Dagupta and F. Gonzalez, "An immunity-based technique to characterize intrusions in computer networks", IEEE Transactions on Evolutionary Computation, Vol. 6, June 2002, pp.28- 291.
[6] H. Jin, J. Sun, H. Chen, and Z. Han, "A Fuzzy Data Mining based Intrusion Detection System", Proceedings of 10thInternational Workshop on future Trends in Distributed Computing Systems (FTDCS04) IEEE Computer Society, Suzhou, China, May 26-28, 2004, pp. 191-197.
[7] J. Twycross, "Immune Systems, Danger Theory and Intrusion Detection", presented at the AISB 2004. Symposium on Immune System and Cognition, Leeds, U.K., March 2004.
[8] R.T. Alves, M.R.B.S. Delgado, H.S. Lopes, A.A. Freitas, "An artificial immune system for fuzzy-rule induction in data mining", Lecture Notes in Computer Science, Berlin: Springer Verlag, v. 3242, 2004, pp. 1011- 1020.
[9] W. Lee, S. Stolfo, and K. Mok, "A data mining framework for building intrusion detection models", Proceedings of the 1999 IEEE Symposium on Security and Privacy, May 1999, pp.120-132.
[10] A. Lazarevic, A. Ozgur, L. Ertoz, J. Srivastava, and V. Kumar, "A comparative study of anomaly detection schemes in network intrusion detection", In SIAM International Conference on Data Mining, 2003.
[11] R. Jensen and Q. Shen, "Rough and fuzzy sets for dimensionality reduction", Proceedings of the 2001 UK Workshop on Computational Intelligence, 2001, pp. 69-74.
[12] D. Sarjon and Mohd Noor Md Sap, "Association Rules using Rough Set and Association Rule Methods", Proceedings of 7th Pacific Rim International Conference on Artificial Intelligence (PRICAI-02), Tokyo, Japan, August 18-22, 2002, pp. 238-243.
[13] S. Theodoridis, K. Koutroubas, "Pattern recognition", Academic Press, 1999.
[14] S. Albayrak, Fatih Amasyali, Fuzzy c-means clustering on Medical Diagnostic Systems, International XII. Turkish Symposium on Artificial Intelligence and Neural Networks, TAINN 2003.
[15] J. Bezkek, "Pattern Recognition with Fuzzy Objective Function Algorithms", Plenum Press, USA, 1981.
[16] KDD data set, 1999; http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html
[17] P. Laskov, K. Rieck, C. Schäfer, K.R. Müller, "Visualization of anomaly detection using prediction sensitivity", Proceeding of Sicherheit, April 2005, pp. 197-208.
[18] MathWorks, Statistical Toolbox for User-s Guide, MathWorks, 2001.
[19] A. Ôêà hrm, "ROSETTA Technical Reference Manual", Department of Computer and Information Science, Norwegian University of Science and Technology (NTNU), Trondheim, Norway, 2000.
[20] W. Chimphlee, Abdul Hanan Abdullah, Mohd Noor Md Sap and S. Chimphlee, "Unsupervised Anomaly Detection with Unlabeled Data using Clustering", Proc. Int. Conf. on ICT- Mercu Buana ICT2005. pp. 42-49.
[21] A. Lazarevic, A. Ozgur, L. Ertoz, J. Srivastava, and V. Kumar, "A comparative study of anomaly detection schemes in network intrusion detection". In SIAM; International Conference on Data Mining, 2003.
[22] Zhengxin Chen, Data Mining and Uncertain Reasoning - An Integrated Approach, Wiley, 2001.
[23] Fernando Godínez, Dieter Hutter, Raul Monroy "Attribute Reduction for Effective Intrusion Detection". AWIC 2004: 74-83.
[24] W. Chimphlee, Abdul Hanan Abdullah, Mohd Noor Md Sap, S.Chimphlee, and S. Srinoy, Unsupervised Anomaly Detection without Prior Knowledge Using Clustering, International workshop on information Technology 2005 (IAIT2005), 25-26 November 2005., Thailand.
[25] W. Chimphlee, Mohd Noor Md Sap, Abdul Hanan Abdullah, and S. Chimphlee, Semi-Supervised Learning to Identify Suspicious Activity for Anomaly Detection, 3rd International Conference on Computational Intelligence, Robotics and Autonomous Systems (CIRAS2005), 13-16 December 2005, Singapore.