Hybrid Machine Learning Approach for Text Categorization

Authors: Nerijus Remeikis, Ignas Skucas, Vida Melninkaite

Abstract:

Text categorization - the assignment of natural language documents to one or more predefined categories based on their semantic content - is an important component in many information organization and management tasks. Performance of neural networks learning is known to be sensitive to the initial weights and architecture. This paper discusses the use multilayer neural network initialization with decision tree classifier for improving text categorization accuracy. An adaptation of the algorithm is proposed in which a decision tree from root node until a final leave is used for initialization of multilayer neural network. The experimental evaluation demonstrates this approach provides better classification accuracy with Reuters-21578 corpus, one of the standard benchmarks for text categorization tasks. We present results comparing the accuracy of this approach with multilayer neural network initialized with traditional random method and decision tree classifiers.

Keywords: Text categorization, decision trees, neural networks, machine learning.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1073114

References:

[1] Banerji, A. (1997). Initializing neural networks using decision trees. Computational Learning Theory and Natural Learning Systems, MIT Press, IV, 3-15.
[2] Frakes, W., and R. Baeza-Yates (1992). Information Retrieval: Data Structures & Algorithms, Prentice Hall.
[3] Haykin, S. (1994). Neural Networks: A comprehensive foundation. Macmillan College Publishing Comp., New York.
[4] Yang, Y., and J. Pedersen (1997). A comparative study on feature selection in text categorization. In Proceedings of ICML-97, 14th International Conference on Machine Learning, Nashville, US, 412-420.
[5] Yang,Y., and X. Liu (1999). A re-examination of text categorization methods. In Proceedings of SIGIR-99, 22nd ACM International Conference on Research and Development in Information Retrieval, Berkeley, US, 42-49.
[6] Lewis, D.D., and M. Ringuette (1994). A comparison of two learning algorithms for text categorization. In Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, Las Vegas, 81-93.
[7] Quinlan, J.R. (1993). C4-5: Programs for machine learning, Morgan Kaufmann, San Mateo, CA.
[8] Rumelhart, D.E., and J.L. Mcclelland (1986). Parallel distributed processing 1. MIT Press, Cambridge, MA.
[9] Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34(1), 1-47.
[10] van Rijsbergen, C.J. (1979). Information Retrieval. Butterworths, London.
[11] Wiener, E.D., J.O. Pedersen, and A. S. Weigend (1995). A neural network approach to topic spotting. In Proceedings of SDAIR-95, 4th Annual Symposium on Document Analysis and Information Retrieval, Las Vegas, 317-332.
[12] Joachims, T. (1998). Text categorization with support vector machines: learning with many relevant features. In Proceedings of ECML-98,10th European Conference on Machine Learning, 137-142.
[13] Dumais, S. T. (1991). Improving the retrieval information from external soures. Behaviour Research Methods, Instruments and Computers, 23(2), 229-236.