Artificial Neural Network Development by means of Genetic Programming with Graph Codification

Authors: Daniel Rivero, Julián Dorado, Juan R. Rabuñal, Alejandro Pazos, Javier Pereira

Abstract:

The development of Artificial Neural Networks (ANNs) is usually a slow process in which the human expert has to test several architectures until he finds the one that achieves best results to solve a certain problem. This work presents a new technique that uses Genetic Programming (GP) for automatically generating ANNs. To do this, the GP algorithm had to be changed in order to work with graph structures, so ANNs can be developed. This technique also allows the obtaining of simplified networks that solve the problem with a small group of neurons. In order to measure the performance of the system and to compare the results with other ANN development methods by means of Evolutionary Computation (EC) techniques, several tests were performed with problems based on some of the most used test databases. The results of those comparisons show that the system achieves good results comparable with the already existing techniques and, in most of the cases, they worked better than those techniques.

Keywords: Artificial Neural Networks, Evolutionary Computation, Genetic Programming.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1077102

References:

[1] S. Haykin, Neural Networks (2nd ed.), Englewood Cliffs, NJ: Prentice Hall, 1999.
[2] J. R. Rabu├▒al and J. Dorado, (eds.) Artificial Neural Networks in Real- Life Applications, Idea Group Inc, 2005.
[3] J. R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, Cambridge, MA, MIT Press, 1992.
[4] J.R. Rabu├▒al, J. Dorado, A. Pazos, J. Pereira and D. Rivero, "A New Approach to the Extraction of ANN Rules and to Their Generalization Capacity Through GP". Neural Computation, vol. 16, n. 7. 2004. pp. 1483-1523.
[5] M. Bot, "Application of Genetic Programming to Induction of Linear Classification Trees", Final Term Project Report, Vrije Universiteit, Amsterdam, 1999.
[6] J. R. Rabu├▒al, J. Dorado, J. Puertas, A. Pazos, A. Santos and D. Rivero, "Prediction and Modelling of the Rainfall-Runoff Transformation of a Typical Urban Basin using ANN and GP", Applied Artificial Intelligence, 2003.
[7] R. S. Sutton, "Two problems with backpropagation and other steepestdescent learning procedure for networks", Proc. 8th Annual Conf. Cognitive Science Society, Hillsdale, NJ: Erlbaum, 1986, pp. 823-831.
[8] D. J. Janson and J. F. Frenzel, "Training product unit neural networks with genetic algorithms", IEEE Expert, vol. 8, 1993, pp. 26-33.
[9] G. W. Greenwood, "Training partially recurrent neural networks using evolutionary strategies", IEEE Trans. Speech Audio Processing, vol. 5, 1997, pp. 192-194.
[10] E. Alba, J. F. Aldana and J. M. Troya, "Fully automatic ANN design: A genetic approach", Proc. Int. Workshop Artificial Neural Networks (IWANN-93), Lecture Notes in Computer Science, vol. 686. Berlin, Germany: Springer-Verlag, 1993, pp. 399-404.
[11] H. Kitano, "Designing neural networks using genetic algorithms with graph generation system", Complex Systems, vol. 4, 1990, pp. 461-476.
[12] X. Yao and Y. Liu, "Toward designing artificial neural networks by evolution", Appl. Math. Computation, vol. 91, no. 1, 1998, pp. 83-90.
[13] S. A. Harp, T. Samad and A. Guha, "Toward the genetic synthesis of neural networks", Proc. 3rd Int. Conf. Genetic Algorithms and Their Applications, J.D. Schafer, Ed. San Mateo, CA: Morgan Kaufmann, 1989, pp. 360-369.
[14] S. Nolfi and D. Parisi, "Evolution of Artificial Neural Networks", Handbook of brain theory and neural networks, Second Edition, Cambridge, MA: MIT Press, 2002, pp. 418-421.
[15] P. Turney, D. Whitley and R. Anderson, "Special issue on the baldwinian effect", Evolutionary Computation, vol. 4, no. 3, 1996, pp. 213-329.
[16] A. Zomorodian, 1995. "Context-free Language Induction by Evolution of Deterministic Push-down Automata Using Genetic Programming", in Working Notes of the Genetic Programming Symposium, AAAI-95, Eric Siegel and John Koza, chairs. AAAI Press. 1995.
[17] Z. Fan, K. Seo, R. C. Rosenberg, J. Hu and E. D. Goodman, "Exploring Multiple Design Topologies Using Genetic Programming And Bond Graphs". GECCO 2002: Proceedings of the Genetic and Evolutionary Computation Conference. Springer-Verlag. 2002, pp. 1073-1080
[18] Z. Fan, K. Seo, J. Hu, R. C. Rosenberg and E. D. Goodman, "System- Level Synthesis of MEMS via Genetic Programming and Bond Graphs", Genetic and Evolutionary Computation -- GECCO-2003. Vol. 2724. 2003, pp. 2058-2071.
[19] F. Gruau, "Genetic micro programming of neural networks", in Kinnear, Jr., K. E., editor, Advances in Genetic Programming, chapter 24, MIT Press, 1994, pp. 495-518.
[20] S. Luke and L. Spector, "Evolving Graphs and Networks with Edge encoding: Preliminary Report". In Late Breaking Papers at the Genetic Programming 1996 Conference (GP96). J. Koza, ed. Stanford: Stanford Bookstore, 1996, pp. 117-124.
[21] A. Teller, "Evolving Programmers: The Co-evolution of Intelligent Recombination Operators", in Advances in Genetic Programming II, P. Angeline and K. Kinnear, editors. Cambridge: MIT Press., 1996.
[22] W. Kantschik, P. Dittrich, M. Brameier and W. Banzhaf, "MetaEvolution in Graph GP", Proceedings of EuroGP'99, LNCS, Vol. 1598. SpringerVerlag, 1999, pp. 15-28.
[23] R. Poli "Evolution of Graph-like Programs with Parallel Distributed Genetic Programming", Genetic Algorithms: Proceedings of the Seventh International Conference, 1997.
[24] W. Kantschik, W. Banzhaf, "Linear-Graph GP - A new GP Structure", in Proceedings of the 4th European Conference on Genetic Programming, EuroGP 2002, 2002.
[25] A. Teller A. and M. Veloso, "Internal reinforcement in a connectionist genetic programming approach", Artificial Intelligence. Vol. 120, N. 2, 2000, pp. 165-198.
[26] D. J. Montana, "Strongly typed genetic programming", Evolutionary Computation, Vol. 3, No. 2, 1995, pp. 199-200.
[27] C. J. Mertz and P. M. Murphy, UCI repository of machine learning databases. http://www-old.ics.uci.edu/pub/machine-learning-databases, 2002
[28] E. Cant├║-Paz and C. Kamath, "An Empirical Comparison of Combinations of Evolutionary Algorithms and Neural Networks for Classification Problems", IEEE Transactions on systems, Man and Cybernetics - Part B: Cybernetics, 2005, pp. 915-927.
[29] T. G. Dietterich, "Approximate statistical tests for comparing supervised classification learning algorithms", Neural Computation, Vol. 10, No. 7, 1998, pp. 1895-1924.