Oscillation Effect of the Multi-stage Learning for the Layered Neural Networks and Its Analysis

Authors: Isao Taguchi, Yasuo Sugai

Abstract:

This paper proposes an efficient learning method for the layered neural networks based on the selection of training data and input characteristics of an output layer unit. Comparing to recent neural networks; pulse neural networks, quantum neuro computation, etc, the multilayer network is widely used due to its simple structure. When learning objects are complicated, the problems, such as unsuccessful learning or a significant time required in learning, remain unsolved. Focusing on the input data during the learning stage, we undertook an experiment to identify the data that makes large errors and interferes with the learning process. Our method devides the learning process into several stages. In general, input characteristics to an output layer unit show oscillation during learning process for complicated problems. The multi-stage learning method proposes by the authors for the function approximation problems of classifying learning data in a phased manner, focusing on their learnabilities prior to learning in the multi layered neural network, and demonstrates validity of the multi-stage learning method. Specifically, this paper verifies by computer experiments that both of learning accuracy and learning time are improved of the BP method as a learning rule of the multi-stage learning method. In learning, oscillatory phenomena of a learning curve serve an important role in learning performance. The authors also discuss the occurrence mechanisms of oscillatory phenomena in learning. Furthermore, the authors discuss the reasons that errors of some data remain large value even after learning, observing behaviors during learning.

Keywords: data selection, function approximation problem, multistage leaning, neural network, voluntary oscillation.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1082373

References:

[1] Makoto Motoki´╝îSeiichi Koakutsu´╝îHironori Hirata:"A Supervised Learning Rule Adjusting Input-Output Pulse Timing for Pulsed Neural Network", The transactions of the Institute of Electronics, Information and Communication Engineers, Vol.J89-D-II, No.12, pp.2744-2756 (2006)
[2] Noriaki Kouda´╝îNobuyuki Matsui´╝îHaruhiko Nishimura:"A Multi- Layered Feed-Forward Network Based on Qubit Neuron Model", The transactions of the Institute of Electronics, Information and Communication Engineers, Vol.J85-D-II, No.4, pp.641-648 (2002)
[3] Isao Taguchi and Yasuo Sugai :"An Efficient Learning Method for the Layered Neural Networks Based on the Selection of Training Data and Input Characteristics of an Output Layer Unit", The Trans. of The Institute of Electrical engineers of Japan, Vol.129-C, No.4, pp.1208-1213, (2009)
[4] Nobuyuki Matsui and kenichi Ishimi:"A Multilayered Neural Network Including Neurons with fluctuated Threshold", The Trans. of The Institute of Electrical Engineers of Japan, Vol.114-C, No.11, pp.1208-1213 (1994)
[5] D.E.Falleman:"An Empirical Study of Learning Speed in Back- Propagation Network", Technical Report CMU-CS-88-162, Carnegie- Mellon University, Computer Sceinece Dept., (1988)
[6] M. Riedmiller and H. Braun:"A DirectbAdaptive Method for Faster Backpropagation Learning: The RPROP Algorithm", Proc. ICNN, San Fransisco, (1993)
[7] Isao Taguchi and Yasuo Sugai :"An Input Characteristic of Output Layer Units in the Layered Neural Networks and Its Application to an Efficient Learning", Proc. of the Electronics, Information and Systems Conference, Electronics, Information and systems Society, IEE. of Japan, pp.931-934 (2004)
[8] Isao Taguchi and Yasuo Sugai :"An Efficient Learning Method for the Layered Neural Networks Based on the Selection of Training Data and Input Characteristics of an Output Layer Unit", The Trans. of The Institute of Electrical engineers of Japan, Vol.129-C, No.4, pp.1208-1213, (2009)
[9] D.E.Rumelhart, J.L.McCleland, and the PDP Research Group: "Parallel Distributed Processing Vo.1", MIT Press (1986).
[10] Takashi Kanemaru and Masatoshi Sekine:"Oscillations and Synchronizations in Class 1 Neural Networks", TECHNICAL REPORT OF THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, NC2003-138, pp.17-22 (2004)
[11] Ting Wang and Yasuo Sugai:"A Wavelet Neural Network for the Approximation of Nonlinear multivariable Functions", The Trans. of The Institute of Electrical engineers of Japan, Vol.120-C, No.2, pp.185-193 (2000)
[12] Ting Wang and Yasuo Sugai:"A Wavelet Neural Network for the Approximation of Nonlinear Multivariable Functions", Proc.of IEEE International Conference on System, Man, and Cybernetics, III, pp.378- 383 (1999)
[13] K. Funahashi:"On the Approximate Realization of Continuous Mapping by Neural Networks", Vol.2, No.3, pp.183-192 (1989)
[14] Yasuo Sugai´╝îHiroshi Horibe, and Tarou Kawase:"Forecast of Daily Maximum Electric Load by Neural Networks Using the Standard Electric Load", The Trans. of The Institute of Electrical engineers of Japan, Vol.117-B, No.6, pp.872-879 (1997)
[15] Souiti Umehara, teru Yamazaki, and Yasuo Sugai:"A Precipitation Estimation System Based on Support Vector Machine and Neural Network", The transactions of the Institute of Electronics, Information and Communication Engineers, Vol.J86-D-II, No.7, pp.1090-1098 (2003)
[16] Charles K.Chui:"An Introduction to Wavelets", Academic Press (1992)
[17] L. ´╝½.Jones:"Constructive Approximations for Neural Networks by Sigmoidal Functions", Proc. IEEE, Vol.78, No.10, pp.(1990)
[18] B.Irie and S.Miyake:"Capabilities of Three Layered Perceptrons", Proc.ICNN, Vol.1, pp.641-648 (1988)