Multi-Context Recurrent Neural Network for Time Series Applications

Authors: B. Q. Huang, Tarik Rashid, M-T. Kechadi

Abstract:

this paper presents a multi-context recurrent network for time series analysis. While simple recurrent network (SRN) are very popular among recurrent neural networks, they still have some shortcomings in terms of learning speed and accuracy that need to be addressed. To solve these problems, we proposed a multi-context recurrent network (MCRN) with three different learning algorithms. The performance of this network is evaluated on some real-world application such as handwriting recognition and energy load forecasting. We study the performance of this network and we compared it to a very well established SRN. The experimental results showed that MCRN is very efficient and very well suited to time series analysis and its applications.

Keywords: Gradient descent method, recurrent neural network, learning algorithms, time series, BP

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1330475

References:

[1] M. Boden, `A guide to recurrent neural networks and backpropagation',The DALLAS project. Report from the NUTEKsupported project AIS-8, SICS.Holst: Application of Data Analysis with Learning Systems, (2001).
[2] M.A. Castano, F. Casacuberta, and A. Bonet, Training Simple recurrent Networks Through Gradient Descent Algorithm, volume 1240 of ISBN 3-540-63047-3, chapter Biological and Arti cial Computation: From Neuroscience to Technology, pp. 493--500, Eds. J. Mira and R. Moreno-Diaz and J. Cabestany, Springer Verlag, 1997.
[3] W. Charytoniuk and M-S. Chen, `Very short-term load forecasting using neural networks', IEEE Tran. On Power Systems, 15(1), 1558--- 1572, (2000).
[4] B.J. Chen, M.W. Change, and C.J. Lin, `Eunite network competition: Electricity load forecasting', Technical report, In EUNITE 2001 symposium, a forecasting competition, (2001).
[5] Y. Cheng, T.W. Karjala, and D.M. Himmelblau, `Closed loop nonliner process identi cation using internal recurrent nets', In Neural Networks, 10(3), pp. 573--586, (1997).
[6] A. Corradini and P. Cohen, `Multimodal speech-gesture interface for hands-free painting on virtual paper using partial recurrent neural networks for gesture recognition', in Proc. of the Int'l Joint Conf. on Neural Networks (IJCNN'02), volume 3, pp. 2293--2298, (2002).
[7] B. de Vries and J.C. Principe, `A theory for neural networks with time delays', in NIPS-3: Proceedings of the 1990 conference on Advances in neural information processing systems 3, pp. 162--168, San Francisco, CA, USA, (1990). Morgan Kaufmann Publishers Inc.
[8] Georg Dorffner, `Neural networks for time series processing', Neural Network World, 6(4), pp. 447--468, (1996).
[9] W. Duch and N.Jankowski, `Transfer functions: Hidden possibilities for better neural networks', in ESANN'2001 proceedings European Symposium on Arti cial Neural Networks, ISBN 2-930307 01-3, pp. 25-27, Belgium, (2001). D-Facto public.
[10] J.L. Elman, `Finding structure in time', Cognitive Science, 14(2), pp.179--211, (1999).
[11] D. Esp, `Adaptive logic networks for east slovakian electrical load forecasting', Technical report, In EUNITE 2001 symposium, a forecasting competition, (2001).
[12] D.V. Prokhorov E.W. Saad and D.C. Wunsch, `Comparative study of stock trend prediction using time delay, recurrent and probabilistic neural networks', IEEE Transactions on Neural Networks, 6(9), PP.1456--1470, (1998).
[13] L. Fausett, Backpropagation Through Time and Derivative Adaptive Critics: A Common Framework for Comparison, chapter Englewood Cliffs, NJ: Prentice Hall, 1994.
[14] G. Gross and F. D. Galianan, `Short-term load forecasting', In Proceedings of the IEEE., 75(12), 1558--1572, (1987).
[15] I. Guyon, L. Schomaker, R. Plamondon, M. Liberman, and S. janet, `Unipen project of on-line data exchange and recognizer benchmarks', in Proceedings of the 12th International Conference on Pattern Recognition, ICPR'94, pp. 29--33, Jerusalem, Israel, (October 1994).
[16] S. Haykin, Neural Networks, A Comprehensive Foundation, MacMillan Publishing Company, New York, 1994.
[17] A. Herve and E. Betty, `Neural networks, quantitative applications', in In the Social Sciences, volume 124, London: Sage Publications, (1999).
[18] B. Q. Huang and M-T. Kechadi, `A recurrent neural network recogniser for online recognition of handwritten symbols.', in ICEIS (2), pp. 27--34, (2005).
[19] B. Q. Huang, T. Rashid, and T. Kechadi, `A new modi ed network based on the elman network', in Proceedings of IASTED International Conference on Arti cial Intelligence and Application, ed., M. H. Hamza, volume 1 of ISBN: 088986-404-7, pp. 379--384, Innsbruck, Austria, (2004). ACTA Press.
[20] M.I. Jordan, `Attractor dynamics and parallelism in a connectionist sequential machine.', in Proceedings of the 8th Annual Conference of the Cognitive Science Society, Englewood Cliffs, NJ: Erlbaum, pp. 531--546. Reprinted in IEEE Tutorials Series, New York: IEEE Publishing Services, 1990, (1986).
[21] I. King and J. Tindle, `Storage of half hourly electric metering data and forecasting with arti cial neural network technology', Technical report, In EUNITE 2001 symposium, a forecasting competition, (2001).
[22] W. Kowalczyk, `Averaging and data enrichment: Two approaches to electricity load forecasting', Technical report, In EUNITE 2001 symposium, a forecasting competition, (2001).
[23] S. Lawrence, C.L. Giles, and S. Fong, `Natural language grammatical inference with recurrent neural networks', IEEE Trans. on Knowledge and Data Engineering, 12(1), pp. 126--140, (2000).
[24] A. Lo, H. Mamaysky, and J. Wang, `Foundations of technical analysis: Computational algorithms, statistical inference, and empirical implementation', Journal of Finance 55, pp. 1705--1765, (2000).
[25] Yee-Ling LU, Man-Wai MAK, and Wan-Chi SIU, `Application of a fast real time recurrent learning algorithm to text-to-phone conversion', in Proceedings of the International Conference of Neural Networks, volume 5, pp. 2853--2857, (1995).
[26] Simone Marinai, Marco Gori, and Giovanni Soda, `Arti cial neural networks for document analysis and recognition.', IEEE Trans. Pattern Anal. Mach. Intell., 27(1), pp. 23--35, (2005).
[27] A. D. Papalxopoulos and T. C. Hiterbeg, `A regression-based approach to short-term load forecasting', In IEEE Tran. On Power Systems, 4(1), pp. 1535--1547, (1990).
[28] D. Park, M. El-Sharkawia, R. Marks, A. Atlas, and M. Damborg, `Electic load forecasting using arti cial neural networks', IEEE Trans. on Power Systems, 6(2), 442--449, (1991).
[29] D. C. Plaut, `Semantic and associative priming in a distributed attractor network', in Proceedings of the 17th Annual Conference of the Cognitive Science Society, pp. 37--42, Hillsdale, (1995). NJ:Erlbaum.
[30] D. Prokhorov, Backpropagation Through Time and Derivative Adaptive Critics: A Common Framework for Comparison, chapter Learning and Approximate Dynamic Programming, Wiley, 2004.
[31] T. Rashid, B. Q. Huang, and T. Kechadi, `A new simple recurrent network with real-time recurrent learning process', in The 14th Irish Artifcial Intelligence and Cognitive Science (AICS'03), ed., Padraig Cunningham, volume 1, pp. 169--174, Dublin, Ireland, (2003).
[32] T. Rashid and T. Kechadi, `A practical approach for electricity load forecasting', in The proceeding WEC'05, The ThirdWorld Enformatika, ed., C. Ardal, volume 5 of ISBN 975-98458-4-9, pp. 201--205, Isanbul, Turky, (2005). ACTA Press.
[33] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning Internal Representations by Error Propagation In D. E. Rumelhart, et al. (Eds.), Parallel Distributed Processing: Explorations in the Microstructures of Cognition, 1: Foundations, MA: MIT Press, Cambridge, 1962.
[34] M. Schnekel, I. Guyon, and D. Henderson, `On-line cursive script recognition using time delay networks and hidden markove models', in Proc. ICASSP'94, volume 2, pp. 637--640, Adelaide, Australia, (April 1994).
[35] P. Stagge and B. Sendho, `Organization of past states in recurrent neural networks: Implicit embedding', in Proc. The Internation conference Computational Intelligence for Modelling, Control & Automation, pp. 21--27, Amsterdam, (1999). IOS Press.
[36] J.C. Tomasz and M.Z. Jacek, `Neural network tools for stellar light prediction', in Proc. of the IEEE Aerospace Conference, volume 3, pp. 415--422, Snowmass, Colorado, USA, (February 1997).
[37] P. J. Werbos, `Backpropagation through time: What it does and how to do it', in Proceedings of the IEEE, volume 78, pp. 1550--1560, (1990).
[38] William H. Wilson, `Learning performance of networks like elman's simple recurrent netwroks but having multiple state vectors', Workshop of the 7th Australian Conference on Neural Networks, Australian National University Canberra, (1996).
[39] Shi XH, YC. Liang, and X. Xu, `An improved elman model and recurrent bck-propagation control neural networks', Journal of Software, 6(14), 1110--1119, (2003).