Enhancing Multi-Frame Images Using Self-Delaying Dynamic Networks

Authors: Lewis E. Hibell, Honghai Liu, David J. Brown

Abstract:

This paper presents the use of a newly created network structure known as a Self-Delaying Dynamic Network (SDN) to create a high resolution image from a set of time stepped input frames. These SDNs are non-recurrent temporal neural networks which can process time sampled data. SDNs can store input data for a lifecycle and feature dynamic logic based connections between layers. Several low resolution images and one high resolution image of a scene were presented to the SDN during training by a Genetic Algorithm. The SDN was trained to process the input frames in order to recreate the high resolution image. The trained SDN was then used to enhance a number of unseen noisy image sets. The quality of high resolution images produced by the SDN is compared to that of high resolution images generated using Bi-Cubic interpolation. The SDN produced images are superior in several ways to the images produced using Bi-Cubic interpolation.

Keywords: Image Enhancement, Neural Networks, Multi-Frame.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1331451

References:

[1] E. H. W. Meijering, "A chronology of interpolation," in Proceedings of the IEEE, vol. 90, no. 3, March 2002, pp. 319-342.
[2] T. M. Lehmann, C. Gonner, and K. Spitzer, "Survey: Interpolation methods in medical image processing," IEEE Trans. Med. Imag., vol. 18, no. 11, pp. 1049-1075, November 1999.
[3] S. Battiato, G. Gallo, M. Mancuso, G. Messina, and F. Stanco, "Analysis and characterization of super-resolution reconstruction methods," in SPIE Electronic Imaging 2003, Sensors, Cameras and Applications for Digital Photography, vol. 5017B-37, 2003.
[4] E. H. W. Meijering, W. J. Niessen, and M. A. Viergever, "Quantitative evaluation of convolution-based methods for medical image interpolation," Medical Image Analysis, vol. 5, no. 2, pp. 111-126, June 2001.
[5] E. H. W. Meijering, "Spline interpolation in medical imaging: Comparison with other convolution based approaches," in Signal Processing X: Theories and Applications, EUSIPCO 2000, M. Gabbouj and P. Kuosmanen, Eds., vol. 4. The European Association for Signal Processing, Tampere, 2000, pp. 1989-1996.
[6] M. Unser, A. Aldroubi, and M. Eden, "Enlargement or reduction of digital images with minimum loss of information," IEEE Trans. Image Processing, vol. 4, no. 3, pp. 247-258, 1995.
[7] N. A. Dodgson, "Quadratic interpolation for image resampling," IEEE Trans. Image Processing, vol. 6, no. 9, pp. 1322-1326, September 1997.
[8] F. Anton, D. Mioc, and A. Fournier, "2d image reconstruction using natural neighbour interpolation," in The Eighth International Conference on Computer Graphics and Visualization in Central Europe (WSCG- 2000), Febuary 2000, pp. 263-269.
[9] X. Li, "New edge-directed interpolation," IEEE Trans. Image Processing, vol. 10, pp. 1521-1527, October 2001.
[10] K. P. Hong, J. K. Paik, H. J. Kim, and C. H. Lee, "An edgepreserving image interpolation system for a digital camcorder," IEEE Trans. Consumer Electron., vol. 42, no. 3, pp. 279-284, 1996.
[11] L. W. Leung, B. King, and V. Vohora, "Comparison of image data fusion techniques using entropy and ini," in 22nd Asian Converence of Remove Sensing. Centre for Remote Imaging, Sensing and Processing CRISP, November 2001.
[12] G. Piella, "A general framework for multiresolution image fusion: from pixels to regions," Information Fusion, pp. 259-280, 2003.
[13] A. Waxman, A. Gove, D. Fay, J. Racamato, J. Carrick, M. Seibert, and D. Savoye, "Color night vision: Opponent processing in the fusion of visible and ir imagery," in Neural Networks, vol. 10, no. 1, 1997.
[14] H. Li, B. S. Manjunath, and S. K. Mitra, "Multi-sensor image fusion using the wavelet transform." in ICIP (1), 1994, pp. 51-55.
[15] J. Nunez, X. Otazu, O. Fors, A. Prades, V. Pala, and R. Arbiol, "Multiresolution-based image fusion with additive wavelet decomposition," IEEE Trans. Geosci. Remote Sensing, vol. 37, pp. 1204-1211, 1999.
[16] A. Garzelli, "Wavelet-based fusion of optical and sar image data over urban area," in PCV02, 2002, p. B: 59.
[17] Y. Zhang and R. Wang, "Multi-resolution and multi-spectral image fusion for urban object extraction," in Proceedings of XXth ISPRS Congress, Commission III, 2004, pp. 960-966.
[18] P. S. Chavez, C. S. Stuart, and J. A. Anderson, "Comparison of three different methods to merge multiresolution and multispectral data: Landsat tm and spot panchromatic," Photogrammetric Engineering and Remote Sensing, vol. 57, pp. 295-303, 1991.
[19] C. K. Munechika, J. S. Warnick, and C. Salvaggo, "Resolution enhancement of multispectral image data to improve calssification accuracy," Photographic Engineering and Remote Sensing, vol. 59, no. 1, pp. 67- 72, Januray 1993.
[20] F. Sroubek and J. Flusser, "Image fusion via probabilistic deconvolution," in Third International Workshop on Pattern Recognition in Remote Sensing, 2004, pp. 1-5.
[21] H. Kiiveri, "Image fusion with conditional probability networks for monitoring salinisation of farmland," 1998.
[22] G. Pajares and J. Manuel de la Cruz, "A wavelet based image fusion tutorial," Pattern Recognition, vol. 37, pp. 1855-1872, 2004.
[23] R. Tsai and T. Huang, Multiframe Image Restoration and Registration, R. Tsai and T. Huang, Eds. JAI Press, 1984, vol. 1.
[24] M. Irani and S. Peleg, "Super resolution from image sequences," 10th ICPR, vol. 2, pp. 115-120, 1990.
[25] R. Schultz and R. Stevenson, "Extraction of high resolution frames from video sequences," IEEE Trans. Image Processing, vol. 5, no. 6, pp. 996- 1011, 1996.
[26] S. Kim, N. Bose, and H. Valenzuela, "Recursive reconstruction of high resolution image from noisy undersampled multiframes," IEEE Trans. Acoust., Speech, Signal Processing, vol. 38, no. 6, 1990.
[27] E. Salari and S. Zhang, "Integrated recurrent neural network for image resolution enhancement from multiple image frames," in IEE Proceedings on Vision, Image and Signal Processing, vol. 150, no. 5, 2003.
[28] M. Van Veelen, J. Nijhuis, and B. Spaanenburg, "Neural network approaches to capture temporal information," in CASYS 1999, 2000, pp. 361-371.
[29] R. Ivry and R. Spencer, "The neural representation of time," Current Opinion in Neurobiology, vol. 14, no. 2, pp. 225-232, 2004.
[30] A. Herz, "How is time represented in the brain," to Appear: Problems in Systems Neuroscience.
[31] J. Elman, "Finding structure in time," Cognitive Science, vol. 14, pp. 179-211, 1990.
[32] S. Hochreiter and J. Schmidhuber, "Long short term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[33] D. Wang, X. Liu, and S. Ahalt, "On temporal generalization of simple recurrent networks," Neural Networks, vol. 9, pp. 1099-1118, 1996.