Low Light Image Enhancement with Multi-Stage Interconnected Autoencoders Integration in Pix-to-Pix GAN

Authors: Muhammad Atif, Cang Yan

Abstract:

The enhancement of low-light images is a significant area of study aimed at enhancing the quality of captured images in challenging lighting environments. Recently, methods based on Convolutional Neural Networks (CNN) have gained prominence as they offer state-of-the-art performance. However, many approaches based on CNN rely on increasing the size and complexity of the neural network. In this study, we propose an alternative method for improving low-light images using an Autoencoders-based multiscale knowledge transfer model. Our method leverages the power of three autoencoders, where the encoders of the first two autoencoders are directly connected to the decoder of the third autoencoder. Additionally, the decoder of the first two autoencoders is connected to the encoder of the third autoencoder. This architecture enables effective knowledge transfer, allowing the third autoencoder to learn and benefit from the enhanced knowledge extracted by the first two autoencoders. We further integrate the proposed model into the Pix-to-Pix GAN framework. By integrating our proposed model as the generator in the GAN framework, we aim to produce enhanced images that not only exhibit improved visual quality but also possess a more authentic and realistic appearance. These experimental results, both qualitative and quantitative, show that our method is better than the state-of-the-art methodologies.

Keywords: Low light image enhancement, deep learning, convolutional neural network, image processing.

References:

[1] Badrinarayanan, V., A. Kendall, and R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 2017. 39(12): p. 2481-2495.
[2] He, K., et al. Mask r-cnn. in Proceedings of the IEEE international conference on computer vision. 2017.
[3] Taigman, Y., et al. Deepface: Closing the gap to human-level performance in face verification. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.
[4] Schroff, F., D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
[5] Zuiderveld, K., VIII. 5. Contrast limited adaptive histogram equalization. Graph. Gems 474–485 (1994).
[6] Lee, J.-T., et al. Depth-guided adaptive contrast enhancement using 2D histograms. in 2014 IEEE International Conference on Image Processing (ICIP). 2014. IEEE.
[7] Land, E.H., The retinex theory of color vision. Scientific American, 1977. 237(6): p. 108-129.
[8] Fu, X., et al. A weighted variational model for simultaneous reflectance and illumination estimation. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[9] Lore, K.G., A. Akintayo, and S. Sarkar, LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, 2017. 61: p. 650-662.
[10] Wei, C., et al., Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560, 2018.
[11] Lv, F., et al. MBLLEN: Low-Light Image/Video Enhancement Using CNNs. in BMVC. 2018.
[12] Li, Z., Y. Wang, and J. Zhang, Low-light image enhancement with knowledge distillation. Neurocomputing, 2023. 518: p. 332-343.
[13] Fan, S., et al., LACN: A lightweight attention-guided ConvNeXt network for low-light image enhancement. Engineering Applications of Artificial Intelligence, 2023. 117: p. 105632.
[14] Zhao, L., et al., Learning deep texture-structure decomposition for low-light image restoration and enhancement. Neurocomputing, 2023. 524: p. 126-141.
[15] Jiang, Y., et al., Enlightengan: Deep light enhancement without paired supervision. IEEE transactions on image processing, 2021. 30: p. 2340-2349.
[16] Baldi, P., Autoencoders, Unsupervised Learning, and Deep Architectures, in Proceedings of ICML Workshop on Unsupervised and Transfer Learning, G. Isabelle, et al., Editors. 2012, PMLR: Proceedings of Machine Learning Research. p. 37--49.
[17] Park, S., et al., Dual autoencoder network for retinex-based low-light image enhancement. IEEE Access, 2018. 6: p. 22084-22093.
[18] Goodfellow, I., et al., Generative adversarial nets. Advances in neural information processing systems, 2014. 27.
[19] Fu, Y., et al., LE-GAN: Unsupervised low-light image enhancement network using attention module and identity invariant loss. Knowledge-Based Systems, 2022. 240: p. 108010.
[20] Stark, J.A., Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Transactions on image processing, 2000. 9(5): p. 889-896.
[21] Lee, C., C. Lee, and C.-S. Kim, Contrast enhancement based on layered difference representation of 2D histograms. IEEE transactions on image processing, 2013. 22(12): p. 5372-5384.
[22] Coltuc, D., P. Bolon, and J.-M. Chassery, Exact histogram specification. IEEE Transactions on Image processing, 2006. 15(5): p. 1143-1152.
[23] Jobson, D.J., Z.-u. Rahman, and G.A. Woodell, Properties and performance of a center/surround retinex. IEEE transactions on image processing, 1997. 6(3): p. 451-462.
[24] Jobson, D.J., Z.-u. Rahman, and G.A. Woodell, A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image processing, 1997. 6(7): p. 965-976.
[25] Guo, X., Y. Li, and H. Ling, LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on image processing, 2016. 26(2): p. 982-993.
[26] Guo, X. LIME: A method for low-light image enhancement. in Proceedings of the 24th ACM international conference on Multimedia. 2016.
[27] Wang, S., et al., Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE transactions on image processing, 2013. 22(9): p. 3538-3548.
[28] Li, M., et al., Structure-revealing low-light image enhancement via robust retinex model. IEEE Transactions on Image Processing, 2018. 27(6): p. 2828-2841.
[29] Fu, X., et al., A fusion-based enhancing method for weakly illuminated images. Signal Processing, 2016. 129: p. 82-96.
[30] Wang, W., et al. Gladnet: Low-light enhancement network with global awareness. in 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). 2018. IEEE.
[31] Wang, R., et al. Underexposed photo enhancement using deep illumination estimation. in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[32] Zhang, Y., J. Zhang, and X. Guo. Kindling the darkness: A practical low-light image enhancer. in Proceedings of the 27th ACM international conference on multimedia. 2019.
[33] Wang, L.-W., et al., Lightening network for low-light image enhancement. IEEE Transactions on Image Processing, 2020. 29: p. 7984-7996.
[34] Zhang, F., et al. Learning temporal consistency for low light video enhancement from single images. in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
[35] Liu, R., et al. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[36] Jiang, Z., et al., A switched view of Retinex: Deep self-regularized low-light image enhancement. Neurocomputing, 2021. 454: p. 361-372.
[37] Guo, C., et al. Zero-reference deep curve estimation for low-light image enhancement. in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
[38] Yang, W., et al. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
[39] Yang, Q., et al., A lowlight image enhancement method learning from both paired and unpaired data by adversarial training. Neurocomputing, 2021. 433: p. 83-95.
[40] Liu, Y., et al. PD-GAN: perceptual-details gan for extremely noisy low light image enhancement. in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021. IEEE.
[41] Zhang, J., et al., DEGAN: Decompose-Enhance-GAN Network for Simultaneous Low-Light Image Lightening and Denoising. Electronics, 2023. 12(14): p. 3038.
[42] Isola, P., et al. Image-to-image translation with conditional adversarial networks. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[43] Cai, J., S. Gu, and L. Zhang, Learning a deep single image contrast enhancer from multi-exposure images. IEEE Transactions on Image Processing, 2018. 27(4): p. 2049-2062.
[44] Singh, K. and A.S. Parihar, DSE-Net: Deep simultaneous estimation network for low-light image enhancement. Journal of Visual Communication and Image Representation, 2023. 91: p. 103780.
[45] Deng, J., et al. Low-light Image Enhancement based on Joint Decomposition and Denoising U-Net Network. in 2020 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom). 2020. IEEE.
[46] C. Li et al., "Low-Light Image and Video Enhancement Using Deep Learning: A Survey," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 9396-9416, 1 Dec. 2022, doi: 10.1109/TPAMI.2021.3126387.