NDENet: End-to-End Nighttime Dehazing and Enhancement

Authors: H. Baskar, A. S. Chakravarthy, P. Garg, D. Goel, A. S. Raj, K. Kumar, Lakshya, R. Parvatham, V. Sushant, B. Kumar Rout

Abstract:

In this paper, we present a computer vision task called nighttime dehaze-enhancement. This task aims to jointly perform dehazing and lightness enhancement. Our task fundamentally differs from nighttime dehazing – our goal is to jointly dehaze and enhance scenes, while nighttime dehazing aims to dehaze scenes under a nighttime setting. In order to facilitate further research on this task, we release a benchmark dataset called Reside-β Night dataset, consisting of 4122 nighttime hazed images from 2061 scenes and 2061 ground truth images. Moreover, we also propose a network called NDENet (Nighttime Dehaze-Enhancement Network), which jointly performs dehazing and low-light enhancement in an end-to-end manner. We evaluate our method on the proposed benchmark and achieve Structural Index Similarity (SSIM) of 0.8962 and Peak Signal to Noise Ratio (PSNR) of 26.25. We also compare our network with other baseline networks on our benchmark to demonstrate the effectiveness of our approach. We believe that nighttime dehaze-enhancement is an essential task particularly for autonomous navigation applications, and hope that our work will open up new frontiers in research. The code for our network is made publicly available.

Keywords: Dehazing, image enhancement, nighttime, computer vision.

References:

[1] K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, 08 2010.
[2] J. Zhang, Y. Cao, S. Fang, Y. Kang, and C. W. Chen, “Fast haze removal for nighttime image using maximum reflectance prior,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[3] Q. Tang, J. Yang, X. He, W. Jia, Q. Zhang, and H. Liu, “Nighttime image dehazing based on retinex and dark channel prior using taylor series expansion,” Computer Vision and Image Understanding, vol. 202, p. 103086, 2021.
[4] S. G. Narasimhan and S. K. Nayar, “Vision and the atmosphere,” International Journal of Computer Vision, vol. 48, no. 3, pp. 233–254, Jul 2002. (Online). Available: https://doi.org/10.1023/A:1016328200723
[5] K. Tang, J. Yang, and J. Wang, “Investigating haze-relevant features in a learning framework for image dehazing,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2995–3002.
[6] W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in Computer Vision – ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 154–169.
[7] H. Zhang, V. Sindagi, and V. M. Patel, “Joint transmission map estimation and dehazing using deep networks,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 7, pp. 1975–1986, 2020.
[8] B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “An all-in-one network for dehazing and beyond,” arXiv preprint arXiv:1707.06543, 2017.
[9] Q. Zhu, J. Mai, and L. Shao, “A fast single image haze removal algorithm using color attenuation prior,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3522–3533, 2015.
[10] D. Berman, S. Avidan et al., “Non-local image dehazing,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1674–1682.
[11] J. Li, H. Zhang, D. Yuan, and M. Sun, “Single image dehazing using the change of detail prior,” Neurocomputing, vol. 156, pp. 1–11, 2015. (Online). Available: https://www.sciencedirect.com/science/article/pii/S0925231215000478
[12] T. M. Bui and W. Kim, “Single image dehazing using color ellipsoid prior,” IEEE Transactions on Image Processing, vol. 27, no. 2, pp. 999– 1009, 2018.
[13] B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “Dehazenet: An end-to-end system for single image haze removal,” IEEE Transactions on Image Processing, vol. 25, no. 11, pp. 5187–5198, 2016.
[14] H. Zhang and V. M. Patel, “Densely connected pyramid dehazing network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3194–3203.
[15] D. Engin, A. Genc, and H. K. Ekenel, “Cycle-dehaze: Enhanced cyclegan for single image dehazing,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018, pp. 938–9388.
[16] X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia, “Ffa-net: Feature fusion attention network for single image dehazing.” in AAAI, 2020, pp. 11908– 11915.
[17] A. Singh, A. Bhave, and D. K. Prasad, “Single image dehazing for a variety of haze scenarios using back projected pyramid network,” in European Conference on Computer Vision. Springer, 2020, pp. 166– 181.
[18] S.-C. Pei and T.-Y. Lee, “Nighttime haze removal using color transfer pre-processing and dark channel prior,” in 2012 19th IEEE International Conference on Image Processing. IEEE, 2012, pp. 957–960.
[19] Y. Li, R. T. Tan, and M. S. Brown, “Nighttime haze removal with glow and multiple light colors,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 226–234.
[20] C. Ancuti, C. O. Ancuti, C. De Vleeschouwer, and A. C. Bovik, “Nighttime dehazing by fusion,” in 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 2016, pp. 2256–2260.
[21] D. Park, D. K. Han, and H. Ko, “Nighttime image dehazing with local atmospheric light and weighted entropy,” in 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 2016, pp. 2261–2265.
[22] J. Zhang, Y. Cao, S. Fang, Y. Kang, and C. Wen Chen, “Fast haze removal for nighttime image using maximum reflectance prior,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7418–7426.
[23] J. Zhang, Y. Cao, Z.-J. Zha, and D. Tao, “Nighttime dehazing with a synthetic benchmark,” in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 2355–2363.
[24] T. Yu, K. Song, P. Miao, G. Yang, H. Yang, and C. Chen, “Nighttime single image dehazing via pixel-wise alpha blending,” IEEE Access, vol. 7, pp. 114619–114630, 2019.
[25] W. Lou, Y. Li, G. Yang, C. Chen, H. Yang, and T. Yu, “Integrating haze density features for fast nighttime image dehazing,” IEEE Access, vol. 8, pp. 113318–113330, 2020.
[26] R. He, X. Guo, and Z. Shi, “Side—a unified framework for simultaneously dehazing and enhancement of nighttime hazy images,” Sensors, vol. 20, no. 18, p. 5300, 2020.
[27] M. Feng, T. Yu, M. Jing, and G. Yang, “Learning a convolutional autoencoder for nighttime image dehazing,” Information, vol. 11, no. 9, p. 424, 2020.
[28] Y.-T. Kim, “Contrast enhancement using brightness preserving bihistogram equalization,” IEEE Transactions on Consumer Electronics, vol. 43, no. 1, pp. 1–8, 1997.
[29] S.-D. Chen and A. Ramli, “Minimum mean brightness error bi-histogram equalization in contrast enhancement,” IEEE Transactions on Consumer Electronics, vol. 49, no. 4, pp. 1310–1319, 2003.
[30] H. Ibrahim and N. S. Pik Kong, “Brightness preserving dynamic histogram equalization for image contrast enhancement,” IEEE Transactions on Consumer Electronics, vol. 53, no. 4, pp. 1752–1758, 2007.
[31] K. Singh, R. Kapoor, and S. K. Sinha, “Enhancement of low exposure images via recursive histogram equalization algorithms,” Optik, vol. 126, no. 20, pp. 2619–2625, 2015. (Online). Available: https://www.sciencedirect.com/science/article/pii/S003040261500532X
[32] E. H. Land, “The retinex theory of color vision,” Scientific american, vol. 237, no. 6, pp. 108–129, 1977.
[33] S. Park, S. Yu, B. Moon, S. Ko, and J. Paik, “Low-light image enhancement using variational optimization-based retinex model,” IEEE Transactions on Consumer Electronics, vol. 63, no. 2, pp. 178–184, 2017.
[34] M. Li, J. Liu, W. Yang, X. Sun, and Z. Guo, “Structure-revealing lowlight image enhancement via robust retinex model,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2828–2841, 2018.
[35] Y. Zhang, J. Zhang, and X. Guo, “Kindling the darkness: A practical low-light image enhancer,” in Proceedings of the 27th ACM International Conference on Multimedia, ser. MM ’19. New York, NY, USA: ACM, 2019, pp. 1632–1640. (Online). Available: http://doi.acm.org/10.1145/3343031.3350926
[36] C. Wei, W. Wang, W. Yang, and J. Liu, “Deep retinex decomposition for low-light enhancement,” in British Machine Vision Conference, 2018.
[37] Y. Zhang, X. Guo, J. Ma, W. Liu, and J. Zhang, “Beyond brightening low-light images,” International Journal of Computer Vision, vol. 129, no. 4, pp. 1013–1037, Apr 2021. (Online). Available: https://doi.org/10.1007/s11263-020-01407-x
[38] C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong, “Zeroreference deep curve estimation for low-light image enhancement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1780–1789.
[39] B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking single-image dehazing and beyond,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 492–505, 2019.
[40] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
[41] X. Guo, Y. Li, and H. Ling, “Lime: Low-light image enhancement via illumination map estimation,” IEEE Transactions on image processing, vol. 26, no. 2, pp. 982–993, 2016.
[42] S. H. Chan, R. Khoshabeh, K. B. Gibson, P. E. Gill, and T. Q. Nguyen, “An augmented lagrangian method for total variation video restoration,” IEEE Transactions on Image Processing, vol. 20, no. 11, pp. 3097–3111, 2011.
[43] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
[44] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[45] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in European conference on computer vision. Springer, 2016, pp. 694–711.
[46] S. Santra and B. Chanda, “Day/night unconstrained image dehazing,” in 2016 23rd International Conference on Pattern Recognition (ICPR), 2016, pp. 1406–1411.