Evaluation of Video Quality Metrics and Performance Comparison on Contents Taken from Most Commonly Used Devices

Authors: Pratik Dhabal Deo, Manoj P.

Abstract:

With the increasing number of social media users, the amount of video content available has also significantly increased. Currently, the number of smartphone users is at its peak, and many are increasingly using their smartphones as their main photography and recording devices. There have been a lot of developments in the field of video quality assessment in since the past years and more research on various other aspects of video and image are being done. Datasets that contain a huge number of videos from different high-end devices make it difficult to analyze the performance of the metrics on the content from most used devices even if they contain contents taken in poor lighting conditions using lower-end devices. These devices face a lot of distortions due to various factors since the spectrum of contents recorded on these devices is huge. In this paper, we have presented an analysis of the objective Video Quality Analysis (VQA) metrics on contents taken only from most used devices and their performance on them, focusing on full-reference metrics. To carry out this research, we created a custom dataset containing a total of 90 videos that have been taken from three most commonly used devices, and Android smartphone, an iOS smartphone and a Digital Single-Lens Reflex (DSLR) camera. On the videos taken on each of these devices, the six most common types of distortions that users face have been applied in addition to already existing H.264 compression based on four reference videos. These six applied distortions have three levels of degradation each. A total of the five most popular VQA metrics have been evaluated on this dataset and the highest values and the lowest values of each of the metrics on the distortions have been recorded. Finally, it is found that blur is the artifact on which most of the metrics did not perform well. Thus, in order to understand the results better the amount of blur in the data set has been calculated and an additional evaluation of the metrics was done using High Efficiency Video Coding (HEVC) codec, which is the next version of H.264 compression, on the camera that proved to be the sharpest among the devices. The results have shown that as the resolution increases, the performance of the metrics tends to become more accurate and the best performing metric among them is VQM with very few inconsistencies and inaccurate results when the compression applied is H.264, but when the compression is applied is HEVC, Structural Similarity (SSIM) metric and Video Multimethod Assessment Fusion (VMAF) have performed significantly better.

Keywords: Distortion, metrics, recording, frame rate, video quality assessment.

References:

[1] K. Seshadrinathan, R. Soundararajan, A. C. Bovik and L. K. Cormack, "Study of Subjective and Objective Quality Assessment of Video", IEEE Transactions on Image Processing, vol.19, no.6, pp.1427-1441, June 2010. W.-K. Chen, Linear Networks and Systems (Book style). Belmont, CA: Wadsworth, 1993, pp. 123–135.
[2] Vlad Hosu; Franz Hahn, Mohsen Jenadeleh; Hanhe Lin, Hui Men, Tamás Szirányi, Shujun Li “The Konstanz natural video database (KoNViD-1k)” IEEE, 3 July 2017
[3] Miguel O. Martínez-Rach,Pablo Piñol, Otoniel M. López, Manuel Perez Malumbres, José Oliver,and Carlos Tavares Calafate 2014
[4] Shahid, M., Rossholm, A., Lövström, B. et al. No-reference image and video quality assessment: a classification and review of recent approaches. J Image Video Proc 2014, 40 (2014).
[5] Sara, U., Akter, M. and Uddin, M. (2019) Image Quality Assessment through FSIM, SSIM, MSE and PSNR—A Comparative Study. Journal of Computer and Communications, 7, 8-18. doi: 10.4236/jcc.2019.73002.
[6] Mariela Fiorenzo, Claudio Righetti, María Cecilia Raggio, Fernando Ochoa & Gabriel Carro, Telecom Argentina S.A, 2019 SCTE.ISBE
[7] Read, Dwight. (2013). Artifact Classification: A Conceptual and Methodological Approach.
[8] Jnastasia Antsiferova, Dmitriy Vatolin, Dmitriy Kulikov, Sergey Zvezdakov “Hacking VMAF with Video Color and Contrast Distortion” july 2019
[9] We are social 2020, We are social, accessed 18 October,
[10] Vox 2016, Vox, accessed 20 October,
[11] Streaming media 2019, streaming media, accessed 20 October,