The Impact of Temporal Impairment on Quality of Experience (QoE) in Video Streaming: A No Reference (NR) Subjective and Objective Study

Authors: Muhammad Arslan Usman, Muhammad Rehan Usman, Soo Young Shin

Abstract:

Live video streaming is one of the most widely used service among end users, yet it is a big challenge for the network operators in terms of quality. The only way to provide excellent Quality of Experience (QoE) to the end users is continuous monitoring of live video streaming. For this purpose, there are several objective algorithms available that monitor the quality of the video in a live stream. Subjective tests play a very important role in fine tuning the results of objective algorithms. As human perception is considered to be the most reliable source for assessing the quality of a video stream subjective tests are conducted in order to develop more reliable objective algorithms. Temporal impairments in a live video stream can have a negative impact on the end users. In this paper we have conducted subjective evaluation tests on a set of video sequences containing temporal impairment known as frame freezing. Frame Freezing is considered as a transmission error as well as a hardware error which can result in loss of video frames on the reception side of a transmission system. In our subjective tests, we have performed tests on videos that contain a single freezing event and also for videos that contain multiple freezing events. We have recorded our subjective test results for all the videos in order to give a comparison on the available No Reference (NR) objective algorithms. Finally, we have shown the performance of no reference algorithms used for objective evaluation of videos and suggested the algorithm that works better. The outcome of this study shows the importance of QoE and its effect on human perception. The results for the subjective evaluation can serve the purpose for validating objective algorithms.

Keywords: Objective evaluation, subjective evaluation, quality of experience (QoE), video quality assessment (VQA).

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1107992

References:

[1] Forecast, Cisco VNI. "Cisco Visual Networking Index: Global Mobile data Traffic Forecast Update 2009-2014." Cisco Public Information, February 9 (2010).
[2] Video Quality Experts Group. "Final report from the Video Quality Experts Group on the validation of objective models of video quality assessment, Phase II(FR_TV2).", 2003.
[3] Cisco, I. "Cisco visual networking index: Forecast and methodology, 2011–2016." CISCO White Paper (2012): 2011-2016.
[4] Usman, Muhammad Arslan, Muhammad Rehan Usman, and Soo Young Shin. "Performance Analysis of a No-Reference Temporal Quality Assessment Metric for Videos Impaired by Frame Freezing Artefacts."
[5] “Subjective video quality assessment methods for multimedia applications,” September 1999, iTU-T, Recommendation ITU-R P910.
[6] “ITU-R Radio communication Sector of ITU, Recommendation ITU-R BT.500-12,” 2009.
[7] Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity." Image Processing, IEEE Transactions on 13.4 (2004): 600-612.
[8] Stephen Wolf, “A no reference (nr) and reduced reference (rr) metric for detecting dropped video frames,” in Second International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM), 2009.
[9] Quan Huynh-Thu and M. Ghanbari, “No-reference temporal quality metric for video impaired by frame freezing artefacts,” in IEEE International Conference on Image Processing (ICIP), 2009, pp. 2221– 2224.
[10] Borer, Silvio. "A model of jerkiness for temporal impairments in video transmission." Quality of Multimedia Experience (QoMEX), 2010 Second International Workshop on. IEEE, 2010.
[11] Shahid, Muhammad, et al. "Subjective quality assessment of H. 264/AVC encoded low resolution videos." Image and Signal Processing (CISP), 2012 5th International Congress on. IEEE, 2012.
[12] Nightingale, James, et al. "The impact of network impairment on quality of experience (QoE) in H. 265/HEVC video streaming." Consumer Electronics, IEEE Transactions on 60.2 (2014): 242-250.
[13] Ou, Yen-Fu, et al. "Modeling the impact of frame rate on perceptual quality of video." City 70.80 (2008): 90.
[14] S. van Kester, T. Xiao, R. E. Kooij, K. Brunnstr¨om, and O. K. Ahmed, “Estimating the impact of single and multiple freezes on video quality,” 2011, vol. 7865, pp. 78650O–78650O–10.
[15] Ponomarenko, N., et al. "Color image database for evaluation of image quality metrics." Multimedia Signal Processing, 2008 IEEE 10th Workshop on. IEEE, 2008.