Video Matting based on Background Estimation

Authors: J.-H. Moon, D.-O Kim, R.-H. Park

Abstract:

This paper presents a video matting method, which extracts the foreground and alpha matte from a video sequence. The objective of video matting is finding the foreground and compositing it with the background that is different from the one in the original image. By finding the motion vectors (MVs) using a sliced block matching algorithm (SBMA), we can extract moving regions from the video sequence under the assumption that the foreground is moving and the background is stationary. In practice, foreground areas are not moving through all frames in an image sequence, thus we accumulate moving regions through the image sequence. The boundaries of moving regions are found by Canny edge detector and the foreground region is separated in each frame of the sequence. Remaining regions are defined as background regions. Extracted backgrounds in each frame are combined and reframed as an integrated single background. Based on the estimated background, we compute the frame difference (FD) of each frame. Regions with the FD larger than the threshold are defined as foreground regions, boundaries of foreground regions are defined as unknown regions and the rest of regions are defined as backgrounds. Segmentation information that classifies an image into foreground, background, and unknown regions is called a trimap. Matting process can extract an alpha matte in the unknown region using pixel information in foreground and background regions, and estimate the values of foreground and background pixels in unknown regions. The proposed video matting approach is adaptive and convenient to extract a foreground automatically and to composite a foreground with a background that is different from the original background.

Keywords: Background estimation, Object segmentation, Blockmatching algorithm, Video matting.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1330631

References:

[1] E. N. Mortensen and ''Intelligent scissor for image Composition,'' in Proc.SIGGRAPH 1995, pp.191-198 , Los Angeles, CA, USA, Aug. 1995.
[2]A.R. Smith and J. F. Blinn, ''Blue screen matting,'' in Proc. SIGGRAPH 1996, pp. 259-268, New Orleans, LA, USA, Aug. 1996.
[3] Y.-Y.Chuang, B. Curless, D. Salesin, and R. Szeliski, '' A Bayesian Approach to digital matting,'' in Proc. Computer Vision and Pattern Recongnition . vol. 2, pp. 264-271, Kauai, Hawaii, USA, Dec. 2001.
[4]Y.-Y. Chuang, A. Agarwala, B. Curless, D. Salesin, and R. Sze "Video matting of complex scenes," in Proc. SIGGRAPH 2002, pp. 243-248, San Antonio, TX, USA, July 2002.
[5] T. Meier and K. N. Ngan, "Video segmentation for coding," IEEE Trans. Circuits and Systems for Video Tech., vol. 9, pp. 1190-1203, Dec. 1999.
[6] E. D. Sciascio and C. Guarangnella, ''Object oriented motion estimation by sliced-block matching algorithm," in Proc. Int. Conf. Pattern Recognition, Barcelona, Spain, vol. 3, pp. 3865-3868, Sep. 2000.
[7] T. Porter and T. Duff, "Compositing digital images," in Proc. SIGGRAPH 1984, vol. 18, no. 3, pp. 253-259, Minneapolis, MN, USA, July 1984.