Frame Camera and Event Camera in Stereo Pair for High-Resolution Sensing

Authors: Khen Cohen, Daniel Yankelevich, David Mendlovic, Dan Raviv

Abstract:

We present a 3D stereo system for high-resolution sensing in both the spatial and the temporal domains by combining a frame-based camera and an event-based camera. We establish a method to merge both devices into one unite system and introduce a calibration process, followed by a correspondence technique and interpolation algorithm for 3D reconstruction. We further provide quantitative analysis about our system in terms of depth resolution and additional parameter analysis. We show experimentally how our system performs temporal super-resolution up to effectively 1ms and can detect fast-moving objects and human micro-movements that can be used for micro-expression analysis. We also demonstrate how our method can extract colored events for an event-based camera without any degradation in the spatial resolution, compared to a colored filter array.

Keywords: DVS-CIS stereo vision, micro-movements, temporal super-resolution, 3D reconstruction