Depth extraction is a key step in 3D video system based on texture plus depth. Real application systems require high efficiency and fast depth extraction. This paper presents a fast inpainting algorithm for Kinect depth map. This paper clips and aligns the origin color and depth images captured by Kinect, and partially fill holes in the clipped depth image using a background based method. Then this paper uses a fast inpainting algorithm based on color match to fill the remaining holes in the depth image to obtain a better depth image. Experimental results show that the proposed algorithm can efficiently repair errors in the original depth map such as holes caused by occlusion. Finally we obtain a smooth depth map with clear edges. The processing speed can reach 25~30 frame/s by using an ordinary desktop computer to realize real-time depth map extraction.