3D human face reconstruction and expression modelling

Abstract

This thesis develops a unique set of computer vision tools for 3D face reconstruction and expression modelling aimed at being low cost and self-contained. The first part involves 3D face reconstruction and begins with an extensive comparison of three popular static 3D face reconstruction approaches: binocular stereo, photometric stereo and structured lighting. A database of 15 test subjects with scanner generated ground truth was created. Results showed that structured lighting gave the most accurate results and binocular stereo algorithms, when aided by active illumination, improve to a consistent level regardless of their algorithmic complexity. The SDPS stereo algorithm coupled with a high spatial frequency illumination pattern was found to be a good choice for fast and accurate 3D face reconstruction. Based on these results a dynamic 3D face reconstruction system using video cameras was implemented. It creates 3D video of faces at up to 15 fps and the dynamic depth and texture of a face provides realistic 3D reconstruction and motion. The second part of the thesis involves expression modelling. Firstly a stereo system is presented that tracks coloured face markers to drive a muscle based 3D face model. Markers are mapped to the model by a radial basis function (RBF) approach and their motion is combined with an inverse kinematics framework to drive facial animation. Results show that the system can reproduce expressions based on a minimal set of markers. However, important visual cues are lost due to the model's static face texture. Additionally a body animation system for attachment to face results is presented with a method to create personalised results from a few images using an RBF approach. Animation is achieved through a skinned approach, where a skeletal rig smoothly controls a body model. In the concluding chapter a 3D video expression creation system is implemented, tying together the reconstruction and expression modelling elements of this thesis. Annotated dynamic 3D reconstructions are recombined to create novel expressions not contained in the original dataset. Results give realistic expressions and animation as they are derived from 3D video.