Face recognition is a classic topic in past decades and now still attracts much attention in the field of computer vision and pattern recognition. Face recognition has a great potential in multimedia applications, e.g. video surveillance, personal identification, digital entertainment and so on [1, 2, 3, 4, 5]. With the rapid development of electric equipment techniques, more and more face images can be easily captured in the wild, especially videos from cameras of surveillance or cell phones. Therefore, video or image set based face recognition becomes more important in most of real-world applications and also becomes a popular topic in face analysis more recently. As face images captured from the unconstrained conditions are usually with complex appearance variations in poses, expressions, illuminations, etc., the existing face recognition algorithms still suffer from a severe challenge in fulfilling real applications to large-scale data scenes, although the current deep learning techniques have made a great progress on the unconstrained small face dataset, e.g., the recent success of deep learning methods on Labeled Faces in the Wild (LFW) [6]. In the task of face recognition, however, we cannot bypass this question of pose variations, which has been extensively studied and explored in past decades, and has not been well-solved yet. The involved methods may be divided into 20 3D [7, 8, 9] and 2D methods [10, 11, 12, 13, 14, 15, 16]. Since pose variations are basically caused by 3D rigid motions of face, 3D methods are more intuitive for pose generation. But 3D methods usually need some 3D data or recovery of 3D model from 2D data which is not a trivial thing. Moreover, the inverse transform from 3D model to 2D space is sensitive to facial appearance varia tions. In contrast to 3D model, due to decreasing one degree of freedom, 2D methods usually attempt to learn some transforms across poses, including linear models [17] or non-linear models [10, 18]. Because of its simplicity, 2D model has been widely used to deal with cross-pose face recognition with a comparable performance with 3D model. However, in many real scenes of face image sets, e.g., face video sequences, the changes of poses may be regarded as a nearly continuous stream of motions, while the existing methods usually neglect or do not make full use of this prior. Moreover, the pose variation is not the only factor between different images even for the same subject, which involves other complex factors.