Face photo-sketch recognition using deeply-learned and engineered features

Abstract

Face sketches created from eyewitness descriptions of criminals have proven to be useful in assisting law enforcement agencies to apprehend perpetrators, particularly in cases lacking evidence. These sketches are typically disseminated to the public and law enforcement officers so that any persons recognising the suspect in the sketch may come forward with information leading to an arrest. However, this process is time consuming and not guaranteed to be successful. In this dissertation, an investigation of popular and state-of-the-art face photo-sketch synthesis and recognition methods which can identify perpetrators automatically is performed using an evaluation set-up that reflects real-world scenarios, through the use of challenging sketches and an extended gallery which simulates the extensive mugshot galleries maintained by law enforcement agencies. The University of Malta Software-Generated Face Sketch (UoM-SGFS) database was also created to enable the design and evaluation of algorithms when using software-generated sketches, that are nowadays being used more often than hand-drawn sketches. This database is the largest software-generated face sketch database, one of the few containing multiple sketches per subject, and the only one containing sketches represented in colour. Several novel methods have also been designed and evaluated, namely: (i) the Eigenpatches (EP) approach which improves upon the performance of the popular Eigentransformation (ET) method by transforming photos into sketches or sketches into photos on a local level, (ii) the log-Gabor-MLBP-SROCC (LGMS) method that extracts modality-invariant features, (iii) the DEEP (face) Photo- Sketch System (DEEPS) framework that applies transfer learning to a state-of-the- art face recognition system based on a Deep Convolutional Neural Network (DCNN) with the aid of an extensive set of synthetic images created using a 3D morphable model, (iv) the use of multiple synthetic sketches during system deployment, and (v) the fusion of intra- and inter-modality methods which are shown to be capable of providing complementary information. The finalised system fuses LGMS with DEEPS to yield a system outperforming state-of-the-art methods for all types of sketches, including real-world forensic sketches. Moreover, the proposed approach is efficient in terms of both computation time and template size, thereby permitting its implementation in the real-world.