Facemark Facial Landmark Detection. We will learn about how to find facial landmarks using OpenCV. You can use landmark detection for face morphing, face averaging and face-swapping.
Until now, we had to depend on an external library. But now you don’t need one because we have a new kind of landmark detection that does not depend on any external libraries.
Limitations
In the OpenCV library, there are some algorithms that can do landmark detection.
Python support: It seems like there is still no python support in OpenCV 3.4
One of the three algorithms for detecting landmarks did not work. This is not a big problem. If it did not work, we will try to make our own model and share it with people so they can use it too.
Because of the two limitations above, this will be updated a few times.
Facemark API
OpenCV has an API called Facemark. It has three different ways of identifying facial landmarks based on three different papers.
FacemarkKazemi: This computer program is based on a paper by V.Kazemi and J. Sullivan that was published in CVPR 2014. There is another implementation of this algorithm in DLIB, which stands for Data Library.
FacemarkAAM: This implementation uses an Active Appearance Model (AAM) and is based on a paper written by G. Tzimiropoulos and M. Pantic in 2013.
FacemarkLBF: This is a new way of doing face alignment. It uses a paper from CVPR 2014 by S. Ren.
Three implementations follow the same patterns even though one seems different.
Real-time facial landmark detection using OpenCV code
In this section, we will show how to detect faces. This post will be updated when Python bindings for the Facemark API are available.
The steps to use the Facemark API are listed in the code.
Load face detector
All facial landmark detection algorithms take a cropped facial image. The first step is to find all faces in the image and pass those rectangles to the landmark detector. It loads OpenCV’s HAAR face detector (haarcascade_frontalface_alt2.xml).
Create Facemark Instance
We use the Facemark class. It will be wrapped in OpenCV’s smart pointer, so you do not have to worry about memory management.
Load landmark detector
Next, we load a landmark detector. This landmark detector is trained on images of faces with landmarks on them. You can find public datasets of those images here.
Capture frames from webcam
The next step is to grab a picture and process it. This sets up the camera that connects to your machine.
Detect faces
The face detector finds faces in the video. It tells you where they are.
Run facial landmark detector
For every face, we get 68 landmarks. We store these points in a vector of points. This is because sometimes there can be more than one person’s face in the same picture.
Draw landmarks
Once we have the landmarks, we can draw them on the frame. The code for how to do that is in draw landmarks.hpp and is not shared here to keep things clean.
