Face AR Capture Overview
At a high level, the facial capture system with ARKit uses the Apple TrueDepth camera to track the motion of a user’s face. In the process, it compares the pose of the face against 51 individual face poses. These poses are native to the Apple ARKit SDK, and each pose targets a specific portion of the face, such as the left eye, right eye, sides of the mouth, etc. As a given part of the user’s face approaches the shape of a pose, the value of that pose blends between 0.0 and 1.0. For example, if the user closes their left eye, the LeftEyeBlink pose would blend from 0.0 to 1.0.
This means that, as a user’s face naturally moves, all 51 poses are being evaluated by the SDK and assigned a value. The Unreal Engine ARKit integration captures the incoming values from the 51 blended face poses, feeding those into the Engine via the Live Link plugin. Those 51 pose values can then drive the motion of a real-time character’s face.
This means that all you really need to utilize face capture to animate a character’s head is to ensure the character content is set up to use data from those 51 shapes. Because those shapes each feed back individual 0.0.to 1.0 values, they are perfect for driving the motion of a list of blend shapes on a character.
If the blend shapes created on the Unreal character are named precisely the same as those in the official list of shapes from Apple, then the connection is automatic. However, if the shape names differ between the Apple mesh and the Unreal character, then a remapping Asset must be used. This process is explained in further detail below.
For a complete list of the blend shapes brought in by Apple's ARKit, please refer to Apple’s official documentation: ARFaceAnchor.BlendShapeLocation