Introduction to people tracking in Spark.pdf

Introduction to people tracking The effects that you can create in Meta Spark Studio fall into 2 broad categories: People effects — People effects are effects that respond to someone’s movement. Capabilities typically used in people effects include face, hand and body trackers, pe...

Introduction to people tracking The effects that you can create in Meta Spark Studio fall into 2 broad categories: People effects — People effects are effects that respond to someone’s movement. Capabilities typically used in people effects include face, hand and body trackers, person segmentation, and deformation. World effects — World effects add virtual objects into real world environments. Unlike people effects, world effects usually use the back facing camera. You make world effects using the plane tracker or target tracker. The Face tracker You can use the face tracker to create effects that respond to the position and orientation of someone’s face. You can combine a face tracker with a face mesh to create a surface that detects facial movements and expressions: the face mesh The face mesh is a 3D model of a face. It works in combination with the face tracker in Meta Spark Studio to create a surface that reconstructs someone's expressions. Once you've added a face tracker and face mesh to your project you can create face tracking effects, add retouching or change the shape of the face. Adding a face mesh To add a face mesh: 1. Click + at the bottom of the Scene panel. 2. Select Face Mesh. 3. Click Insert Note: If you add a face mesh without a face tracker, a face tracker will be added automatically. Applying a material to the face mesh One of the simplest effects you can make in Meta Spark Studio is a mask effect that places a texture over someone's face. All you need to do is apply a material to the face mesh. To do this: 1. Select the face mesh in the Scene panel. 2. In the Inspector, click + to the right of Materials.  This will create a new material and attach it to your face mesh. You’ll see the checkerboard pattern on the face mesh has changed to a default, standard material. To build the appearance of the mask, add custom textures and color to the material. Changing the shape of the face mesh Use a 3D object called a blend shape to adjust the shape of the face mesh. Either make your own using a 3D modeling program, such as Blender. 1. With the face mesh selected, click + next to Deformation in the Inspector. 2. Select the blend shape you want to use from your computer. Under deformation a range of sliders will appear, depending on what's included in the blend shape. Adjust the sliders until you're happy with the effect: Fading the edges of the face mesh Meta Spark Studio provides a range of face reference assets you can use to make face effects. One of these is the face mesh mask. This is a texture you can use to fade the edges of the face mesh, so it blends seamlessly with the rest of the head. In the image below, we've used the face mesh mask in the effect on the right: Face mesh - properties Visible - Clear this box to stop the face mesh from rendering in the scene. Tracked Face - You can have more than one face tracker in the scene. Choose which face tracker will drive the face mesh. Eyes - Choose whether you want the face mesh to have an opening for someone's eyes, or not. Mouth - Choose whether you want the face mesh to have an opening for a mouth, or not. Transformations - Edit the position, scale and rotation of the face mesh. Materials - Choose the material you want to apply to the mesh, or create a new one. Deformation - Make changes to the shape of the face, using blend shapes you've created in 3D modeling software. Enable For - Choose the camera or cameras on a mobile device in which you want to render the face mesh. Morph Object - If you want to deform the shape of the face, you'd apply the blend shape here. Use the slider to change the appearance of the face, if you've applied a blend shape to your face mesh. Creating patches to represent the face mesh To create a patch to represent the face mesh, all you need to do is select the face mesh in the Scene panel and drag it into the Patch Editor. Connect this patch to the Object Tap patch, to create a boolean signal each time the face mesh in the scene is tapped. Optimization Avoid including more than 5 face meshes in a project. Including too many will impact the performance of your effect. Tracking more than one face  You can add more than one face tracker to a Meta Spark Studio project, to create an effect that can be used by more than one person. Adding a second face tracker The first face tracker you add will track the first face that appears in the scene. You'll need to assign a face to every other face tracker you add. To do this: 1. Select the face tracker in the Scene panel. 2. In the Inspector, change Tracked face. For example if you've added a second face tracker, you'd change the Tracked face to Face 2. Adding more face trackers Each time you add a face tracker, an additional Face will be added to the Tracked face dropdown list. Assign each new face tracker to a different Face. Adding more face trackers will have an impact on performance, so it's good idea test your effect as you build it using the Meta Spark Player app. Moving objects with the face tracker You can use the Patch Editor to bind the movement of a scene object to a face tracker, so the object moves with the face. Face reference assets for Meta Spark Studio The face reference assets are a collection of textures and 3D objects. Use them when you're creating face effects in Meta Spark Studio. Download the face reference assets directly using this link , or click the pink button labeled Face Reference Assets on the right-hand side of this page. The downloaded folder contains all of the assets mentioned in this article: faceMesh.fbx, faceMesh.dae and faceMesh.obj headOccluder.fbx, headOccluder.dae and headOccluder.obj faceMesh.png and faceMeshTrackers.png FaceMeshMask.png faceFeminine.jpg and faceMasculine.jpg What's included with the face reference assets When you open the downloaded files on your computer you'll see two folders: Mesh and Textures. The Mesh folder contains 3D meshes and head occluders in different file formats. The Texture folder contains 2D texture files. Mesh Note: You'll need to open these objects in 3D software or in Meta Spark Studio directly to view them. This folder contains 2 types of 3D objects in different file formats: faceMesh.fbx, faceMesh.dae and faceMesh.obj — the face mesh used in our facial tracking. Use it as a scale reference when creating objects and textures. headOccluder.fbx, headOccluder.dae and headOccluder.obj — use this object to occlude the volume of the head. Textures This folder contains 3 types of 2D textures in JPG and PNG format: faceMesh.png and faceMeshTrackers.png — these textures show the topology and points on the face tracked by Meta Spark Studio. Use them for debugging effects. FaceMeshMask.png — use this texture to blend the edges of a material with the face. faceFeminine.jpg and faceMasculine.jpg — these are textures illustrated to look like real faces. Use them as templates when creating custom textures for face effects, to give you an idea of the position of the features of the person using your effect. faceMesh.fbx, faceMesh.dae and faceMesh.obj This is the face mesh Meta Spark Studio uses to to define someone's face, and track it:  Use this mesh to help you create assets in external software. We recommend using FaceMesh.fbx for maximum compatibility. As a scale reference Use the face mesh as a scale reference, for example when you're creating objects like glasses or hats that you'll track to the face. Using this mesh as a reference will mean objects are the right size when you import them into Meta Spark Studio. Create facial distortion blendshapes Use 3D software to move the points in this object, to distort the shape of the mesh. You can then import it into Meta Spark Studio, and use it as a blendshape to change the shape of someone's face when they're using the effect. Create custom mask and makeup effects Use the mesh as a base when you're making textures in external software. For example, as a starting point for making masks, make up effects or maps to use with the physically-based shader. headOccluder.fbx, headOccluder.dae and headOccluder.obj Combine this object with an occluder material to occlude the volume of the head. The mesh won't completely match the user's head, but it can give you a good idea. This is useful when you're creating effects with items like crowns or hats - because it will hide parts of the item that would be covered by the head in real life.  The face portion is flat. This means it won't interfere with the face mesh you'll add to the scene, which will reflect the face and movement of the user. faceMesh.png and faceMeshTrackers.png These textures represent the UV layout and topology of the face mesh used in Meta Spark Studio. faceMeshTrackers.png features red dots, highlighting the points on the face that are tracked by our system. Creating textures using these grids as a reference will help you understand where parts of the texture will be positioned in the effect. Keep in mind these textures omit the points that represent the edge of the face mesh at the side of the face. FaceMeshMask.png  This texture can be used to fade the edges of a material applied to someone's face. This is useful when you want to blend a material or texture with the face. In the image below, we've used FaceMeshMask in the effect on the right: To add this texture to your project: 1. Create a material for a face mesh in your scene. 2. Select the material in the Assets panel. 3. In the Inspector panel, check the box next to Alpha. 4. Import this texture from your computer, using the drop down next to Texture. faceFeminine.jpg and faceMasculine.jpg Use these textures as templates when creating your own textures for face effects. Import them into the software you use to create textures, and create your texture on a new layer. They'll help you see facial features in more detail — like the start and end of the eyebrows or the volume of the lips. These textures are grey in tone. The face tracker texture You can use the video of someone's face when they're using the effect as a texture in your scene, by extracting the face tracker texture. You can then apply the texture to a material and object - for example, to create a face swap effect. Creating a texture from the face tracker 1. Add a face tracker to your scene. 2. Select the face tracker in the Scene panel. 3. In the Inspector, click + next to Texture Extraction.  You'll see a texture called faceTracker0 in the Assets panel: Using the texture in an effect The texture can be applied to any material channel. It's a good idea to use a flat material. Or, drag it into the Patch Editor to combine with other visual shader patches. If you select the texture in the Assets panel, you'll be able to see and edit its properties in the Inspector. Copy Mode There are 2 options to choose from here. 3D Direct Use this option to create a 3D representation of the user's face. You'd generally use this mode when applying the texture to a face mesh, for example in a face swap effect. 2D Bounding box This mode will create a flat cutout of the face. You could use this mode to create 2D animations featuring the user's face. Enable Opening and Alignment If the texture is applied to a face mesh, check the boxes next to Eyes and Mouth. This will make sure the texture aligns properly with the face mesh. Creating a face swap effect Start by adding the face trackers: 1. Add two face trackers to your scene. 2. To help keep track, rename the face trackers - name the first face_1, and the second face_2. Then, edit the second face tracker so that it responds to the second face to appear in the scene: 1. Select face_2 in the Scene panel. 2. In the Inspector, set Tracked Face to Face 2.  Next, create the face mesh, textures and materials: 1. Extract the face tracker texture for each face tracker. 2. Create 2 face meshes, each as a child of one of the face trackers. 3. Create a material for each face mesh, and set the Shader Type to Flat. 4. Apply each face texture to the material applied to the opposite face tracker in the scene. The face mesh will have hard edges. To blend these with the face, use the face mesh mask included in the face reference assets. Segmentation Segmentation is used to detect and separate parts of the camera input. For example, coloring someone’s hair without changing anything else in the scene. There are 3 types of segmentation: Person segmentation — Separates the person using the effect from their background. Hair segmentation — Separates a person's hair from the rest of the scene. Skin segmentation — Separates any skin detected by the camera from the rest of the scene. The hand tracker Use the hand tracker to create effects that respond to the movement and position of a person’s hand. There are 2 ways to create hand tracking effects: Using the hand tracker object — Objects added to your scene as a child of the hand tracker move with the user’s hand. Only one hand can be tracked in this way. Using a hand bounding box — Plots an invisible rectangle around a selected hand calculating 2D coordinates for all four corners and the center of the rectangle, as well as its height and width. You can track both hands with this method. The body tracker Body tracking effects use body landmark patches to track features and objects to specific parts of the body. There are 2 ways to create body tracking effects: Body landmark patches — Body landmark patches enable the camera to track 20 unique points on a person’s body. Objects and effects can then be anchored to those points. Body bounding box patch — The body bounding box patches allow you to draw an invisible rectangle around a person’s body, calculating 2D coordinates for all four corners and the center of the rectangle, as well as its height and width. This rectangle resizes and moves based on the person’s position.