Constructing a Actual-Time 3D Face Masks with MediaPipe, Threlte and Three.js

Editor’s Word: As our Three.js Convention celebration continues, we’re particularly excited to shine a lightweight on Marek Jóźwiak right now. Marek was so wonderful to submit this incredible tutorial and demo, and his expertise and a spotlight to element actually shine via on this exploration of MediaPipe, Threlte, and Three.js. We’re thrilled to have his work as a part of our celebration and may’t wait so that you can dive in!

I began this experiment with a modest objective: put a Three.js materials on my face.

The primary prototype was much less convincing than that sentence sounds. MediaPipe tracked the face, however Three.js nonetheless had no floor to render. After I provided a topology, the mesh and the mirrored, cropped video disagreed about framing. Then the feel appeared the wrong way up.

That sequence turned the precise topic of the experiment: recovering Google’s fastened face topology, transferring UVs from the canonical mannequin, and projecting the outcome into the identical view because the digital camera picture. Initializing MediaPipe and Three.js was the straightforward half.

From landmarks to a textured mesh

The runtime has two clocks. MediaPipe analyzes the video and writes its newest outcome to a mutable reference. Threlte reads that reference inside its render job and mutates the present BufferGeometry. Inference can miss a render body with out pushing 468 altering positions via Svelte’s reactive graph.

The digital camera layer itself is typical. The element that issues later is visible: the video makes use of object-cover and is mirrored, so the projection utilized to the mesh should reproduce each operations.

I wrapped MediaPipe in a small service class. It resolves the Duties Imaginative and prescient WASM information, hundreds the float16 face mannequin, and selects the GPU delegate.

const VISION_BASE_URL = 'https://cdn.jsdelivr.web/npm/@mediapipe/tasks-vision@0.10.35/wasm';

const filesetResolver = await FilesetResolver.forVisionTasks(VISION_BASE_URL);

this.faceLandmarker = await FaceLandmarker.createFromOptions(filesetResolver, {
	baseOptions: {
		modelAssetPath:
			'https://storage.googleapis.com/mediapipe-models/' +
			'face_landmarker/face_landmarker/float16/1/' +
			'face_landmarker.job',
		delegate: 'GPU'
	},
	outputFaceBlendshapes: true,
	runningMode: 'VIDEO',
	numFaces: 1
});

Video mode expects a monotonically rising timestamp. efficiency.now() is a handy supply:

startPrediction(videoElement: HTMLVideoElement) {
  const predict = () => {
    if (
      this.faceLandmarker &&
      videoElement.readyState >= 2
    ) {
      const outcomes = this.faceLandmarker.detectForVideo(
        videoElement,
        efficiency.now()
      );

      this.landmarksRef.present = outcomes;
    }

    this.requestRef = requestAnimationFrame(predict);
  };

  predict();
}

I retailer the end in landmarksRef, a plain object moderately than a deeply reactive Svelte array:

landmarksRef = { present: null };

Nothing within the DOM must rerender when 468 positions change. Three.js will learn them throughout its personal body loop. Holding this scorching information path outdoors Svelte’s reactive graph avoids asking the UI framework to look at 1000’s of quantity assignments per second.

The bundled mannequin returns 478 landmarks. This mesh consumes the primary 468 as a result of they match MediaPipe’s canonical face mannequin and its fastened topology. The remaining ten describe the irises and will not be wanted for the masks floor.

That produced secure landmark positions, however not a floor. I nonetheless wanted the index buffer that describes which landmark IDs kind every face:

[
	127,
	34,
	139, // triangle 1
	11,
	0,
	37, // triangle 2
	232,
	231,
	120 // triangle 3
];

The order is important as a result of it determines winding and due to this fact which aspect Three.js treats because the entrance face.

Google had already printed the face mesh topology I wanted. I discovered the flattened array within the face landmark detection demo in Google’s TensorFlow.js fashions repository. That file begins with the identical sequence:

127, 34, 139,
11, 0, 37,
232, 231, 120,

It incorporates 2,640 indices, or 880 triangles. I copied the flattened information into FaceTriangulation.ts and use it instantly as FACE_MESH_TRIANGULATION.

The present MediaPipe repository additionally publishes FACE_LANDMARKS_TESSELATION within the Python Face Landmarker API. It shops connections moderately than triangle triples. Its opening entries are:

Connection(127, 34),
Connection(34, 139),
Connection(139, 127),

Connection(11, 0),
Connection(0, 37),
Connection(37, 11),

The connection desk encodes the identical opening triangles as closed edge cycles. 127 → 34 → 139 → 127 turns into [127, 34, 139]; 11 → 0 → 37 → 11 turns into [11, 0, 37].

The TensorFlow.js illustration is extra handy for an listed BufferGeometry as a result of it’s already flattened into triangle triples. The MediaPipe connection units stay helpful when an impact wants named areas such because the eyes, lips, or face oval.

Each repositories publish the related code below the Apache 2.0 license.

With the index recovered, I may allocate the geometry buffers as soon as:

const vertexCount = 468;
const indices = new Uint16Array(FACE_MESH_TRIANGULATION);
const positions = new Float32Array(vertexCount * 3);
const uvs = FACE_MESH_UVS;

Then Threlte creates the Three.js buffer attributes:

DynamicDrawUsage is a driver hint, not an update mechanism. The later position.needsUpdate = true marks the buffer for upload. Topology and UVs stay fixed; only the 468 XYZ positions are rewritten.

Wireframe mode is the best first test. A solid black material can hide bad topology surprisingly well, while a wireframe immediately shows disconnected vertices, a reversed face, or a badly scaled depth axis.

The triangle index solved connectivity. Texture mapping introduced a second indexing problem: the UV attribute still had to be reordered into the same 468-entry vertex order.

Google provides a canonical_face_model.obj in the MediaPipe repository. The Face Geometry documentation describes it as the bridge between a static asset and the runtime landmark set. The useful invariant here is the shared landmark index space: vertex 127 in the canonical asset corresponds to landmark 127 returned by the detector.

The same directory also contains Google’s canonical_face_model_uv_visualization.png. It is an excellent debug texture because its printed grid makes flipped or mismatched UVs obvious.

The OBJ contains:

• 468 v records for vertex positions;
• 468 vt records for texture coordinates;
• 898 f records describing faces.

The counts reveal an important distinction between the two source files. The TensorFlow.js index has 880 triangles, while the canonical OBJ has 898 faces, and the face lists are not identical. I do not combine them. The runtime index comes entirely from triangulation.js; the OBJ is only used to recover the UV assigned to each of the 468 vertex IDs.

This is valid because a BufferGeometry index addresses every vertex attribute at once. Once the UV array is reordered to match the landmark positions, each selected triangle interpolates position and UV from the same three indices. The OBJ’s own f records are necessary during extraction because they reveal the vertex-to-UV mapping, not because they become the runtime index buffer.

The next problem was recovering the actual vertex-to-UV relation. The number of v and vt records happens to match, but their file order is not the mapping. OBJ face records carry that relationship explicitly:

f 174/43 156/119 134/220

Here 174/43 means vertex 174 uses texture coordinate 43. OBJ indices are one-based, so both values are decremented before they address JavaScript arrays.

I wrote a small Node script to extract that relationship and generate a typed TypeScript array. Parsing an OBJ at runtime would make the demo heavier for no benefit because these UVs never change.

The input and output paths are resolved relative to the script, so it can be run from the project root with node scripts/extract-uvs.js:

const objPath = join(__dirname, '../src/lib/assets/models/canonical_face_model.obj');

const outputPath = join(__dirname, '../src/lib/utils/FaceUVs.ts');

The parser collects vt records, then resolves their relationship to vertices from each f token:

const textureCoords = [];
const vertexToUV = new Map();

for (const line of traces) {
	const trimmed = line.trim();

	if (trimmed.startsWith('vt ')) {
		const elements = trimmed.break up(/s+/);
		const u = parseFloat(elements[1]);
		const v = parseFloat(elements[2]);

		textureCoords.push([u, v]);
	} else if (trimmed.startsWith('f ')) {
		const elements = trimmed.break up(/s+/).slice(1);

		for (const a part of elements) {
			const indices = half.break up('/');
			const vertexIdx = parseInt(indices[0]) - 1;
			const uvIdx = parseInt(indices[1]) - 1;

			if (!vertexToUV.has(vertexIdx)) {
				vertexToUV.set(vertexIdx, uvIdx);
			}
		}
	}
}

On this canonical file, all 468 vertices resolve to precisely one distinctive UV, so the relation is a bijection. Holding the primary project is protected right here, however it isn’t a general-purpose OBJ technique: a mannequin with UV seams might assign a number of texture coordinates to at least one geometric vertex and should duplicate that vertex within the render buffer.

With that map constructed, the script emits UVs in landmark order:

const uvArray = [];

for (let i = 0; i < 468; i++) {
	const uvIdx = vertexToUV.get(i);

	if (uvIdx !== undefined && textureCoords[uvIdx]) {
		const [u, v] = textureCoords[uvIdx];
		uvArray.push(u, 1.0 - v);
	} else {
		uvArray.push(0.5, 0.5);
	}
}

The generated Float32Array incorporates 936 values, two for every vertex, and the 1.0 - v conversion flips the vertical texture coordinate. Since that flip already occurs within the generated information, I disable Three.js’s typical picture flip when loading the feel:

texture.flipY = false;
texture.colorSpace = THREE.SRGBColorSpace;

Lacking that relationship gave me an upside-down masks. Flipping both the UV information or the feel is okay; flipping each will not be.

Matching the mesh to the digital camera

With topology and UVs solved, the subsequent mismatch was spatial. MediaPipe returns normalized coordinates, whereas the mesh lives in Three.js world models.

For a perspective digital camera, the seen top at a distance d is:

const distance = digital camera.place.z;
const vFov = (digital camera.fov * Math.PI) / 180;
const top = 2 * Math.tan(vFov / 2) * distance;
const width = top * viewportAspect;

If the video had been stretched to the viewport, width and top could be sufficient. It makes use of object-cover, so we have to reproduce that sizing rule:

let scaleX = width;
let scaleY = top;

const videoAspect = videoElement.videoWidth / videoElement.videoHeight;
const screenAspect = viewportWidth / viewportHeight;

if (screenAspect > videoAspect) {
	scaleY = width / videoAspect;
} else {
	scaleX = top * videoAspect;
}

When the display screen is wider than the digital camera picture, object-cover matches the video to the display screen width and crops it vertically. In that case the WebGL airplane should additionally turn out to be taller. On a comparatively slim display screen, the video matches by top and overflows horizontally, so the WebGL airplane turns into wider.

Now every landmark may be mapped into the seen airplane whereas retaining MediaPipe’s relative depth:

const z = -landmark.z * scaleX * maskScale * depthScale + offsetZ;
const depthRatio = (distance - z) / distance;

const x = ((0.5 - landmark.x) * scaleX * maskScale + offsetX) * depthRatio;

const y = (-(landmark.y - 0.5) * scaleY * maskScale + offsetY) * depthRatio;

Subtracting 0.5 recenters normalized coordinates. The indicators on X and Y account for the mirrored video and opposing picture/WebGL Y axes. Z follows the seen width, with a separate multiplier as a result of MediaPipe depth is relative moderately than a Three.js world-space measurement.

MediaPipe’s X and Y values are already projected display screen coordinates. Giving every vertex a non-zero Z place after which passing it via a perspective digital camera would challenge it once more, shifting it away from the tracked place. Right here the digital camera is fastened at (0, 0, 5) and the reference airplane is z = 0, so multiplying X and Y by (distance - z) / distance strikes the vertex alongside the identical digital camera ray. Perspective projection then returns it to the tracked display screen place whereas Z nonetheless shapes the floor normals and lighting.

The size and offsets are uncovered in Tweakpane. They’re helpful whereas growing, particularly when testing totally different webcams, however the facet calculation does many of the alignment work.

With the projection outlined, the Threlte useTask callback can write the present landmark positions into the present buffer. It hides the masks when no face is offered:

useTask(() => {
	if (!landmarkRef.present?.faceLandmarks?.size || !videoElement || !geometry || !mesh) {
		if (mesh) mesh.seen = false;
		return;
	}

	mesh.seen = true;

	const face = landmarkRef.present.faceLandmarks[0];
	const place = geometry.attributes.place;

	for (let i = 0; i < 468; i++) {
		const landmark = face[i];
		if (!landmark) proceed;

		const z = -landmark.z * scaleX * maskScale * depthScale + offsetZ;
		const depthRatio = (distance - z) / distance;
		const x = ((0.5 - landmark.x) * scaleX * maskScale + offsetX) * depthRatio;
		const y = (-(landmark.y - 0.5) * scaleY * maskScale + offsetY) * depthRatio;

		place.setXYZ(i, x, y, z);
	}

	place.needsUpdate = true;
	geometry.computeVertexNormals();
});

No geometry is allotted inside the new loop. The index and UV attributes by no means transfer, and the place attribute retains the identical dimension for the lifetime of the element. Changing it each body would create rubbish and pressure Three.js to rebuild GPU sources.

Normals do want to alter. A MeshPhysicalMaterial reacts to mild in accordance with the floor regular, and the floor modifications once I flip my head or open my mouth. computeVertexNormals() recalculates them after the place replace.

As soon as the geometry was secure, I wished to examine it in two methods. The experiment retains one MeshPhysicalMaterial and switches its map at runtime. Texture mode units the bottom coloration to white, then neutralizes the bodily properties that may obscure the UV debug picture:

const useTexture = maskState.renderMode === 'texture';

materials.map = useTexture ? maskTexture : null;
materials.coloration.set(useTexture ? '#ffffff' : maskState.coloration);
materials.metalness = useTexture ? 0 : maskState.metalness;
materials.roughness = useTexture ? 1 : maskState.roughness;
materials.clearcoat = useTexture ? 0 : maskState.clearcoat;
materials.clearcoatRoughness = useTexture ? 0 : maskState.clearcoatRoughness;
materials.transmission = useTexture ? 0 : maskState.transmission;
materials.thickness = useTexture ? 0 : maskState.thickness;
materials.iridescence = useTexture ? 0 : maskState.iridescence;
materials.flatShading = useTexture ? false : maskState.flatShading;
materials.needsUpdate = true;

needsUpdate issues as a result of including or eradicating a map modifications the shader defines. The feel is disposed when the element unmounts; the MediaPipe animation body is canceled and the landmarker is closed by its service.

The identical separation formed the element boundary. FaceLandmarkerService owns inference and its newest outcome; FaceMask.svelte owns the per-frame geometry mutation. Materials controls dwell in slower reactive state. Making the landmark array deeply reactive would add bookkeeping with out making a helpful DOM replace.

Threlte additionally retains the scene markup brief with out hiding Three.js. BufferGeometry, BufferAttribute, MeshPhysicalMaterial, DynamicDrawUsage, and computeVertexNormals() are nonetheless the identical Three.js objects and APIs.

The shocking a part of this experiment was not loading a machine-learning mannequin. MediaPipe makes that pretty direct. The true work was understanding the information across the mannequin: the fastened face topology, the canonical OBJ, UV indexing, picture orientation, and the digital camera crop.

As soon as these items agree, the face turns into an extraordinary dynamic Three.js mesh. It will possibly use a bodily materials, a diagnostic UV picture, or any texture designed across the canonical structure. The monitoring mannequin provides the movement, however the mesh is open to the identical rendering concepts as every other geometry in a WebGL scene.

And for those who ever end up observing a file that begins with 127, 34, 139, no less than now you recognize the place it got here from.