Interactive Video Projection Mapping with Three.js

Projection mapping has lengthy fascinated audiences within the bodily world, turning buildings, sculptures, and whole cityscapes into shifting canvases. What in the event you might recreate that very same sense of spectacle instantly contained in the browser? With WebGL and Three.js, you may undertaking video not onto partitions or monuments however onto dynamic 3D grids product of a whole bunch of cubes, each carrying a fraction of the video like a digital mosaic.

On this tutorial we’ll discover simulate video projection mapping in a purely digital atmosphere, from constructing a grid of cubes, to UV-mapping video textures, to making use of masks that decide which cubes seem. The result’s a mesmerizing impact that feels each sculptural and cinematic, good for interactive installations, portfolio showcases, or just as a playground to push your artistic coding abilities additional.

What’s Video Projection Mapping within the Actual World?

When describing video projection mapping, it’s best to consider enormous buildings lit up with animations throughout festivals, or artwork installations the place a shifting picture is “painted” onto sculptures.

Listed here are some examples of real-world video projections:

Bringing it to our 3D World

In 3D graphics, we are able to do one thing related: as a substitute of shining a bodily projector, we map a video texture onto objects in a scene.

Subsequently, let’s construct a grid of cubes utilizing a masks picture that may decide which cubes are seen. A video texture is UV-mapped so every dice reveals the precise video fragment that corresponds to its grid cell—collectively they reconstruct the video, however solely the place the masks is darkish.

Prerequesites:

• Three.js r155+
• A small, high-contrast masks picture (e.g. a coronary heart silhouette).
• A video URL with CORS enabled.

Our Boilerplate and Beginning Level

Here’s a primary starter setup, i.e. the minimal quantity of code and construction it is advisable get a scene rendering within the browser, with out worrying in regards to the particular artistic content material but.

export default class Fashions {
	constructor(gl_app) {
        ...
        this.createGrid()
    }

    createGrid() {
        const geometry = new THREE.BoxGeometry( 1, 1, 1 );
        this.materials = new THREE.MeshStandardMaterial( { coloration: 0xff0000 } );
        const dice = new THREE.Mesh( geometry, this.materials );
        this.group.add( dice );
        this.is_ready = true
    }
    
    ...
}

The result’s a spinning crimson dice:

Creating the Grid

A centered grid of cubes (10×10 by default). Each dice has the identical measurement and materials. The grid spacing and general scale are configurable.

export default class Fashions {
	constructor(gl_app) {
        ...

		this.gridSize = 10;
        this.spacing = 0.75;
        this.createGrid()
    }

    createGrid() {
        this.materials = new THREE.MeshStandardMaterial( { coloration: 0xff0000 } );
        
        // Grid parameters
        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);
                const mesh = new THREE.Mesh(geometry, this.materials);
                mesh.place.x = (x - (this.gridSize - 1) / 2) * this.spacing;
                mesh.place.y = (y - (this.gridSize - 1) / 2) * this.spacing;
                mesh.place.z = 0;

                this.group.add(mesh);
            }
        }
        this.group.scale.setScalar(0.5)
        ...
    }   
    ...
}

Key parameters

World-space distance between dice facilities. Enhance for bigger gaps, lower to pack tighter.

What number of cells per facet. A ten×10 grid ⇒ 100 cubes

Creating the Video Texture

This operate creates a video texture in Three.js so you should utilize a taking part in HTML as the feel on 3D objects.

• Creates an HTML component solely in JavaScript (not added to the DOM).
• We’ll feed this component to Three.js to make use of its frames as a texture.
• loop = true → restarts routinely when it reaches the tip.
• muted = true → most browsers block autoplay for unmuted movies, so muting ensures it performs with out person interplay.
• .play() → begins playback.
• ⚠️ Some browsers nonetheless want a click on/contact earlier than autoplay works — you may add a fallback listener if wanted.

export default class Fashions {
	constructor(gl_app) {
        ...
        this.createGrid()
    }

    createVideoTexture() {
		this.video = doc.createElement('video')
		this.video.src = 'https://commondatastorage.googleapis.com/gtv-videos-bucket/pattern/BigBuckBunny.mp4'
		this.video.crossOrigin = 'nameless'
		this.video.loop = true
		this.video.muted = true
		this.video.play()

		// Create video texture
		this.videoTexture = new THREE.VideoTexture(this.video)
		this.videoTexture.minFilter = THREE.LinearFilter
		this.videoTexture.magFilter = THREE.LinearFilter
		this.videoTexture.colorSpace = THREE.SRGBColorSpace
		this.videoTexture.wrapS = THREE.ClampToEdgeWrap
		this.videoTexture.wrapT = THREE.ClampToEdgeWrap

		// Create materials with video texture
		this.materials = new THREE.MeshBasicMaterial({ 
			map: this.videoTexture,
			facet: THREE.FrontSide
		})
    }

    createGrid() {
        this.createVideoTexture()
        ...
    }
    ...
}

That is the video we’re utilizing: Huge Buck Bunny (with out CORS)

All of the meshes have the identical texture utilized:

Attributing Projection to the Grid

We can be turning the video right into a texture atlas break up right into a gridSize × gridSize lattice.
Every dice within the grid will get its personal little UV window (sub-rectangle) of the video so, collectively, all cubes reconstruct the total body.

Why per-cube geometry? As a result of we are able to create a brand new BoxGeometry for every dice for the reason that UVs should be distinctive per dice. If all cubes shared one geometry, they’d additionally share the identical UVs and present the identical a part of the video.

export default class Fashions {
	constructor(gl_app) {
        ...
        this.createGrid()
    }

    createGrid() {
        ...
		// Grid parameters
        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                
                const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);
                
				// Create particular person geometry for every field to have distinctive UV mapping
				// Calculate UV coordinates for this particular field
				const uvX = x / this.gridSize
				const uvY = y / this.gridSize // Take away the flip to match appropriate orientation
				const uvWidth = 1 / this.gridSize
				const uvHeight = 1 / this.gridSize
				
				// Get the UV attribute
				const uvAttribute = geometry.attributes.uv
				const uvArray = uvAttribute.array
				
				// Map every face of the field to indicate the identical portion of video
				// We'll deal with the entrance face (face 4) for the principle projection
				for (let i = 0; i < uvArray.size; i += 2) {
					// Map all faces to the identical UV area for consistency
					uvArray[i] = uvX + (uvArray[i] * uvWidth)     // U coordinate
					uvArray[i + 1] = uvY + (uvArray[i + 1] * uvHeight) // V coordinate
				}
				
				// Mark the attribute as needing replace
				uvAttribute.needsUpdate = true
                ...
            }
        }
        ...
    }
    ...
}

The UV window for cell (x, y)
For a grid of measurement N = gridSize:

• UV origin of this cell:
  – uvX = x / N
  – uvY = y / N
• UV measurement of every cell:
  – uvWidth = 1 / N
  – uvHeight = 1 / N

Outcome: each face of the field now samples the identical sub-region of the video (and we famous “deal with the entrance face”; this method maps all faces to that area for consistency).

Creating Masks

We have to create a canvas utilizing a masks that determines which cubes are seen within the grid.

• Black (darkish) pixels → dice is created.
• White (mild) pixels → dice is skipped.

To do that, we have to:

1. Load the masks picture.
2. Scale it right down to match our grid measurement.
3. Learn its pixel coloration information.
4. Cross that information into the grid-building step.

export default class Fashions {
	constructor(gl_app) {
        ...
		this.createMask()
    }

	createMask() {
        // Create a canvas to learn masks pixel information
        const canvas = doc.createElement('canvas')
        const ctx = canvas.getContext('2nd')

        const maskImage = new Picture()
        maskImage.crossOrigin = 'nameless'
        maskImage.onload = () => {
            // Get authentic picture dimensions to protect side ratio
            const originalWidth = maskImage.width
            const originalHeight = maskImage.peak
            const aspectRatio = originalWidth / originalHeight

            // Calculate grid dimensions based mostly on side ratio
            this.gridWidth
			this.gridHeight
            if (aspectRatio > 1) {
                // Picture is wider than tall
                this.gridWidth = this.gridSize
                this.gridHeight = Math.spherical(this.gridSize / aspectRatio)
            } else {
                // Picture is taller than extensive or sq.
                this.gridHeight = this.gridSize
                this.gridWidth = Math.spherical(this.gridSize * aspectRatio)
            }

            canvas.width = this.gridWidth
            canvas.peak = this.gridHeight
            ctx.drawImage(maskImage, 0, 0, this.gridWidth, this.gridHeight)

            const imageData = ctx.getImageData(0, 0, this.gridWidth, this.gridHeight)
            this.information = imageData.information
			this.createGrid()
		}

        maskImage.src = '../pictures/coronary heart.jpg'
	}
    ...
}

Match masks decision to grid

• We don’t need to stretch the masks — this retains it proportional to the grid.
• gridWidth and gridHeight are what number of masks pixels we’ll pattern horizontally and vertically.
• This matches the logical dice grid, so every dice can correspond to at least one pixel within the masks.

Making use of the Masks to the Grid

Let’s combines mask-based filtering with customized UV mapping to resolve the place within the grid containers ought to seem, and how every field maps to a piece of the projected video.
Right here’s the idea step-by-step:

• Loops by each potential (x, y) place in a digital grid.
• At every grid cell, it can resolve whether or not to position a field and, if that’s the case, texture it.
• flippedY: Flips the Y-axis as a result of picture coordinates begin from the top-left, whereas the grid’s origin begins from the bottom-left.
• pixelIndex: Locates the pixel within the this.information array.
• Every pixel shops 4 values: crimson, inexperienced, blue, alpha.
• Extracts the R, G, and B values for that masks pixel.
• Brightness is calculated as the typical of R, G, B.
• If the pixel is darkish sufficient (brightness < 128), a dice can be created.
• White pixels are ignored → these positions keep empty.

export default class Fashions {
	constructor(gl_app) {
        ...
		this.createMask()
    }

	createMask() {
        ...
	}

    createGrid() {
        ...
        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                
                const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);

                // Get pixel coloration from masks (pattern at grid place)
                // Flip Y coordinate to match picture orientation
                const flippedY = this.gridHeight - 1 - y
                const pixelIndex = (flippedY * this.gridWidth + x) * 4
                const r = this.information[pixelIndex]
                const g = this.information[pixelIndex + 1]
                const b = this.information[pixelIndex + 2]

                // Calculate brightness (0 = black, 255 = white)
                const brightness = (r + g + b) / 3

                // Solely create field if pixel is darkish (black reveals, white hides)
                if (brightness < 128) { // Threshold for black vs white

                    // Create particular person geometry for every field to have distinctive UV mapping
                    // Calculate UV coordinates for this particular field
                    const uvX = x / this.gridSize
                    const uvY = y / this.gridSize // Take away the flip to match appropriate orientation
                    const uvWidth = 1 / this.gridSize
                    const uvHeight = 1 / this.gridSize
                    
                    // Get the UV attribute
                    const uvAttribute = geometry.attributes.uv
                    const uvArray = uvAttribute.array
                    
                    // Map every face of the field to indicate the identical portion of video
                    // We'll deal with the entrance face (face 4) for the principle projection
                    for (let i = 0; i < uvArray.size; i += 2) {
                        // Map all faces to the identical UV area for consistency
                        uvArray[i] = uvX + (uvArray[i] * uvWidth)     // U coordinate
                        uvArray[i + 1] = uvY + (uvArray[i + 1] * uvHeight) // V coordinate
                    }
                    
                    // Mark the attribute as needing replace
                    uvAttribute.needsUpdate = true
                    
                    const mesh = new THREE.Mesh(geometry, this.materials);

                    mesh.place.x = (x - (this.gridSize - 1) / 2) * this.spacing;
                    mesh.place.y = (y - (this.gridSize - 1) / 2) * this.spacing;
                    mesh.place.z = 0;

                    this.group.add(mesh);
                }
            }
        }
        ...
    }
    ...
}

Additional steps

• UV mapping is the method of mapping 2D video pixels onto 3D geometry.
• Every dice will get its personal distinctive UV coordinates similar to its place within the grid.
• uvWidth and uvHeight are how a lot of the video texture every dice covers.
• Modifies the dice’s uv attribute so all faces show the very same portion of the video.

Right here is the outcome with the masks utilized:

Including Some Depth and Movement to the Grid

Including delicate movement alongside the Z-axis brings the in any other case static grid to life, making the projection really feel extra dynamic and dimensional.

replace() {
    if (this.is_ready) {
        this.group.youngsters.forEach((mannequin, index) => {
            mannequin.place.z = Math.sin(Date.now() * 0.005 + index * 0.1) * 0.6
        })
    }
}

It’s the time for A number of Grids

Up till now we’ve been working with a single masks and a single video, however the actual enjoyable begins after we begin layering a number of projections collectively. By combining completely different masks pictures with their very own video sources, we are able to create a set of impartial grids that coexist in the identical scene. Every grid can carry its personal identification and movement, opening the door to richer compositions, transitions, and storytelling results.

1. A Playlist of Masks and Movies

export default class Fashions {
	constructor(gl_app) {
        ...
        this.grids_config = [
            {
                id: 'heart',
                mask: `heart.jpg`,
                video: `fruits_trail_squared-transcode.mp4`
            },
            {
                id: 'codrops',
                mask: `codrops.jpg`,
                video: `KinectCube_1350-transcode.mp4`
            },
            {
                id: 'smile',
                mask: `smile.jpg`,
                video: `infinte-grid_squared-transcode.mp4`
            },
        ]
        this.grids_config.forEach((config, index) => this.createMask(config, index))
        this.grids = []
    }
...
}

As a substitute of 1 masks and one video, we now have an inventory of mask-video pairs.

Every object defines:

• id → title/id for every grid.
• masks → the black/white picture that controls which cubes seem.
• video → the feel that can be mapped onto these cubes.

This lets you have a number of completely different projections in the identical scene.

2. Looping Over All Grids

As soon as we now have our playlist of masks–video pairs outlined, the following step is to undergo every merchandise and put together it for rendering.

For each configuration within the checklist we name createMask(config, index), which takes care of loading the masks picture, studying its pixels, after which passing the information alongside to construct the corresponding grid.

On the identical time, we maintain observe of all of the grids by storing them in a this.grids array, so afterward we are able to animate them, present or disguise them, and change between them interactively.

3. createMask(config, index)

createMask(config, index) {
    ...
    maskImage.onload = () => {
        ...
        this.createGrid(config, index)
    }
    maskImage.src = `../pictures/${config.masks}`
}

• Masses the masks picture for the present grid.
• When the picture is loaded, runs the masks pixel-reading logic (as defined earlier than) after which calls createGrid() with the identical config and index.
• The masks determines which cubes are seen for this particular grid.

4. createVideoTexture(config, index)

createVideoTexture(config, index) {
    this.video = doc.createElement('video')
    this.video.src = `../movies/${config.video}`
    ...
}

• Creates a component utilizing the particular video file for this grid.
• The video is then transformed to a THREE.VideoTexture and assigned as the fabric for the cubes on this grid.
• Every grid can have its personal impartial video taking part in.

5. createGrid(config, index)

createGrid(config, index) {
        this.createVideoTexture(config, index)
        const grid_group = new THREE.Group()
        this.group.add(grid_group)

        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                    ...
                    grid_group.add(mesh);
            }
        }
        grid_group.title = config.id
        this.grids.push(grid_group);
        grid_group.place.z = - 2 * index 
        ...
    }

• Creates a brand new THREE.Group for this grid so all its cubes could be moved collectively.
• This retains every masks/video projection remoted.
• grid_group.title: Assigns a reputation (you may later use config.id right here).
• this.grids.push(grid_group): Shops this grid in an array so you may management it later (e.g., present/disguise, animate, change movies).
• grid_group.place.z: Offsets every grid additional again in Z-space so that they don’t overlap visually.

And right here is the outcome for the a number of grids:

And eventually: Interplay & Animations

Let’s begin by making a easy UI with some buttons on our HTML:

[data-current="heart"] {
	background-color: #e19800;
}

[data-current="codrops"] {
	background-color: #00a00b
}

[data-current="smile"] {
	background-color: #b90000;
}

createGrid(config, index) {
    ...
    this.initInteractions()
}

initInteractions() {
    this.present = 'coronary heart'
    this.previous = null
    this.is_animating = false
    this.period = 1

    this.DOM = {
        $btns: doc.querySelectorAll('.btns__item button'),
        $canvas: doc.querySelector('canvas')
    }
    this.grids.forEach(grid => {
        if(grid.title != this.present) {
            grid.youngsters.forEach(mesh => mesh.scale.setScalar(0))
        }
    })
    this.bindEvents()
}

bindEvents() {
    this.DOM.$btns.forEach(($btn, index) => {
        $btn.addEventListener('click on', () => {
            if (this.is_animating) return
            this.is_animating = true
            this.DOM.$btns.forEach(($btn, btnIndex) => {
                btnIndex === index ? $btn.classList.add('lively') : $btn.classList.take away('lively')
            })
            this.previous = this.present
            this.present = `${$btn.dataset.id}`
            this.revealGrid()
            this.hideGrid()
        })
    })
}

revealGrid() {
    // Filter the present grid based mostly on this.present worth
    const grid = this.grids.discover(merchandise => merchandise.title === this.present);
    
    this.DOM.$canvas.dataset.present = `${this.present}` 
    const tl = gsap.timeline({ delay: this.period * 0.25, defaults: { ease: 'power1.out', period: this.period } })
    grid.youngsters.forEach((youngster, index) => {
        tl
            .to(youngster.scale, { x: 1, y: 1, z: 1, ease: 'power3.inOut' }, index * 0.001)
            .to(youngster.place, { z: 0 }, '<')
    })
}

hideGrid() {
    // Filter the present grid based mostly on this.previous worth
    const grid = this.grids.discover(merchandise => merchandise.title === this.previous);
    const tl = gsap.timeline({
        defaults: { ease: 'power1.out', period: this.period },
        onComplete: () => { this.is_animating = false }
    })
    grid.youngsters.forEach((youngster, index) => {
        tl
            .to(youngster.scale, { x: 0, y: 0, z: 0, ease: 'power3.inOut' }, index * 0.001)
            .to(youngster.place, {
                z: 6, onComplete: () => {
                    gsap.set(youngster.scale, { x: 0, y: 0, z: 0 })
                    gsap.set(youngster.place, { z: - 6 })
                }
            }, '<')
    })
}