Find out how to Create a Pixel-to-Voxel Video Drop Impact with Three.js and Rapier

On this tutorial, we’ll discover how one can use Three.js and shaders to take a daily video, pixelate the 2D feed, and extrude these outcomes into 3D voxels, lastly controlling them with the Rapier physics engine.

Let’s kick issues off by having a look on the demo.

Whoa, what is that this?! Or as we are saying in Japanese: “Nanja korya!!!”

Study trendy internet animation utilizing GSAP 3 with 34 hands-on video classes and sensible initiatives — good for all ability ranges.

Test it out

Idea

Pixels

Pixels are the basic items that make up the “type” we see on digital screens. And the general “form” is nothing greater than a set of these pixels… I’ve at all times been fascinated by the idea of pixels, so I’ve been creating pixel artwork utilizing Three.js and shaders. I really like pursuing distinctive visuals that spark curiosity by surprising expressions. For instance, I’ve beforehand explored reconstructing dwell digicam streams by changing particular person pixels with numerous concrete objects to create a playful digital transformation. Whereas making issues like that, a thought immediately struck me:What would occur if I extruded these pixels into 3D house?

Gravity

One other main theme I’ve been exploring lately is “Gravity”. Not anti-gravity! Simply plain gravity! Whereas constructing web sites powered by physics engines, I’ve been chasing the chaos and concord that unfold inside them. I’ve additionally lengthy wished to take the Gravity I’ve been working with in 2D areas and increase it into 3D.

All of those concepts and explorations type the start line for this venture.Similar to with pixel artwork, I deal with a single “InstancedMesh” in Three.js as one “pixel”. I organize these neatly in a grid so that they type a flat airplane… then flip each right into a 3D voxel. On high of that, I create a corresponding inflexible physique for every voxel and place them below a physics engine.

Tech stack used on this demo:

• Rendering: Three.js + Shaders
• Physics: Rapier
• Animation/Management: GSAP
• GUI: Tweakpane

Alright – let’s get began!

Implementation

We’ll implement this in three primary steps.

1. Create a Flat Aircraft from InstancedMeshes and Inflexible Our bodies

Strategy: The aim is to coordinate a set of InstancedMesh items with their corresponding inflexible our bodies in order that, collectively, they seem as a single, flat video airplane.

Consider it as creating one thing like this: (For readability, I’ve added small gaps between every InstancedMesh within the picture beneath.)

Pixel Unit Settings

We resolve the dimensions of every pixel by dividing the video texture primarily based on the specified variety of columns. From this, we calculate the mandatory values to rearrange sq. InstancedMesh correctly.

We retailer all the mandatory knowledge in this.gridInfo. This calculation is dealt with by the calculateGridMetrics perform inside utils/mathGrid.js.

// index.js

/**
 * Computes and registers grid metrics into this.gridInfo:
 * - Dimensions (cols, rows, rely)
 * - Sizing (cellSize, spacing)
 * - Format (width, top)
 */
buildScene() {
//...
  this.gridInfo = calculateGridMetrics(
    this.videoController.width,
    this.videoController.top,
    this.app.digicam,
    this.settings.cameraDistance,
    this.params,
    this.settings.fitMargin
  );
//...
}

Creating Inflexible Our bodies and InstancedMeshes

Utilizing the grid data we obtained earlier, we’ll now create the inflexible our bodies to position within the physics world, together with the corresponding InstancedMeshes. We’ll begin by creating and positioning the inflexible our bodies first, then create the InstancedMeshes in sync with them.

The important thing level right here is that – anticipating the transition to voxels, we instantiate every unit utilizing BoxGeometry. Nonetheless, we initially flatten them within the vertex shader by scaling the depth near zero. This permits us to initialize them so that they visually seem as flat tiles, identical to ones made with PlaneGeometry.

Every of those particular person InstancedMeshes will symbolize one pixel.

The createGridBodies methodology is the muse of every part. As quickly because the gridInfo is decided, this methodology generates all the mandatory parts without delay: creating the inflexible our bodies, setting their preliminary positions and rotations, and producing the noise and random values required for the vertex results and field shapes.

// core/PhysicsWorld.js (simplified)

createGridBodies(gridInfo, params, settings) {
  // Put together knowledge constructions (TypedArrays for efficiency)
  this.cachedUVs = new Float32Array(rely * 2);
  this.noiseOffsets = new Float32Array(rely);
  const randomDepths = new Float32Array(rely);

  for (let i = 0; i < rely; i++) {
    const x = startX + col * spacing;
    const y = startY + row * spacing;
    const u = (col + 0.5) / cols;
    const v = (row + 0.5) / rows;

    // Caching UVs for the dice drop
    this.cachedUVs[i * 2] = u;
    this.cachedUVs[i * 2 + 1] = v;

    // Generate noise and random depth scales
    this.noiseOffsets[i] = computeNoiseOffset(i);
    randomDepths[i] = params.form === "random" ? (Math.random() < 0.2 ? 1.0 : 1.0 + Math.random() * 1.5) : 1.0;

    // Create RAPIER Inflexible Our bodies and Colliders
    const rbDesc = RAPIER.RigidBodyDesc.fastened().setTranslation(x, y, 0);
    const rb = this.world.createRigidBody(rbDesc);
    const clDesc = RAPIER.ColliderDesc.cuboid(halfSize, halfSize, halfSize * depthScale);
    this.world.createCollider(clDesc, rb);
  }
}

We initialize every occasion as a dice utilizing BoxGeometry.

// view/PixelVoxelMesh.js

createGeometry() {
  return new THREE.BoxGeometry(
  this.gridInfo.cellSize, this.gridInfo.cellSize, this.gridInfo.cellSize);
}

We use setMatrixAt to synchronize the positions and rotations of the InstancedMesh with their corresponding inflexible our bodies.

// view/PixelVoxelMesh.js

setMatrixAt(index, pos, rot) {
  this.dummy.place.set(pos.x, pos.y, pos.z);
  this.dummy.quaternion.set(rot.x, rot.y, rot.z, rot.w);
  this.dummy.scale.set(1, 1, 1);
  this.dummy.updateMatrix();
  this.mesh.setMatrixAt(index, this.dummy.matrix);
}

Enjoying the Video on a Single Massive Aircraft

We go the video as a texture to the fabric of those InstancedMeshes, turning the gathering of small items into one giant, cohesive airplane. Within the vertex shader, we specify the corresponding video coordinates for every InstancedMesh, defining the precise width and top every occasion is answerable for. At first look, it seems as a daily video taking part in on a flat floor – which is how the visible seen originally is achieved.

// view/PixelVoxelMesh.js

createMaterial(videoTexture) {
  return new THREE.ShaderMaterial({
    uniforms: {
      uMap: {worth: videoTexture},
      uGridDims: {worth: new THREE.Vector2(this.gridInfo.cols, this.gridInfo.rows)},
      uCubeSize: {worth: new THREE.Vector2(this.gridInfo.cellSize, this.gridInfo.cellSize)},
//...

// vertex.glsl

float instanceId = float(gl_InstanceID);
float col = mod(instanceId, uGridDims.x);
float row = flooring(instanceId / uGridDims.x);
vec2 uvStep = 1.0 / uGridDims;
vec2 baseUv = vec2(col, row) * uvStep;

2. Including Results with Shaders

Strategy: Within the vertex shader, we initially set the depth to a really small worth (0.05 on this case) to maintain every part flat. By progressively growing this depth worth over time, the unique dice form – created with BoxGeometry – will get restored.

This straightforward mechanism kinds the core of our voxelization course of.

One of many key factors is how we animate this depth enhance. For this demo, we’ll do it by a “ripple impact.” Ranging from a particular level, a wave spreads outward, progressively turning the flat InstancedMeshes into full cubes.

On high of that, we’ll combine in some glitch results and RGB shifts in the course of the ripple, then transition to a pixelated single-color look (the place every total InstancedMesh is full of the colour sampled from the middle of its UV). Lastly, the shapes settle into correct 3D voxels.

Within the demo, I’ve ready two totally different modes for this ripple impact. You may change between them utilizing the “Natural” toggle within the GUI.

Ripple Impact

JavaScript: Calculating the Wave Vary

In controllers/SequenceController.js, the targetVal – the ultimate aim of the animation – is calculated primarily based on the space from the wave’s origin to the farthest InstancedMesh. This ensures the ripple covers each aspect earlier than ending.

//controllers/SequenceController.js

startCollapse (u, v) {
//...
  pixelVoxelMesh.setUniform("uRippleCenter", { x: u, y: v });
  const side = this.primary.gridInfo.cols / this.primary.gridInfo.rows;
  const centerX = u * side;
  const centerY = v;
  const corners = [ { x: 0, y: 0 }, { x: aspect, y: 0 }, { x: 0, y: 1 }, { x: aspect, y: 1 } ];
  const maxDist = Math.max(...corners.map((c) => Math.hypot(c.x - centerX, c.y - centerY)));

　// Calculate last ripple radius primarily based on most distance and unfold
  const targetVal = maxDist + params.effectSpread + 0.1;

  gsap.to(pixelVoxelMesh.materials.uniforms.uVoxelateProgress, {
    worth: targetVal,
//...

Vertex Shader: Controlling the Transformation Progress

• International Progress (uVoxelateProgress): This worth will increase as much as targetVal, guaranteeing the wave expands till it reaches the furthest corners of the display.
• Native Progress: Every InstancedMesh begins its transformation the second the wave passes its place. By utilizing features like clamp and smoothstep, the progress worth is locked at 1.0 as soon as the wave has handed – this ensures that each mesh finally completes its transition right into a full dice.

We begin with a flat 0.05 depth and scale as much as the targetDepth throughout voxelization, making use of the progress worth from the chosen mode to drive the transformation.

// vertex.glsl

// --- Progress ---
float progressOrganic = clamp((uVoxelateProgress - noisyDist) / uEffectSpread, 0.0, 1.0);
float progressSmooth = smoothstep(distFromCenter, distFromCenter + uEffectSpread, uVoxelateProgress);
  
// --- Coordinate Transformation (Depth) ---
float finalScaleVal = combine(valSmooth, valOrganic, uOrganicMode); // Combine between Clean and Natural values primarily based on mode uniform
float baseDepth = 0.05;
float targetDepth = combine(1.0, aRandomDepth, uUseRandomDepth);
float currentDepth = baseDepth + (targetDepth - baseDepth) * finalScaleVal;
vec3 remodeled = place;
remodeled.z *= currentDepth;

Natural Mode On

The wave’s place and propagation are pushed by noise. The voxelization transforms with a bouncy, natural elasticity.

// vertex.glsl

// --- A. Natural Mode (Noise, Elastic) ---

// Add noise to the space to create an irregular wave entrance
float noisyDist = max(0.0, distFromCenter + (rndPhase - 0.5) * 0.3);

// Calculate native progress primarily based on voxelization progress and unfold
float progressOrganic = clamp((uVoxelateProgress - noisyDist) / uEffectSpread, 0.0, 1.0);

// Apply elastic easing to the progress to get the natural form worth
float valOrganic = elasticOut(progressOrganic, 0.2 + rndPhase * 0.2, currentDamp);

Natural Mode Off (Clean Mode)

The wave propagates linearly throughout the grid in an orderly method. The voxelization progresses with a clear, sine-based bounce, synchronized completely with the enlargement.

// vertex.glsl

// --- B. Clean Mode (Linear, Sine)  ---

// Calculate easy progress utilizing a easy distance threshold
float progressSmooth = smoothstep(distFromCenter, distFromCenter + uEffectSpread, uVoxelateProgress);

// Calculate bounce utilizing a sine wave
float valSmoothBounce = sin(progressSmooth * PI) * step(progressSmooth, 0.99);

// Mix linear progress with sine bounce
float valSmooth = progressSmooth + (valSmoothBounce * uEffectDepth);

Inside every mode, quite a few parameters – RGB shift, wave coloration, width, and deformation depth – work collectively in concord. Be at liberty to mess around with these settings below the “VISUAL” part within the GUI. I feel you’ll uncover some actually fascinating and enjoyable Pixel-to-Voxel transformations!

3. Let’s get physics, physics. I wanna get physics.

Strategy: To date, we’ve centered on the transformation within the InstancedMeshes – from flat tiles to full cubes. Subsequent, we’ll let the pre-created inflexible our bodies come to life contained in the physics engine. The aim is that this: the second the shapes absolutely flip into cubes, the corresponding inflexible our bodies “get up” and begin responding to the physics engine’s gravity, inflicting them to drop and scatter naturally.

Simulating the Ripple Impact in JavaScript

The transformation from flat to full dice occurs completely within the vertex shader, so JavaScript has no environment friendly solution to detect precisely when this transformation happens for every occasion on the GPU. The vertex shader doesn’t inform us which particular InstancedMeshes have accomplished their dice transformation. Nonetheless, we’d like this timing data to know precisely when to “get up” the corresponding inflexible our bodies to allow them to reply to gravity and start their drop. To know precisely when every particular person InstancedMesh has completed changing into a dice, we simulate the identical ripple impact logic on the JavaScript facet that’s operating within the vertex shader.

// vertex.glsl

float noisyDist = max(0.0, distFromCenter + (rndPhase - 0.5) * 0.3);
float progressOrganic = clamp((uVoxelateProgress - noisyDist) / uEffectSpread, 0.0, 1.0);

float progressSmooth = smoothstep(distFromCenter, distFromCenter + uEffectSpread, uVoxelateProgress);

We simulate the development talked about above on the JavaScript facet.

// core/PhysicsWorld.js

checkAndActivateDrop(rippleCenter, currentProgress, aspectVecX, organicMode, unfold, spacing, rely, onDropCallback) {
  if (!this.dropFlags) return;
  const spreadThreshold = unfold * 0.98;
  for (let i = 0; i < rely; i++) {
    if (this.dropFlags[i] === 1) proceed;
    const u = this.cachedUVs[i * 2];
    const v = this.cachedUVs[i * 2 + 1];
    const noiseOffset = organicMode ? this.noiseOffsets[i] : 0;
    const dx = (u - rippleCenter.x) * aspectVecX;
    const dy = v - rippleCenter.y;
    const distSq = dx * dx + dy * dy;
    const threshold = currentProgress - spreadThreshold - noiseOffset;

    // Replicating vertex shader ripple logic for physics synchronization
    if (threshold > 0 && threshold * threshold > distSq) {
      this.dropFlags[i] = 1;
      const pressure = spacing * 2.0;
      this.activateBody(i, pressure);
      onDropCallback(i);
    }
  }
}

Progress – Unfold – Noise > Distance

How can a single JavaScript if assertion keep completely in sync with the shader? The key lies in the truth that each are on the lookout for the very same second: when voxelization completes (the worth reaches 1.0). Capabilities like clamp and smoothstep are merely used to clip the values so that they don’t exceed 1.0 for visible depth. When you strip away these visible “clamps”, the logic in JavaScript and the Shader is an identical.

JavaScript

Situation-Natural:

$currentProgress – spreadThreshold – noiseOffset > dist$

Situation-Clean:

$currentProgress – spreadThreshold > dist$

In “Natural” mode, we use this.noiseOffsets—pre-calculated with the computeNoiseOffset perform in utils/widespread.js– following the identical logic used within the vertex shader.

Natural ModeLogic:

$progressOrganic = (uVoxelateProgress – noiseOffset – dist) / uEffectSpread$

When progressOrganic reaches 1.0:

$(uVoxelateProgress – noiseOffset – dist) / uEffectSpread >= 1.0$

Rearranged:

$uVoxelateProgress – uEffectSpread – noiseOffset >= dist$

Clean ModeLogic:

$progressSmooth = smoothstep(dist, dist + uEffectSpread, uVoxelateProgress)$

When progressSmooth reaches 1.0:

Rearranged:

$uVoxelateProgress – uEffectSpread >= dist$

One last tip: The logic that decides which cubes to drop at present loops by each occasion. I’ve already optimized it fairly a bit, but when the dice rely grows a lot bigger, we’ll want extra mathematical optimizations—for instance, skipping calculations for grid cells exterior the present wave propagation space (rectangular bounds).

Let me hear inflexible physique discuss, physique discuss

Manipulating the inflexible our bodies can result in all kinds of enjoyable and numerous behaviors relying on the values you tweak. Since there are such a lot of parameters to play with, I believed it is likely to be overwhelming to indicate them unexpectedly. For this demo and tutorial, I’ve centered the GUI controls on simply “DropDelay” and “Dance” mode.

When dropping, the activateBody methodology applies a random preliminary velocity to every inflexible physique. This methodology is triggered primarily based on the “DropDelay” timing. For the “Dance” mode, the updateAllPhysicsParams methodology updates the inflexible our bodies utilizing the precise parameter values.

// core/PhysicsWorld.js

activateBody(index, initialForceScale = 1.0) {
  const rb = this.rigidBodies[index];
  if (!rb) return;
  rb.setBodyType(RAPIER.RigidBodyType.Dynamic);
  rb.setGravityScale(0.01);
  rb.setLinearDamping(1.2);
  rb.wakeUp();
  const pressure = initialForceScale;

　// Apply random preliminary velocity and rotation
  rb.setLinvel({
    x: (Math.random() - 0.5) * pressure, y: (Math.random() - 0.5) * pressure, z: (Math.random() - 0.5) * pressure
  }, true);
  rb.setAngvel({
    x: Math.random() - 0.5, y: Math.random() - 0.5, z: Math.random() - 0.5
  },true);
}

updateAllPhysicsParams(bodyDamping, bodyRestitution) {
  this.rigidBodies.forEach((rb) => {
    rb.setLinearDamping(bodyDamping);
    rb.setAngularDamping(bodyDamping);
    const collider = rb.collider(0);
    if (collider) collider.setRestitution(bodyRestitution);
    rb.wakeUp();
  });
}

To affix the dialog, you’ll be able to work together with the inflexible our bodies utilizing your mouse. These interactions are managed by controllers/InteractionController.js, which communicates with core/PhysicsWorld.js through devoted strategies that allow you to seize and transfer our bodies with Three.js’s Raycaster.

Restoring after the discuss

When the “BACK TO PLANE” button is triggered, startReverse in controllers/SequenceController.js briefly disables gravity for all inflexible our bodies. To revive the unique grid, I applied a staggered linear interpolation (Lerp/Slerp) that animates every physique again to its preliminary place and rotation saved throughout creation. This creates a easy, orderly transition from a chaotic state again to the flat airplane.

// controllers/SequenceController.js

startReverse() {
//...
  gsap.to(progressObj, {
    worth: 1 + staggerStrength,
    length: 1.0,
    onUpdate: () => {
      const globalT = progressObj.worth;
      cachedData.forEach((knowledge) => {
        // Calculate progress (t) with particular person stagger delay
        let t = Math.max(0, Math.min(1, globalT - knowledge.delay * staggerStrength));
        if (t <= 0 || (t >= 1 && globalT < 1)) return;

        // Interpolate place and rotation
        const pos = {
          x: knowledge.startPos.x + (knowledge.endPos.x - knowledge.startPos.x) * t,
          y: knowledge.startPos.y + (knowledge.endPos.y - knowledge.startPos.y) * t,
          z: knowledge.startPos.z + (knowledge.endPos.z - knowledge.startPos.z) * t
        };
        qTmp.copy(knowledge.startRot).slerp(knowledge.endRot, t);

        // Replace RAPIER RigidBody transformation
        knowledge.rb.setTranslation(pos, true);
        knowledge.rb.setRotation(qTmp, true);
      });
      // Sync InstancedMesh
      physics.syncMesh(pixelVoxelMesh);
    }
//...

Implementation Notes

Contact Cache (The Reminiscence of the Speak): You would possibly discover the second drop feels barely much less intense than the primary. That is possible resulting from Rapier’s “Contact Pairs” cache persisting inside the our bodies. For a superbly recent simulation, you may regenerate the inflexible our bodies throughout every restoration. I’ve saved it easy right here because the visible impression is minor.

Syncing with the Render Cycle: You may also discover that the physics step() is known as straight inside animate() in index.js, pushed by requestAnimationFrame. Whereas a Mounted Timestep with an accumulator is commonplace for consistency, I deliberately omitted it to synchronize with the monitor’s refresh price. Reasonably than settling for “one-size-fits-all” precision, I selected to harness the total refresh potential of the machine to attain a extra pure aesthetic move and fluidity. Consequently, this app is sort of CPU-intensive, and efficiency will fluctuate relying in your machine. If issues begin feeling heavy, I like to recommend lowering the “Column” worth within the GUI to lower the general occasion rely.

As for potential technical enhancements, the present setup will hit bottlenecks if the dice rely will increase considerably. I really feel the present rely strikes a great steadiness – letting us really feel the transition from pixel to voxel – however dealing with tens of 1000’s of cubes would require a distinct strategy.

Rendering Optimization: As a substitute of CPU-based matrix calculations, place and rotation knowledge will be saved in DataTextures to deal with computations in shaders. This represents a considerable enchancment and is a concrete subsequent step for additional optimization. Alternatively, using Three.js’s WebGPURenderer (with TSL) is one other path for optimization.

Physics Technique: Whereas we might transfer physics completely to the GPU – which excels at huge, easy simulations – it will possibly usually restrict the logic-based flexibility required for particular interactions. For a venture like this, utilizing Rapier on the CPU is the correct alternative, because it offers the right steadiness between high-performance physics and exact, direct management.

Wrap-Up

Utilizing these as a basis, attempt creating your individual authentic Pixel-Voxel-Drop variations! Begin easy – swap out the video for a distinct one, or exchange the feel with a nonetheless picture, for instance.

From there, experiment by altering the geometry or shaders: flip the voxels into icy bins, use Voronoi patterns to create a shattering glass impact, or make solely cubes of a particular coloration drop… Alternatively, you may even combine this impact into a web site’s hero part – which is precisely what I’m planning on doing subsequent. The concepts will unfold outward, identical to the ripple impact itself!

Ultimate Ideas

Lastly, I’d like to specific my gratitude to Codrops for reaching out to me this time. Codrops has at all times impressed me, so it’s actually an honor to be invited by them. Thanks a lot.

A particular large thank-you goes to Manoela. I imagine it was her single remark concerning the ripple impact that formed the path of this demo. I’ve even named this impact “Manoela’s Ripple.” Thanks for the fantastic trace and inspiration.

And in addition, concerning the video used on this demo: Shibuya Crossing Video from Pexels.com.

Watching the dynamic move of Shibuya Crossing, it completely captures the vibe of how inflexible our bodies collide and work together inside a physics engine. Simply as pixels and voxels are the basic items of a digital form, every particular person and every object exists as an unbiased entity, colliding and transferring in their very own method. On this gathering of people, there’s concord inside chaos, and chaos inside concord – that is the very world all of us dwell in.