Breaking the Body: Constructing a Actual-Time Datamosh Impact with Three.js

Editor’s Be aware: As our Three.js Convention celebration continues, we’re particularly excited to have Niccolò Fanton becoming a member of us right this moment with a mesmerizing exploration of datamoshing. Drawing from video codecs and shader methods, he exhibits how a couple of rigorously damaged guidelines can flip a Three.js scene into one thing splendidly unpredictable.

A video codec periodically encodes an intra image that may be reconstructed with out referring to an earlier image. Between these factors, inter-coded footage are predicted from beforehand reconstructed reference footage. On the block degree, an encoder might choose a displacement right into a reference image and encode a residual correction, or it might encode the block utilizing intra prediction when reference prediction will not be helpful. As a result of consecutive footage usually resemble each other, prediction can save an excessive amount of knowledge.

Suppress the intra refresh at a scene change, and the decoder continues reconstructing from references that also belong to the outgoing shot whereas processing knowledge supposed for the incoming one. Pixels from the outdated scene are dragged round till a later refresh restores a clear picture.

Video artists have lengthy used variants of this trick on actual information, utilizing instruments equivalent to hex editors and modified decoders. Right here, I borrow the concept in actual time: a fraction shader acts because the decoder for a Three.js scene and, by default, suppresses refresh at some stage in the gesture.

Step 1: The decoder in a single line

The entire implementation follows one concept:

new body = warp(earlier body, movement vectors) + residual

The earlier body means the decoder’s final reconstructed output, not the scene renderer’s present output. On this undertaking, the movement texture shops graphics-derived screen-space velocity: an approximation of the displacement discipline a video encoder would possibly select, not the codec’s authentic movement vectors or a measurement of bodily floor movement.

Step 2: The smallest datamosh that works

We begin with the suggestions loop, the half that makes every broken body grow to be the supply for the following. For now, a placeholder provides the movement vector.

The important thing change is the place the shader reads its image. A post-processing cross often samples the body the renderer has simply drawn. This cross as an alternative samples its personal earlier output at an offset and mixes in a small quantity of the contemporary render. With a continuing equivalent to vec2(0.002, 0.0), the historical past drifts sideways whereas the incoming scene frequently replenishes it, producing a persistent trailing smear somewhat than an endlessly translated, unchanged picture.

uniform sampler2D uScene;    // this body, as rendered
uniform sampler2D uHistory;  // what we output final body
uniform vec2 uMotion;        // one fixed vector for the entire display screen

in vec2 vUv;
out vec4 fragColour;

void essential() {
  vec3 scene   = texture(uScene, vUv).rgb;
  vec3 dragged = texture(uHistory, vUv - uMotion).rgb;
  // A small fixed trickle of the contemporary render retains the loop steady;
  // with the vector at zero the suggestions settles again onto the scene.
  fragColour = vec4(combine(dragged, scene, 0.12), 1.0);
}

WebGL imposes one constraint on this loop: a shader can’t pattern a texture hooked up to the framebuffer it’s at the moment drawing into. Updating one historical past buffer in place is due to this fact not possible. We preserve two buffers, learn from one, write to the opposite, and swap them on the finish of every body.

A 3rd goal holds the rendered scene for the decode cross to pattern. The canvas receives a duplicate of the historical past buffer we simply wrote. In pseudocode, the association appears to be like like this:

// Learn one goal, write the opposite, then swap: the reference body is
// all the time a texture nothing is at the moment drawing into.
cross.uniforms.uScene.worth   = color.texture;
cross.uniforms.uHistory.worth = historical past.learn.texture;

cross.render(historical past.write);   // decode into the spare goal...
historical past.swap();               // ...which turns into subsequent body's reference
blit(historical past.learn, null);     // and in addition what reaches the canvas

The completed undertaking wants just one express suggestions buffer as a result of postprocessing already swaps an inside pair. The decode writes ahead, then a duplicate cross shops the end result within the suggestions goal. The diagram beneath exhibits that association, together with the movement goal launched by the speed cross.

That backside loop creates the impact. Every body inherits the earlier body’s injury, so the distortion retains constructing for so long as the vector retains pointing someplace.

Step 3: The speed cross

A continuing vector appears to be like synthetic as a result of it assigns the identical displacement to each pixel. The renderer can derive how seen geometry moved in display screen house, so we use that velocity discipline as a codec-inspired approximation.

We draw the scene a second time with an override materials that writes velocity as an alternative of color. Every vertex is projected as soon as with the present mannequin and digital camera matrices, then once more with the matrices saved for a similar object one body earlier. The distinction between these projected positions provides the displacement utilized by the impact.

uniform mat4 uPreviousModelMatrix; uniform mat4 uPreviousViewProjection; uniform float uHasPrevious;
various vec4 vClipCurrent, vClipPrevious;

void essential() {
  vClipCurrent = projectionMatrix * modelViewMatrix * vec4(place, 1.0);
  // And not using a saved earlier matrix (first body, object simply added) the
  // earlier place is the present one: precisely zero velocity.
  vClipPrevious = combine(vClipCurrent,
    uPreviousViewProjection * uPreviousModelMatrix * vec4(place, 1.0), uHasPrevious);

  gl_Position = vClipCurrent;
}

One element is simple to get incorrect. Each clip positions should attain the fragment shader undivided; the angle divide belongs there, per fragment. Doing it within the vertex shader seems cheaper and equal, but produces incorrect values. A display screen place is a ratio, and interpolating that ratio differs from dividing two interpolated portions.

various vec4 vClipCurrent, vClipPrevious;

void essential() {
  // The divide occurs right here, per fragment. By no means within the vertex shader.
  vec2 velocity = vClipPrevious.w <= 0.0 ? vec2(0.0)
    : (vClipCurrent.xy / vClipCurrent.w - vClipPrevious.xy / vClipPrevious.w) * 0.5;

  gl_FragColor = vec4(clamp(velocity, -0.25, 0.25), 0.0, 1.0);  // alpha 1: geometry right here
}

Step 4: How you’re taking the keyframe away

Thus far, the pipeline behaves like an elaborate movement blur. About 4 traces of scheduling flip it right into a datamosh.

Urgent the pointer, Area key, or contact enter cuts to a shot with new geometry, palette, and digital camera. The decoder receives no notification. It retains studying the final output from the outgoing shot whereas making use of vectors measured from the incoming one. That mismatch produces the smear.

The ordering issues: operating the minimize after composer.render buys one deliberate body of latency. The outgoing shot sits within the suggestions goal when the incoming one is first rendered in opposition to it. And capturing the previous-frame state instantly after the minimize means the digital camera leap itself by no means enters the speed measurement.

const gestureStart = lively && !this.wasActive;

// A gesture by no means raises the keyframe. Solely the periodic refresh does.
this.impact.set('uKeyframe', gopRefresh ? 1 : 0);

this.composer.render(deltaTime);

// The minimize occurs after the body is on display screen, so the suggestions goal nonetheless
// holds the outgoing shot when the incoming one is first rendered.
if ((gestureStart || gopRefresh) && controls.sceneCut && this.onCut()) {
  this.digital camera.updateMatrixWorld(true);
}

// Retailer the post-cut state. The digital camera leap itself must not ever grow to be a vector.
this.velocityPass.capturePreviousState();
this.impact.capturePreviousState();

The refresh additionally raises uKeyframe, handing the decoder a clear body.

Releasing begins a brief restoration. Over 370 ms by default, a counter ramps from 0 to 1 and scales the movement and residual collectively. The smear slows to a cease, then dissolves into the clear render.

Step 5: Placing the element again

An actual encoder computes its residual in opposition to a motion-compensated prediction created from reconstructed reference footage. If I used the whole distinction between the present render and this intentionally incorrect prediction, the residual would cancel the mismatch and restore the datamosh. The correction due to this fact has to stay intentionally incomplete.

The model used right here compares the high-frequency element within the incoming render with the element already current within the prediction produced by warping the earlier body. It transmits solely the portion that’s stronger within the incoming render. This gate makes the correction self-limiting: as quickly because the prediction carries an edge strongly sufficient, the residual falls to zero as an alternative of including the identical contour to the suggestions loop once more.

// Excessive frequencies of the sincere render, and of the prediction we simply made.
float hc = dot(present - lowCurrent * 0.25, vec3(0.299, 0.587, 0.114));
float hp = dot(predicted - lowPredicted * 0.25, vec3(0.299, 0.587, 0.114));

// As soon as the prediction carries the stronger edge, the gate returns zero.
// The quantiser follows: beneath half a step there's nothing to ship,
// so flat areas transmit nothing in any respect.
float steps = max(uResidualQuant, 1.0);
return vec3(flooring((abs(hc) > abs(hp) ? hc - hp : 0.0) * steps + 0.5) / steps);

The residual carries a luminance-like brightness sign with out color. An RGB residual deposits the brand new shot’s palette onto the outdated image and makes the held body seem semi-transparent.

Step 6: Slowing the suggestions blur

Held for a couple of seconds, the picture begins to melt. Every movement vector often factors between texture pixels, so the GPU blends neighbouring values. That mix is innocent as soon as, however this impact feeds the end result again into the following body. Repeating it as soon as per rendered body, sixty occasions a second at 60 fps, regularly removes distinction and element.

Catmull-Rom reconstruction slows that loss. It estimates a sharper worth from the encompassing pixels, so edges survive longer than they do with peculiar bilinear interpolation. The loop nonetheless resamples the picture on each body, which suggests some softness stays inevitable. We additionally clamp the end result to the legitimate color vary as a result of even a small overshoot can be amplified by the suggestions.

Step 7: Making it look compressed

The pipeline now produces a clean, liquid smear. It will possibly look lovely, although damaged video has a special visible grammar: rectangles. Codecs course of the picture in blocks, and the impact wants to show that construction.

4 mechanisms add that compressed vibe, all managed by uBlockiness.

Blocks. The shader reads one vector on the centre of every tile and applies it throughout the tile. With the default 8px block, sixty-four pixels share a vector.

Precision. Many codecs characterize luma movement at subpixel precision. Snapping to that lattice turns a slide right into a step. With out it, neighbouring blocks can differ by tiny quantities that learn as a gradient. As soon as snapped, they match or diverge by a full increment, creating arduous tears between inflexible slabs. I exploit “block” right here in a visible sense: the default 8×8 tile is an inventive selection, not a declare about one codec’s fastened macroblock measurement.

Skip. Beneath a threshold, a block’s movement fades to zero, so its historical past pattern stops being displaced. The residual stays lively, making this a codec-inspired maintain somewhat than a literal skipped-block implementation. A brief ramp eases the movement into that state to forestall a visual pop.

// 1. One vector for the entire tile, learn at its centre. warp is the discharge
//    fade, and mvUV is the block centre at full blockiness.
vec2 mvUV = combine(uv, blockUV, uBlockiness);
vec2 movement = rawMotionAt(mvUV) * uMotionGain * warp;

// 2. Snapped to a half- or quarter-pixel lattice ("Vector Precision"), so
//    neighbours both match or differ by a complete step.
float mvSteps = max(uMvPrecision, 1.0);
vec2 motionPx = movement * decision;
movement = combine(motionPx, flooring(motionPx * mvSteps + 0.5) / mvSteps, uBlockiness) / decision;

// 3. Below the brink, movement fades to zero and the historical past pattern holds.
//    The residual path stays lively later within the shader. The guard is
//    load-bearing: smoothstep with equal edges is undefined in GLSL.
float skip = uSkipThreshold <= 0.0 ? 0.0
    : (1.0 - smoothstep(uSkipThreshold * 0.5, uSkipThreshold, size(movement * decision))) * uBlockiness;
movement *= 1.0 - skip;  // the total shader additionally folds within the lost-sector masks

The incorrect vector. The fourth mechanism hides within the snippet’s first line. Within the full supply, a share of block centres in mvUV shift diagonally and pattern a neighbour’s vector. This mimics a block match locking onto the incorrect object.

A word on methodology: none of those constants is derived. The block measurement, threshold and mismatch charge got here from shifting sliders till the end result regarded like a corrupted file.

Step 8: Packet loss

The final layer provides the display screen’s most seen artefact: rectangles whose movement freezes earlier than they reappear elsewhere. In an actual stream, an analogous visible failure can comply with transport loss, when knowledge wanted to reconstruct an space by no means arrives and the injury persists till an intra refresh repairs it. What follows is a loss-inspired sector masks somewhat than a simulation of that mechanism: it suppresses movement in chosen rectangles whereas the residual path can nonetheless introduce element.

The mannequin makes use of a number of overlapping grids, every finer than the final. Each cell runs on a clock offset by a hash of its coordinates and layer index. That retains the part steady from one time slot to the following. On every tick, the cell decides whether or not to drop a block, the place to put the rectangle, and the way lengthy to carry it.

for (float fi = 0.0; fi < 4.0; fi++) {
  if (fi >= uLostLayers) break;  // fastened higher sure retains execution predictable throughout WebGL drivers

  vec2 scale = vec2(7.0, 5.0) * (1.0 + fi * 1.6) / max(uLostScale, 0.05);
  vec2 cellId = flooring(uv * scale);

  // Each cell by itself clock, phase-offset by a hash of the cell. The
  // max() flooring that clock at about one body.
  float t = uTime / max(uLostLife, 16.0) + hash13(vec3(cellId, fi + 5.0));
  float slot = flooring(t);
  if (hash13(vec3(cellId, slot * 7.0 + fi)) > uFrozenBlocks) proceed;

  // ...a rectangle of hashed measurement and place for this slot, plus an obligation
  // cycle so it blinks out once more earlier than the slot is over.
}

The lost-sector masks keep disabled for the primary 300 ms of the impact. The delay lets the image open with a clear smear earlier than packet loss lands on high of it, nearer to the rhythm of a stream that fails after decoding has begun.

Step 9: What it prices

Whereas the impact runs, the scene is drawn twice as a result of the speed cross wants its personal rasterisation of the geometry. As soon as the gesture and restoration fade finish, each the speed and decode passes swap off when the diagnostic overlays are additionally disabled. Their value is in any other case restricted to the interplay.

At relaxation, the pipeline runs FXAA and two copy passes. FXAA cleans up jagged edges earlier than they enter the suggestions loop; Catmull-Rom serves a special objective, preserving element whereas the saved picture is moved and sampled repeatedly.

With the gesture held, the scene is drawn as soon as for color and as soon as for velocity, adopted by 4 full-screen passes: FXAA, decode, suggestions copy and the ultimate blit. Decode is predicted to be the most costly full-screen cross.

The primary optimisation goal I might measure is the movement discipline. A macroblock makes use of one vector sampled at its centre, but the shader recomputes that worth for each fragment within the block. On the default eight-pixel measurement, the identical movement calculation runs sixty-four occasions for one reply. A block-resolution prepass would get rid of that work.

Afterword

This started with a small concept: a transition between two Three.js scenes that felt like a corrupted file. I anticipated a day of shader work. As an alternative, I saved discovering items of an actual codec value reproducing. Every mechanism made the end result really feel extra appropriate, or no less than extra convincingly damaged.

The hash features come from Dave Hoskins’ Hash with out Sine. The historical past cross makes use of Catmull-Rom reconstruction to gradual the lack of element by means of repeated suggestions. The arrow overlay within the motion-field views follows Maxime Heckel’s Shading Movement.