Relighting Pictures with Depth Maps and Three.js

Editor’s notice: And we’re persevering with our celebration of the Three.js group as we depend right down to the primary Three.js Convention in Paris this September. On this new tutorial, Dominik Fojcik shares just a little magic trick for making 2D photographs come alive with depth, mild, and shadow, utilizing Three.js, TSL, and WebGPU.

I’ve seen many cool picture results on the internet, however most of them keep on the floor. What in case you might take it additional and make the impact dive into the picture? One thing that truly will get inside the image like a light-weight.

The important thing for that may be a depth map. One thing that earlier than appeared to be magic is now doable due to depth estimation fashions, which have develop into actually good at estimating depth from 2D photographs.

Depth Map

That is the principle ingredient of our impact. To get one, we have to feed a depth estimation mannequin with our picture and use it to generate a depth map. You possibly can set up a mannequin like Depth Something 3 your self or use a Depth Technology Device I created for this text.

The uncooked depth map has an issue you possibly can’t see, however the mild can. It’s an 8-bit picture, so it solely has 256 attainable depth values, and clean surfaces get quantized into flat steps. The lighting reads the slope from this map, and at each step edge the slope immediately spikes, so shading that ought to circulation throughout clean stone breaks up into gritty, blotchy noise. To repair it, we convert the depth map to floats and blur it to clean out these steps.

First, we convert the 8-bit depth values into floating-point values, blur them to take away the seen steps, after which retailer the end result as half-float knowledge so we are able to protect smoother depth data.

const { knowledge } = context.getImageData(0, 0, width, peak)
const values = new Float32Array(width * peak)
for (let i = 0; i < values.size; i++) {
  values[i] = knowledge[i * 4] / 255
}

// blur simply sufficient to soften the 8-bit steps collectively
smoothBands({ values, width, peak }, radius)

const halfFloats = new Uint16Array(values.size)
for (let i = 0; i < halfFloats.size; i++) {
  halfFloats[i] = DataUtils.toHalfFloat(values[i])
}

Faking the Floor with a Regular Map

How will we make mild act like our picture is 3D? The reply is a regular map.

Lighting doesn’t really care in regards to the form itself, it cares about normals: the course every level on a floor faces. A degree dealing with the sunshine is shiny, a degree tilted away is darkish. That’s your complete trick of this impact.

So as a substitute of constructing actual geometry, we can provide every pixel a standard that makes the aircraft seem three-dimensional to the sunshine.

To create the traditional map, we’re going to make use of our depth map. The depthGradient operate samples the depth map texture and calculates the floor slope that we are able to use to create our regular map.

const depthGradient = Fn(([vUv, step]) => {
  const left = smoothDepthNode.pattern(vUv.sub(alongX)).r
  const proper = smoothDepthNode.pattern(vUv.add(alongX)).r
  const backside = smoothDepthNode.pattern(vUv.sub(alongY)).r
  const prime = smoothDepthNode.pattern(vUv.add(alongY)).r

  return vec2(proper.sub(left), prime.sub(backside)).mul(0.5)
})

When visualized as colours, the traditional map appears to be like like this:

So as to add much more element, I run the identical trick on the picture itself, utilizing its brightness as a substitute of depth. It’s a cheat—a darkish painted stripe tilts the traditional in the identical manner an actual groove would—however underneath a shifting mild it reads as floor element. The 2 gradients are merely added collectively:

const form = vec3(slope.x.negate(), slope.y.negate(), float(1))
return form.add(vec3(element.x.negate(), element.y.negate(), 0)).normalize()

Right here is the traditional earlier than and after including the small print.

Shadows

Normals make the picture react to mild like a 3D floor, however they will’t make one half forged a shadow on one other. To seek out shadows, every pixel traces a line towards the sunshine via the depth map. If it detects a bump alongside the way in which, the pixel is in shadow.

The depth map is sampled at a number of factors alongside this path, and the quantity of occlusion is collected to create a delicate shadow.

const occlusion = float(0).toVar()

Loop(SHADOW_STEPS, ({ i }) => {
  const journey = float(i).add(1).div(SHADOW_STEPS)
  const rayDepth = surfaceDepth.add(headroom.mul(journey))
  const blockerDepth = smoothDepthNode.pattern(vUv.add(sweep.mul(journey))).r
  const softness = uShadowSoftness.mul(journey.mul(SOFTNESS_GROWTH).add(1))
  const blocked = blockerDepth.sub(rayDepth).div(softness).clamp(0, 1)

  occlusion.assign(occlusion.max(blocked))
})

Materials

All three elements are handed to MeshPhongNodeMaterial, which handles the precise lighting calculation for us:

const materials = new MeshPhongNodeMaterial({ specular: 0x000000 })
materials.colorNode = diffuseNode(vUv, depth)   // our picture
materials.normalNode = normalNode(vUv)          // faux normals from depth-map
materials.aoNode = shadowNode(vUv, depth)       // shadows

Right here, the picture turns into the fabric’s coloration, our generated normals management how the floor reacts to mild, and the depth-based shadows are added as ambient occlusion.

Closing Phrases

I hope this little magic trick offers you some inspiration to create your personal fascinating results with depth maps. There may be nonetheless a variety of unexplored potential on this area.

Wishing you all one of the best on the upcoming Three.js convention! I couldn’t make it this yr, however hopefully subsequent time!