Back to blog

I can relight a video in the browser: delight it, then drag my own lights over it

A few days ago I removed baked lighting from a 3D model. Now I have done it to a video, and then put new lights in. One page takes a clip, strips the original light from every frame, works out the shape of what is in the picture, and lets me drag orange, blue and pink lights around while the shading follows. The video shows it live.

Screen recording of the tool. The lighting of the orc clip has been removed, and I add orange, blue and pink lights, then move them with the mouse and the X, Y and Z sliders. Nothing is pre-rendered: the shading is recalculated as I drag.

Relighting is one of those effects that sounds like a plug-in and turns out to be three hard problems in a row. You need the original light gone, otherwise your new lights fight with it. You need to know which way every surface faces, otherwise there is nothing to shade. And you need a way to place lights that feels like holding a torch, not editing numbers.

I already had the first part: the delighting network from the Blender add-on. This post is about the other two, and about gluing all of it into one page.

The short version

How a frame travels through it

The processing runs once per clip, one frame at a time, with a progress bar. After that the lighting is cheap: the frames are sitting in the graphics card as textures, and moving a light just re-runs the shader.

The relight tool showing an orc bust lit by a warm orange light from the right, with the light controls open on the side
A warm light from the right. Each light has colour, intensity, falloff, X and Y across the picture, and Z for how far in front of it the light sits.

The part I got wrong first

My first version took the normals from the depth map: look at how depth changes between neighbouring pixels and you get a slope. It is the obvious method and it looked bad. Skin turned crinkly, because small noise in the depth became bumps. Shadows came out in visible steps. And wherever the depth jumped, like the edge of a shoulder against the background, the normal pointed away from every light and left a black outline.

Depth models are trained to get the ordering of things right, not the exact slope of a surface. Normals are a different quantity and deserve their own model. Swapping in MoGe-2 was the single biggest improvement. The face and shoulders now have smooth, believable shape, and the depth map only adds fine detail on top, with a slider to decide how much.

If a derived quantity looks wrong, check whether you are asking the model for something it was never trained to give. Differentiating a depth map for normals is exactly that.

The orc bust lit with a pink light from the left and a blue light from behind in the relight tool
A pink light from the left. Colour, direction and falloff each respond as the lights move, and the shadow side of the face keeps its form.

What still does not work

Why this matters

Lighting is the thing that makes footage and renders look expensive, and it is also the thing you cannot change after the fact. Being able to pull the original light out and place your own, on footage that is already shot, is a real shortcut for product visuals, character work and creative direction. It is the same idea as my neural rendering and splat sharpening experiments: keep the structure under control and let a trained network do the part that is painful by hand. The orc in the demo comes from the Gaussian splat I made from an AI-generated model.

The tool is a prototype on my machine and not released. If you have footage you want relit, or a pipeline where a tool like this would save time, get in touch. Custom AI tools and pipelines are what I build.

Frequently asked questions

What does it mean to relight a video?

Relighting means changing the lighting of footage after it was shot. The original light is removed first (delighting), then new lights are placed and the shading is recalculated from the shape of the surfaces, so you can move, recolour or add lights.

Does the video leave my computer?

No. The tool runs in the browser on my own machine and the video is never uploaded. The only download is the model weights themselves, which are fetched once if they are not already on disk.

Where do the normals and the depth come from?

Both are estimated from the single image of each frame by neural networks. A model called MoGe-2 predicts the surface normals, and Depth Anything V2 predicts depth, which is used for shadows and fine surface detail. No 3D model of the scene is needed.

Can I move the camera after relighting?

No. The result is a relit version of the original camera view. It is a 2.5D effect, because the tool knows each pixel's depth and direction but does not rebuild the hidden sides of the object.

Is the tool available to use?

Not yet. It is a working prototype on my machine. If you have footage you want relit, or a pipeline where a tool like this would help, get in touch.

Want footage or assets relit in your own style?

I combine 3D production with custom AI tools: delighting, normal and depth estimation, and real-time relighting. Get in touch if you want to explore it for your own material.

Get in touch