Back to blog

I trained a 1 MB neural network to make my Gaussian splat look like the real photos, live in your browser

A Gaussian splat is a little soft and a little flat next to the photos it was made from. So I used the photos as the answer key and trained a 1 MB neural network to close the gap. It runs on your graphics card, on every frame, inside this page. Drag the divider and see.

Full screen
Live and interactive. Walk with W, A, S and D, drag to look around, drag the divider to compare the plain splat (left) with the AI-enhanced view (right), press E to switch the AI layer off. Needs a desktop browser with WebGPU, such as Chrome or Edge. Open it full screen for the best experience.

What you are looking at

I filmed a ruin on my phone, turned the video into a Gaussian splat and built a walk through it. That part is the same recipe as my Eremitageslottet walk.

The new part is the right half of the picture. A small neural network looks at what the splat renderer draws, on every frame, and pushes it towards what the real camera saw: crisper mortar lines, stronger contrast, colours closer to the footage.

The same view of a brick ruin with a doorway, split down the middle: the plain Gaussian splat on the left and the AI-enhanced version on the right, which has sharper brick texture and stronger contrast
The same view, split in the middle. Plain splat on the left, after the network on the right. Look at the brickwork and the dark patches of moss.

The short version

The idea: every video frame is an answer key

A splat is trained to match its photos, but it never matches them perfectly, and the web viewer adds its own softness on top with the way it sorts and blends the splats. The gap between what the viewer draws and what the camera saw is a pattern. A network can learn a pattern.

And I already have both sides of every example, because I know where the camera was for every frame. Render the splat from that exact position and you have the input. The real frame is the target.

Step 1: render the splat the way the browser does

I rendered all 717 views in a headless browser using the same viewer code that runs on this page. That matters. The network should learn the look of the web viewer, not the look of the trainer's own preview.

Step 2: line the pairs up

If the input and the target are off by a few pixels, the network learns to blur. So I measured the leftover shift between every render and its real frame with phase correlation. The median was 0.08 pixels and 95% were under 0.3. The camera alignment from RealityScan was that good.

Then I threw away the frames that make a bad answer key: the ones with the most motion blur, and any pair where the match looked unreliable. That left 609 pairs. Every tenth frame was held back for testing and never trained on.

Three upright views of the same brick archway: the splat rendered in the browser, the neural network output, and the real video frame it is trying to match
A held-out frame the network never trained on. Left: the splat rendered in the browser. Middle: what the network makes of it. Right: the real video frame it is trying to match. The video is stored sideways, so I turned the frames upright here.

Step 3: train on patches

609 whole frames is not much, and they are big. So the network trains on random 256 by 256 patches, flipped and rotated, which gives it thousands of different crops to learn from.

It predicts a correction that is added on top of the render. It can nudge the picture, never redraw it. The loss is the pixel difference on a slightly blurred copy, which forgives the last bit of misalignment, plus a structure term that rewards sharp edges and contrast.

Step 4: make it fast enough for a web page

My first network had 1.5 million parameters and scored 21.6 dB. It ran at about 5 frames per second on my GTX 1070 at this resolution, which is no good for walking around. Two changes fixed that.

Together that is about 45 frames per second, measured while another training run was still using the same card.

A close-up of a brick archway, plain Gaussian splat on the left and AI-enhanced on the right, the right with crisper mortar lines and richer brick colour
Close-up of an archway. Plain splat on the left, enhanced on the right. The mortar lines are sharper and the brick colour is closer to the footage. It is a clear improvement and not a photograph.

What is not perfect

A badly smeared area of the splat on the left, and on the right the same area after the network, with a large bright white blob where the smear was
Where the splat is a mess, the network cannot rescue it, and sometimes it makes things worse. Here it turned a smeared patch into a bright blob.

The gain is about two decibels, a clear but moderate improvement. It makes a splat look more like a photo, it does not make a splat into a photo.

Why this matters

Any splat built from photos or video with known cameras gives you this paired dataset for free. That makes a cheap finishing step: a tiny network, trained per scene in about an hour, shipped as a 1 MB file next to the 3D scene.

I tried the sharpening version of this idea on an orc in a tiny neural network that sharpens a splat in real time, trained on a handful of reference images. This one learns the look from a whole walk of real footage, and for a space like this it is the more natural fit. For a shop, a property or a heritage site it means the 3D can look closer to the real thing without a heavier splat. That is the kind of 3D visualization work I find most interesting right now.

Frequently asked questions

What does the neural network do to the splat?

It looks at what the splat renderer draws, on every frame, and adds a correction that moves the picture towards what the real camera saw. In practice that means crisper mortar lines, stronger contrast and colours closer to the original footage. It is a correction on top of the render, so it can only nudge the picture, not redraw it.

Do I need a special browser?

The AI layer needs WebGPU, which today means Chrome or Edge on a desktop computer with a reasonably recent graphics card. Without WebGPU the page still works and shows the plain splat. I have only tested it on a desktop with a GeForce GTX 1070 from 2016, not on phones.

Could this work on my own scene?

Yes, if the scene was captured with known camera positions, for example from a photo set or video that has been through RealityScan or similar. Those cameras give you a paired dataset for free: the splat rendered from each camera, and the real photo from that camera. The network has to be trained per scene, which took about an hour on a GTX 1070 here.

Why not a GAN or a diffusion model?

A GAN or diffusion model invents more dramatic detail, but it invents different detail on every frame, which shows up as flicker when the camera moves. A plain regression network is a smooth function of its input, so the picture stays calm while you walk. It is less impressive in stills and much better to look at in motion.

How big is the download?

The 3D scene is 38 MB and the neural network is 1 MB. The scene loads in a few seconds on Wi-Fi and takes longer on mobile data.

Want a place or product people can walk around online?

I build 3D visualization and interactive web 3D, from classic renders to Gaussian splats made from your own photos or video. Get in touch if you want to see what your space, building or product could look like.

Get in touch