Back to blog

I turned a free AI 3D model into photoreal video with neural rendering

A free AI-generated 3D model, a real-time Blender viewport, a dataset of AI images and a CycleGAN trained on a graphics card from 2016. The result is the video below: the same 3D scene, repainted frame by frame by a neural network that learned what a real version of this character should look like.

A screen recording of the Blender viewport. Everything inside the camera frame has been passed through the trained CycleGAN. The interface around it is untouched.

This started as a simple question. AI image tools can make a 3D character look like a photograph, but only one image at a time, and every image comes out a little different. 3D gives you the opposite: perfect consistency from every angle, but it rarely looks real unless you spend serious time and render power on it.

So what happens if you let the 3D model handle shape and consistency, and train a small neural network to handle the realism?

The short version

Step 1: a free 3D model, good enough to work with

The starting point is an AI-generated 3D model of an orc. It cost nothing, and it is honestly not bad: decent shapes, fur, leather, metal, tusks. In a real-time Blender viewport it looks like what it is, a semi-realistic 3D asset. Plastic skin, painted-looking leather, fur that reads as cards.

That gap between "3D" and "photo" is mostly not about shape. It is micro detail, subsurface light in skin, how worn metal catches light. Exactly the kind of thing neural networks are good at learning from examples.

Step 2: a dataset of AI-rendered images

A network can only learn "realistic" if you show it what realistic looks like for this exact character. I did not have photos of an orc, so I generated them.

First I rendered a few angles from Blender and asked ChatGPT to make each one hyperrealistic while keeping the same angle. The results looked great, but they were not exact. Measured against the renders, features had moved 10 to 20 pixels and the fur had grown. For training that matters, so I warped every ChatGPT image back onto its render using optical flow, which moves the large shapes into place while keeping the new detail.

A Blender render of the orc next to the same angle re-imagined as a hyperrealistic image by ChatGPT
Left: the real-time Blender render. Right: the same angle after ChatGPT made it hyperrealistic. Pairs like this became the anchor of the dataset.

A handful of stills is not enough to learn from, so I added two short AI-generated videos of the same character, one slow turn all the way around and one close orbit around the face. Split into frames, that gave about 290 realistic images, including angles the stills never covered, like the back.

Six frames from two AI-generated videos of the orc character, from front, side, back and close-up angles
Frames from the two AI videos. Neither the stills nor the videos line up with the 3D model frame for frame, which is why the choice of model matters.

Step 3: training the network

I did not go straight to CycleGAN. I set up several approaches and trained them against each other on the same data:

To compare them fairly, one camera angle with a ChatGPT version was kept out of training completely. Every model was scored on how close it got to that unseen image. I built a small dashboard to follow all the runs side by side.

Training dashboard comparing several neural network setups, with a table of runs, an error chart and side-by-side output images
The training dashboard. Each line is a different setup, scored on an angle no model was trained on. Below the chart, the latest output of every run next to the 3D render and the ChatGPT reference.

On the numbers, most runs ended up close to each other, with the VAE-GAN clearly behind. By eye, CycleGAN won clearly. The autoencoders played it safe and went soft and washed out, because blurry images score well on pixel error. CycleGAN produced actual texture: pores, worn leather, wet-looking metal, individual fur strands.

Lesson worth keeping: a pixel-based score rewards blur. For anything where realism is the goal, look at the images, not just the number.

What went wrong along the way

Plenty, and it is the useful part.

Step 4: applying it to the video

The final step is almost boring, which is how it should be. I recorded the Blender viewport while the orc turns, cut out the camera frame from every screen frame, ran it through the trained generator and pasted it back. The interface around it stays exactly as recorded.

Side by side: the real-time 3D render on the left, the neural render on the right. Same frames, same camera, same model.
Before and after: the real-time 3D orc render next to the photoreal CycleGAN version, front view
Before and after: the real-time 3D orc render next to the photoreal CycleGAN version, side view
Before and after: close-up of the real-time 3D orc face next to the photoreal CycleGAN version
Three stills from the video. The shapes are identical, because they come from the 3D model. What changes is skin, leather, metal and fur.

It is not perfect. Each frame is converted on its own, so fine detail like fur can shimmer slightly in motion. The network has learned this one character and would need new examples for a different one. And it inherits the look of the AI images it learned from, flaws included.

Why this matters beyond an orc

The interesting part is not the orc. It is the split of work.

The 3D model controls everything that has to be right: proportions, the camera, consistency from one angle to the next. That is exactly where pure AI image generation fails, and why I still argue for 3D renders over photography and over AI images when the result has to match reality. The neural layer only adds the last stretch of realism, and it does it on top of a cheap real-time render instead of hours of path tracing.

For product visualization, games, configurators and pre-visualization, that combination is where this is heading: fast, controllable 3D underneath, and a learned layer that makes it look shot instead of rendered. This was a small experiment on an old graphics card. The approach scales.

If you are working on 3D visualization or an AI pipeline and want to explore what a layer like this could do for your renders, get in touch.

Frequently asked questions

What is neural rendering?

Neural rendering means using a neural network as part of the rendering process. In this project the 3D scene is rendered normally in real time, and a trained network then repaints every frame so it looks like a photograph instead of a 3D render.

Why use CycleGAN instead of a normal image-to-image model?

CycleGAN can learn from two piles of images that do not match pixel for pixel: 3D renders on one side, realistic images on the other. It learns to translate between them while a cycle check keeps the shapes in place. That fits a project where the realistic images come from AI tools and never line up perfectly with the renders.

Do you need an expensive GPU to train it?

No. Everything in this post was trained on a GeForce GTX 1070 with 8 GB of memory, a graphics card from 2016. The final generator is small, about two million parameters, and trains at around five steps per second on that card.

Does it run in real time?

Not yet. Converting the 1,862 frames of the video took about six minutes on the GTX 1070, roughly 0.2 seconds per frame including reading and saving each image. A newer GPU and an optimised model would get much closer to real time, but this version is an offline pass over a real-time render.

Can this replace traditional 3D rendering?

No, and that is the point. The 3D model controls the shape, the camera and the consistency between angles. The neural layer only adds photographic realism on top. Without the 3D underneath you are back to AI images that change every time you generate them.

Want your 3D to look shot, not rendered?

I combine 3D visualization with custom AI pipelines: real-time 3D for control and consistency, and trained models for the last stretch of realism. Get in touch if you want to explore it for your products, characters or scenes.

Get in touch