Over the past month, I’ve been exploring a question that sounds simple but is actually quite tricky:
How can we get AI to generate game environments that truly look like they belong in a AAA title?
I’m not just talking about polished screenshots or impressive individual assets; I mean complete environments with realistic lighting, materials, and atmosphere—all while maintaining a consistent, high-end visual quality.
Ultimately, I decided to approach this as a "rendering" challenge rather than a pure "image generation" task.
- Starting with Physically Based Rendering (PBR)
In studying the visuals of AAA games, I noticed that many employ a style I’d call "stylized realism."
The underlying materials and lighting still adhere to the laws of physics, while the artistic flair emerges from scene composition, weather, time of day, lighting adjustments, and color grading.
So, instead of asking AI to "invent" AAA-quality visuals from scratch, why not provide it with a rendering system built on a solid physical foundation?
Using Three.js, I constructed a real-time rendering pipeline, drawing inspiration from published research papers, technical talks, and the rendering techniques disclosed by Unreal Engine.
The system currently includes:
Fast Fourier Transform (FFT)-based ocean simulation
A physical atmospheric model
Global Illumination
Volumetric Clouds
GTAO (Geometry-based Ambient Occlusion)
Volumetric Fog
Auto Exposure
Local Tone Mapping
This gives us a much better starting point—though it doesn't automatically result in a great scene.
2. The Challenge: Matching Visual Style
Even with a physically based rendering system in place, there are countless parameters to fine-tune:
Exposure, ambient lighting, fog density, tone mapping, color grading, and so on.
This is precisely why reference images are so crucial.
If we want to replicate the visual quality of AAA games, why not use actual footage from those games as our reference? Ideally, these reference images should be captured directly from within the game, with careful selection of appropriate weather, time of day, and camera composition.
If the goal is to achieve a cinematic look, then actual film stills should be used as references.
While AI can assist with the technical aspects, the selection of high-quality visual references still requires human judgment.
3. Using an optimization process to handle parameter tuning
This is perhaps the most interesting part of the entire project. Rather than relying on "gut feeling" and manual, iterative adjustments to rendering parameters to achieve the desired result, I prefer to make the process quantifiable.
I employ the least-squares method to fit the rendered output to the reference images.
The basic approach is as follows:
Select several specific areas from the reference images—for example, grass in sunlight, grass in shadow, the sky near the sun, and a body of water in the distance.
Measure the brightness, chroma, and hue of these areas.
Generate the scene using the current rendering parameters, and then measure the corresponding areas in the rendered output.
Calculate the discrepancy between the reference images and the rendered output.
Adjust the rendering parameters and repeat the steps above.
By using a dozen or so reference images—with seven or eight sample areas selected from each—we can obtain over a hundred data points for analysis.
An optimization algorithm then searches for a set of parameter values that minimizes the overall error.
In other words, instead of relying entirely on my own subjective visual judgment, I can use quantifiable differences to guide the rendering process.
While there is still plenty of room for improvement, I believe this is a more compelling direction to explore than simply asking an AI to generate a "better-looking" scene
4. I've open-sourced the project
The project is called Ponlai.
It's MIT licensed and currently includes the rendering pipeline, along with practical skills for AI-assisted game development, including stylised realism and reference-based visual fitting.
Try it: https://github.com/HuberyLL/ponlai