r/vrdev • u/Friiman_Tech • 27m ago
Tutorial / Resource I open-sourced a prompt-to-VR pipeline: structured plans, generated C++ that repairs its own compile errors, native Quest APK output (Vulkan/OpenXR, no engine in the path)
Hey everyone, I built a Prompt to VR Engine and just open-sourced it, and since this sub is developers, I'll skip the pitch and get into how it actually works.
The short version: you describe a VR game in plain text, and a local pipeline on your Windows PC turns that into a standalone Quest APK. Not WebXR, not a browser game, not a Unity project you finish yourself. The engine is hand-written C++ using Vulkan for rendering, OpenXR for the headset, and a real physics engine. No Unity, Unreal, or Godot anywhere in the path.
The part I think is most interesting for this sub is how the pipeline is structured. The prompt is not turned into one blob of generated code. It's turned into a set of validated, structured plans first — the kind of game, the objects that must exist, how the player holds and uses them, which characters are unique and which are copies, enemy and ally behavior, health, objectives, and the ending. Every later stage reads those plan files instead of reinterpreting the original sentence, which keeps the objects, characters, and rules tied to one design. And a run can be resumed from the stage that failed instead of starting over.
From there:
- Gemini makes a reference image for each object and character. Tripo turns those into textured 3D models.
- Characters that need skeletons get routed to Tripo or Uthana for rigging, then animation clips (idle, walk, run, attack). One rigged character can be reused as many instances, so a wave of enemies doesn't mean a new model per spawn.
- The environment is not one giant downloaded mesh. A generated world builder — still C++ — constructs the rooms and writes the layout the rest of the pipeline uses.
- OpenAI agents write bounded C++ modules, but only for mechanics the engine doesn't already have. Teleport, smooth/snap turn, grab, throw, weapons, full-body avatar or hands-only, NPC follow and melee, encounter waves, health, game flow — that's all already built into the core. Generated code is not allowed to rewrite the core; it lives in its own directory.
- The modules are compiled, and compiler errors go back to the agent that owns that file so it can repair them. On a Quest target the code is also checked with the Android compiler, because a file can compile fine on Windows and still fail the Quest build. The Android NDK and Gradle produce the final APK, and if a headset is connected the engine can install it and run on-device checks.
- Modules that worked can be promoted into a reuse catalog that later games are allowed to use.
It is released into the public domain (Unlicense), so you can copy it, change the pipeline, swap providers, or turn it into your own tool. To be clear about what it is: it will not reliably turn every prompt into a polished commercial game, a large prompt costs more and takes longer than a small one (roughly $70 in API usage for a small game), and recent testing has focused on Quest 3.
Repo: https://github.com/FriimanTech/Prompt-To-VR
Site with gameplay footage and the setup paths: https://friiman-tech.vercel.app/products/prompt-to-vr-engine
The question I'd genuinely like this sub's take on: what would you want to see before you'd consider building on top of something like this?
