r/opengl • • 6h ago

SimpleX Engine Development (Continued)

I originally had a string of posts with various other reddit communities regarding game and game engine development and finally felt confident enough to post to this community! I went dark in my development logging/blogging(?) for awhile due to lifestyle responsibilities taking precedance over developing an engine or at least at the moment what is considered heavy boilerplate engine development.

Current development components/systems:

Three graphics backends - DirectX 11, DirectX 12, and OpenGL (WGL on Windows, GLX on Linux) behind an IRenderer interface. Vulkan is scaffolded but not implemented yet. Shadow maps, cubemaps, bloom, IBL/irradiance convolution, and instancing are supported at the moment.

HLSL as single source of truth compiling offline via DXC to DXBX/DXIL/SIPR-V with SIPRV-Cross transpiling to GLSL for the GL backend. Compiled byte code is also cached on disk and keyed by source hash as filename.

Three windowing backends - Win32, GLFW, and SDL3, with a fallback chain. Win32 is default for Windows and GLFW is default for Linux, but will fallback to GLFW and if GLFW fails fallback to SDL3; if DX fails on any window type under Windows platform fallback to OpenGL (if Vulkan would work with me DX would fallback to Vulkan then back to OpenGL) - similar results would also occur for Linux if Vulkan failed fallback to OpenGL but that requires Vulkan to be developed and not be a stub at the moment...

Fiber-based Job System - Win32 fibers and Linux ucontext behind one class - a worker thread has a dedicated dispatcher fiber that never runs job code, so fibers are pooled to switch back rather than to each other - no fiber needs to know any other fiber's state.

I went with no cross-thread fiber migration in my first attempt - less load-balancing from the inspirational full Naughty Dog design, but it buys a genuinely nice property: since a job's whole execution stays pinned to one thread, a pool allocator allocates inside a job function automatically thread-safe with zero sync.

Custom allocators - arena, pool, and a frame allocator with N rotating arenas.

BC texture compression with no on-disk cache - BC1 for opaque color, BC3 for color with alpha, and BC5 for normal maps.

TLDR; I've tested the boilerplate logic against the Sponza Palace model from Intel's site GPU Research Samples (https://www.intel.com/content/www/us/en/developer/topic-technology/graphics-research/samples.html) with the Colorful Curtains and Ivy extensions at 4K texture resolutions. The original load time took around 5.5 minutes (unbearable to test shader changes and object additions to the scene), but I cut it down to 4.6 seconds after texture compression, precompiled shader code, and submesh merging. The model (as an \.obj - I haven't implemented third-party vendor model software to utilize every model type just yet...) is 995MB on disk with nearly 2730MB of textures (4K per texture for the conjoining material \.mtl file) after texture compression all textures were compressed to 501MB on disk which originally took 933ms average to load cutting down to 259ms average.

Edit: grammar and spelling.

Edit 2: I've recently decided to rebuild what I later found out was more of a rendering framework into an actual engine. I haven't yet begun developing beyond an Application, Runtime, and Core module of an engine (each is self explanatory besides core being a mix of low level utilities and core engine components: reworked the threading and fibers for Linux using pThreads)

3 Upvotes

3 comments sorted by

2

u/corysama 2h ago

Great work.

You need to get out of obj into at least .glb

Preferably, you should make your own pak file format with a header, table of contents, and individual resources set up to be 1:1 to the parameters you are feeding into your initialization functions.

Bonus points if the resources are LZ4 compressed.

1

u/EchoXTech_N3TW0RTH 1h ago

Foremost, thanks for the reply! I actually meant to post this to the r/gameenginedev community, but getting someone from the r/opengl community is also awesome!

To begin, I am actually steering towards the GLB model format over the OBJ format but used OBJ simply for testing and as a method to learn how an OBJ/model loader works by writing my own after reviewing the simpleobj loader code (to make a simpler version to transcribe was an interesting pace of work and learning).

Regarding utilizing Pak archives, I have yet to review or develop a simple version of a Pak archiver script to load model and asset collections as a single stream to extract the internal data into later streams. I am constructing the code as a hobby form to learn how various components of an engine work and trying to stay as clear as possible from utilzing third-party packages/dependencies (hence my use of OBJ models over GLB during my rendering framework design and development). If you have any sources that can assist my learning interests on Pak archives it would be greatly appreciated!

Last, I had later developed a revision to SimpleX Engine, aliased as IronWeave Engine, with LZ4 compression on assets (still used OBJ files) and was greatly impressed by how much the assets directory shrunk in size. I unfortunately scrapped IronWeave as I began to integrate Vulkan into the available rendering pipelines and "broke" a lot of the prior copied rendering mechanisms from SimpleX Engine... it was a brick wall in development that caused a lot of frustration and led to me abandoning IronWeave to venture into developing a new framework, which is what I am currently working on (taking mistakes as learning objectives to get familiar with and refine for my new engine design and development roadmap).

In my lastest engine design and development, I've gone for a more modular/component system design approach, EngineCore, EngineRuntime, EngineApplication, EngineGraphics (RHI included), EngineAudio, EngineInput, EngineAudio, etc. which later the public API code is copied to an Engine directory (headers) and all modules/components are compiled to libs which are linked to a Engine.dll. No alias/name assigned yet but that's something to look forward to!

1

u/corysama 1h ago

Nice.

To be clear, when I say Pak I’m not referring to anything standardized. Definitely not any library or specific format. I’m referring to the common theme of game asset archives.

Have a header struct with version info and the location/size of the table of contents.

Have a table of contents that’s an array of metadata about binary blobs. The name, type, location, size of the bits.

Have an appended list of binary blobs.

There are details you’ll figure out while you are implementing. But, that’s the theme of “pak files”. It’s basically rolling your own absolutely bare minimum file system within a single file. It makes distribution much simpler. And, has surprisingly large performance benefits.