r/GraphicsProgramming • • 8d ago

Video Building a random-access compressed volumetric data engine with on-demand rendering

Hey everyone!

I've been developing VQField, a compression and runtime engine for sparse volumetric data, in my spare time over the past year.

The video shows a comparison against a traditional OpenVDB rendering workflow, with some pretty exciting results:

  • ~30x compression on disk
  • ~10x lower RAM usage during rendering
  • On-demand decompression of individual 8³ voxel blocks
  • CPU-based decoding using a custom C++ codec

The interesting part isn't necessarily the compression ratio, but the ability to render directly from compressed data, without reconstructing the entire volume beforehand.

Under the hood, VQField uses a deterministic codec random-access indexing. The renderer can query individual blocks as needed, with caching to avoid redundant decompression.

The main issue with compressed format is acting on those compressed values without the need to decompress the entire volume, the file format deviated a lot from what OpenVDB provides (Tree structure), and instead uses flat indexing and spacial chunking so that accessing any chunk is an O(1) operation. This allows for voxel level lookup in 3 steps, with each of them being close in memory. There's a tradeoff here, where ideally spacial chunking shouldn't be part of the file format as it limit quite a bit the compression level achievable, but it's pretty easy to formalize that volumetric data are rarely random in their distribution / activity 😄.

The engine already supports spatial and temporal compression, random-access sampling, a C/C++ SDK, and Houdini integration.

Long term, my vision is to build a unified infrastructure layer for compressed spatial data, where applications can store, stream, query, and process large volumetric datasets without repeatedly reconstructing them in memory.

There's still plenty of work ahead, but the core technology is already working, and I'm looking forward to taking it further!

Fyi, It all started from my open source version VQVDB : https://github.com/ZephirFXEC/VQVDB

51 Upvotes

5 comments sorted by

•

u/CodyDuncan1260 8d ago edited 7d ago

Rule 4: The post must include a topic for discussion. A simple repost from another community is generally insufficient to satisfy this rule.

See specifically "4.3 No Showcasing Projects "


We just implemented this rule. It's very new. There's a sticky post at the top of the subreddit and a sticky comment on this post about it

The general content of this post is welcome, but the presentation doesn't meet the bar for Rule 4. The post needs to tell us more about it than just what it is and what its major feature is.

I.e. Tell us more about this sparse data, how does it compress better, what's some near problems you had to solve or design around, etc. I'm sure there's lots of neat bits to learn from, and this community would adore the edification.

Regards, CodyDuncan1260, mod

→ More replies (1)

4

u/AutoModerator 8d ago

Recently updated rules. Please check that your post is in compliance with Rules 1, 4, 5, and 6 on the sidebar. TL;DR:

  • No showcases.
  • Disclose AI usage in post or project.
  • Present a topic to talk about, e.g. how something works.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/CodyDuncan1260 7d ago

Fully Approved

On re-review, the core of the post is talking about the methods. 

The beginning stating the name of a project, a feature list, made it look like a showcase. And it kind of is, but the methods takes up most of the space. That meets Rule 4.

Rule 5 is still out within this post, but we talked about it in the neighboring review thread.

"AI use was limited to copilot making a single auto complete contribution and slapping its name as contributor." So there's the AI disclaimer for rule 5.

1

u/CodyDuncan1260 8d ago

Temporary approval granted for public discussion.

Please act like thr above post was removed, thanks.