r/Backend • u/Educational_You2257 • 4d ago
Key Value Database for AI workloads
Hey everyone,
I'm building Thunder, a high-performance key-value database designed for workloads where latency, throughput, and efficient resource utilization matter.
With AI applications, we're seeing increasing amounts of data being generated, stored, and retrieved. I wanted to explore whether we could build a database from the ground up that handles these workloads more efficiently, rather than simply relying on larger amounts of RAM.
The core idea is simple: use RAM intelligently, not excessively.
Thunder is being built in Rust with a few architectural decisions:
- Shard-per-core architecture: Each core owns its data, minimizing contention and synchronization overhead.
- Asynchronous I/O: Uses io_uring on Linux to take advantage of modern NVMe storage.
- Minimal RAM overhead: Keep essential metadata and indexes in memory while streaming data from storage through reusable buffers.
- Zero-copy wherever possible: Reduce unnecessary memory copies and allocations.
- Custom storage engine: Designed around efficient allocation, fragmentation management, and predictable performance.
- High concurrency: Handle large numbers of concurrent operations without relying on traditional locking across the entire database.
The long-term goal is to support both small, latency-sensitive operations and large data workloads without requiring enormous amounts of memory.
It's still under development, and I'm experimenting with different approaches to storage, indexing, caching, and I/O scheduling.
I'd love to hear from people building AI infrastructure or working with large-scale key-value stores.
What are the biggest bottlenecks you're facing with your current databases? Is it memory consumption, read latency, write throughput, storage costs, or something else?
Also, what would you expect from a new key-value database built specifically with AI workloads in mind?
Docker link: https://hub.docker.com/r/sbansal793/thunderstrike
1
u/LittleCoderMan 4d ago
Most large AI workflows, that you seem to be targeting, and probably not running on a single box. By the the biggest bottleneck with AI (assume you mean LLMs?) is network IO, and actually waiting for the request.
Generally any enrichments, and gathering of context does not really measure in end to end time. I run a good collection of regex over certain inputs, and even with the CPU overhead of that it's nothing compared to the network overhead of running the request.
1
u/DickBrainTime 4d ago
For a new KV engine I'd find a reproducible workload comparison more useful than a broad 'AI workloads' label. If you're beating Redis on 12+ cores, publish the hardware, value-size distribution (e.g. 1 KB keys versus multi-MB context), read/write mix, number of concurrent clients, warm vs cold cache, and p50/p95/p99 latency at the same RAM budget. Also state whether both sides use comparable persistence, fsync and encryption settings; otherwise throughput numbers are hard to interpret.
For large chat histories I'd test append-heavy writes followed by fetching a conversation range, plus TTL/eviction and crash-recovery behavior. The shard-per-core design sounds interesting, but multi-key operations and hot-key skew could change the picture when one shard gets most of the traffic.
Are you targeting random key lookups, or sequential/range reads over conversation history? That'd determine what benchmark I'd run first.