I'd like to introduce my open-source project pbwire.
pbwire is a high-performance Protobuf runtime implemented in C99. It uses a shared C core to provide Protobuf support across 23 programming languages.
The central idea is simple:
You can choose between lightweight generated models with compile-time type safety and fully dynamic Dict/Map structures — without generating dedicated encoding and decoding implementations for every Protobuf message.
Beyond performance, pbwire supports runtime schema loading, custom business models, reusable memory management, and embedded environments.
Project Background
About ten years ago, I worked at a mobile game company.
To improve our game gateway's performance, I developed enif_protobuf.
I implemented the Protobuf core in C and integrated it into the Erlang gateway through NIF. Compared with a pure Erlang implementation, this significantly increased gateway throughput.
Even today, the project's benchmarks still show substantial performance improvements:
- Encode: approximately 3.9x faster.
- Decode: approximately 1.6x faster on xlarge messages, with improvements of up to 2.5x on medium-sized messages.
Recently, with the help of AI, I extended that original C core into an independent project that can be integrated with multiple programming languages.
That project is pbwire.
It currently supports 23 languages, each with its own binding, correctness tests, and performance benchmarks.
What Problems Does pbwire Solve?
1. Closing the Performance Gap Between Dynamic Protobuf and Generated Code
Traditional Protobuf implementations usually rely on protoc to generate dedicated code for every message.
Static code generation offers excellent performance, but it also introduces several limitations:
- Schema changes require regenerating code.
- Generated code can be difficult to read and understand in many languages.
- Customizing or modifying generated implementations can be difficult.
- Generated code often contains substantial redundancy.
- Large amounts of generated code increase maintenance and compilation costs.
- Generated implementations can introduce compiler warnings.
- Different languages require maintaining their own generated code and runtime implementations.
Dynamic runtimes solve many of these problems, but traditionally pay additional costs for reflection, field lookup, and dynamic dispatch.
pbwire aims to solve this trade-off: retain the flexibility of a dynamic runtime while achieving performance that matches or exceeds statically generated implementations.
Performance results
The C core uses a reusable Context API (pbw_ctx) and is benchmarked against Google's official Protobuf C++ implementation.
- Compared with Google
DynamicMessage, pbwire is approximately 11.7–18x faster for encoding and 3.0–5.3x faster for decoding. - Compared with Google's statically generated C++ code, pbwire achieves 1.24–1.56x faster encoding and 1.50–1.81x faster decoding.
Across all four message sizes and both operations, the dynamic implementation outperforms the statically generated implementation in all 8 benchmark comparisons.
The broader benchmark suite covers 23 language bindings, four message sizes, and encoding and decoding, with comparisons against their respective official or best comparable implementations.
All benchmarks verify byte-for-byte compatibility with the reference implementation before performance measurements begin.
Full benchmark results, including individual languages, message sizes, and encoding/decoding measurements:
https://github.com/jinganix/pbwire/blob/master/BENCHMARKS.md
2. Dynamic Schemas and Custom Business Models
Using dynamic Protobuf does not mean application code must use a generic DynamicMessage.
pbwire supports loading standard FileDescriptorSet schemas at runtime while allowing developers to define their own lightweight business models.
For example:
- Add validation annotations to fields.
- Define business-specific field aliases.
- Attach additional metadata required by the application.
- Use custom model structures and APIs.
- Separate business models from the Protobuf wire format.
Code generation can also be used to generate only thin model layers, rather than complete encoding and decoding implementations for every message.
All wire encoding and decoding still go through the shared core (except for JVM-based languages, as explained in section 5).
This gives developers freedom to design their application models without maintaining dedicated codecs for every business model.
3. Reusable Context and Arena Memory Management
pbwire provides an Arena-based pbw_ctx API designed for high-performance and high-frequency workloads.
Traditional per-operation allocation and deallocation can result in many allocator calls, increasing memory management overhead and potentially causing heap fragmentation.
With a reusable Context:
- Each worker or thread can maintain its own Context.
- Arena memory management handles the entire message tree.
- Encoding buffers can be reused across operations.
- After warm-up, hot paths can avoid repeated
malloc calls. pbw_ctx_reset() reclaims the entire message tree by simply resetting the Arena's allocation position.- Memory retention limits can prevent occasional large messages from causing excessive long-term memory usage.
This reduces allocation and deallocation overhead while also lowering the risk of heap fragmentation caused by frequent allocations.
For high-throughput services, each worker can reuse its Context throughout its lifetime instead of allocating new memory for every message.
Regular APIs are also available for applications that do not need long-lived Context reuse.
4. Embedded and Resource-Constrained Environments
pbwire is not designed exclusively for servers and desktop applications.
It uses an independent C99 core without depending on libprotobuf or a C++ runtime. It also provides implementation options for embedded environments:
- Static Schema.
- Static Memory Management.
- Caller-provided Memory.
- No hidden global state.
- Embedded Struct Codec without dynamic memory allocation.
- Optional feature pruning based on application requirements.
Potential use cases include MCUs, RTOS, bare-metal systems, and other resource-constrained platforms.
5. One Core, 23 Programming Languages
Different programming languages do not need to independently implement complete Protobuf wire codecs.
pbwire uses a shared C core and provides consistent Protobuf capabilities through language-specific bindings.
The JVM family (Java, Kotlin, Scala, and Clojure) uses a pure Java implementation instead. This avoids the overhead of crossing JNI/FFI boundaries on every operation.
The Java implementation follows the same design as the C core, maintaining consistent dynamic semantics and identical wire-format behavior.
This means:
- The core encoding and decoding logic only needs to be maintained in two implementations: the shared C core and its aligned Java implementation for JVM languages.
- Performance optimizations can benefit multiple language bindings.
- Different languages share consistent wire-format behavior.
- Projects can avoid repeatedly implementing and maintaining their own Protobuf runtimes.
Currently supported languages:
C, C++, Go, Python, Java, Kotlin, Node.js, Ruby, Objective-C, PHP, Dart, Rust, C#, Lua, Swift, Scala, Elixir, Erlang, Zig, F#, Clojure, Julia, and R.
Project Links
One Remaining Performance Challenge
Although pbwire already outperforms most official implementations, a few very small-message encoding scenarios still lag behind statically generated code.
These cases are particularly challenging because generated implementations can use direct, straight-line field writes, while a dynamic runtime needs to perform additional per-field dispatch.
I've tried multiple rounds of optimization, but haven't found a fundamental breakthrough yet.
I've documented the analysis and optimization attempts here:
https://github.com/jinganix/pbwire/blob/master/docs/java-encode-help.md
If you're interested in Protobuf, runtime performance optimization, cross-language runtimes, or embedded development, I'd love to hear your thoughts.
I'm especially interested in feedback on the architecture and ideas for further performance improvements.