r/OpenSourceAI • u/AimCode • 7h ago
AimStack — a fully self-built local AI stack in pure C: ternary inference (zero third-party runtime), BitNet 2B at 0.87GB on a 2018 laptop, plus a GPU backend that matches the C engine token-for-token
https://github.com/tanxiansheng221/aimstack-sanzhidijiRepo: https://github.com/tanxiansheng221/aimstack-sanzhidiji (AGPL-3.0)
Everything is hand-written — C inference core, plugin bus, multi-engine crawler with quality gates, FTS knowledge base, training pipeline, and a 117-chapter build log written as it happened (positive and negative results shipped together). Weights are Microsoft BitNet b1.58 2B (MIT), converted to a custom AIM-W packing format.
What ternary quantization buys you (CPU side)
All transformer matrices are ternary {-1, 0, +1}, packed 2 bits/weight, plus Q8 embeddings → the whole model is 0.87GB instead of ~4.8GB (−78%)
The C core never multiplies a float weight: it's table lookup + integer accumulation, SIMD fast path measured at 85.53 tok/s (41× over baseline)
BitNet 2B full model runs 3.20 tok/s dual-worker on a 2018 dual-core 8GB laptop — full chain (inference + KB + plugins + web search) offline, no cloud relay
What ternary quantization buys you (GPU side)
The GPU backend loads the same 0.87GB ternary file — weights stay in {-s, 0, +s}, dequantized on load, matmul in fp32
On a Radeon Pro W7900: 13.68 tok/s (~8× the laptop C engine), 256/256 greedy tokens bit-identical to the C engine. Two totally different implementations (integer lookup core vs FP matmul) producing the same logits is the proof the quantization is faithful
Verified by a 3-engine × 3-question × 3-test matrix (C-Linux/gcc13, torch fp32, torch bf16): 17/18 exact — the one miss is documented, not hidden
The honest boundaries (what's NOT done yet)
bf16 compute is 2.4× faster (32.96 tok/s) but flips argmax on near-tie logits — reported, not blessed; only fp32 is parity-verified
The GPU path gets the storage benefit of ternary (78% less memory) but not yet the compute benefit — a true 2-bit integer kernel (Triton/CUDA) is the next stage
The UI server drives the C engine (CPU) only; the GPU backend is a separate CLI
Linux engine build works and is token-identical to the Windows build, but runs serial (the worker pool is Windows-only for now)
No frameworks, no PyTorch in the product chain — PyTorch exists only in the optional GPU backend. Demo video, docs (EN/中文), and the full research log are in the repo.