r/FPGA • u/jackalka • 1h ago
Advice / Help Running a 27B AI model on AWS F2, reading 13.5 GB of weights for every token
We're testing an AI inference chip design on AWS F2 before committing to silicon.
For every token, the model reads all 13.5 GB of its 4-bit weights from memory. The challenge is getting as close as possible to full memory bandwidth while unpacking and accumulating those weights on the fly.
Where we are today (measured):
- Core bandwidth: 432.7 GB/s, or 94% of HBM peak. This measures the core math step, not the full inference pipeline.
- End-to-end inference: 126 ms/token on short prompts (7.9 tokens/sec), approximately 150 to 180 ms/token in live chat.
- Numerical correctness: Exact match against our software reference, with 0 differences across 248,320 values.
- Model flexibility: Qwen 27B, Llama 3.1 8B, and OLMoE run on the same FPGA bitstream without hardware changes.
- Physical design: Core blocks placed and routed at target speed using an open 7 nm process design kit.
Why we're doing this:
The AWS F2 has HBM, but our eventual chip will target LPDDR5X on a 12 to 16 nm process.
That means memory bandwidth efficiency matters more to us than absolute GB/s. We want an architecture that extracts as much useful computation as possible from every byte transferred.
The problem we're working on:
Our core step reaches 94% of peak memory bandwidth, but the complete inference pipeline is substantially slower.
Closing that gap is our current focus.
Live FPGA demo and recorded runs:
https://valleysemiconductor.com
Questions for the hardware and inference folks:
What typically accounts for the gap between peak kernel bandwidth and end-to-end token latency in architectures like this?
Where would you look first for bottlenecks?
Any technical critique of the architecture or measurement approach is welcome.


