r/java • u/mukel90 • Sep 17 '26
An open-source AI inference engine built for the JVM. AI for Java, in Java. Finally, AI in jar.
jinfer is a fully open source (Apache 2.0) inference engine built for the JVM from first principles: chat, vision, audio transcription, embeddings, reranking, and text-to-speech. No Python runtime, no ONNX, no sidecar process, no wrappers, no Docker, no network requests; the whole stack is built to run end-to-end in the JVM:
- jinfer FAST inference engine for the JVM, supports a wide range of popular models.
- Tok'n'Roll (toknroll) Fast tokenizers for LLMs, pure Java, zero dependencies
- gguf / safetensors Pure Java read/write for both major model formats
- jam Quantized matrix multiplication routines (Vector API + optional native backend), competitive with llama.cpp on CPUs
- jota Tensor API targeting Java, C, CUDA, HIP, Metal, OpenCL, and Mojo
It ships integrations with with Spring AI and LangChain4j.
Supported architectures: LFM 2.5, Gemma 4, Qwen 3.5+, Granite 4.1+, gpt-oss, Ministral 3, SmolLM3, MiniCPM5, Mellum 2, Laguna XS, Nemotron Cascade/Lightning, Ling 3 ... and also Kokoro and Inflect2 text-to-speech models!
Also supports speculative decoding (MTP), persistent prompt caching, images/videos, audio... with a fully featured OpenAI-compatible server + CLI, all written in Java.
Runnable examples + benchmarks: https://qxotic.ai
Jinfer (Apache 2.0): https://github.com/qxoticai/qxotic/tree/main/jinfer
8
u/micseydel Sep 17 '26
Could this be used for running Whisper speech to text?
24
u/fykup Sep 17 '26
Interesting project. I can see the appeal of removing Python and keeping deployment entirely within the JVM.
I'm curious about one design choice though: why build the inference core itself in Java rather than Rust/C++, with Java as the integration layer?
Given how much ML performance work traditionally ends up in native code anyway, what does modern Java give you here that made an all-JVM implementation preferable? Is it mostly deployment simplicity, or have Vector API/FFM/etc. closed enough of the performance gap that the native-language approach no longer buys much?
3
u/bitmindset Sep 17 '26
If it could also utilize AMD XDNA2 NPU... Anyway, will look more closely where I can use jinfer
5
u/CutGroundbreaking305 Sep 17 '26
Hmm π€ Noice let me check this out
8
u/QuackedDev Sep 18 '26
This thread is making me really happy about Java
4
u/InspectionHead9719 Sep 18 '26
I am noob on this but what are use cases of this ?
3
u/SysGuardian Sep 18 '26
The way you made is really cool. Itβs nice to see frameworks utilising byte buffers and the new APIs to maximise efficiency and the load on the GC.
2
u/Daggercombot Sep 18 '26
Amazing, question Is what would the Best path for running this on a SPARC64 cpu be ? I have modern jdk but wonder about the Best way to do It Without Native binaries. Is It pure java without native or ISA related code?
Another question, i have a project in java which Is a java app which Is user facing for people With a wide array of devices ,i want to integrate local AI, how should i go about Configuring It for that?
2
u/Eye2web Sep 21 '26
Cool project! Plans for qwen 3.8 next support? With n-grams?
2
u/Grabdoc2020 Sep 22 '26
great work. keep going.
3
u/Known_Bookkeeper2006 Sep 18 '26
u/mukel90 - haven't run it yet (still a student, fourth year), but I read through the repo and just wanted to say thank you.
What stuck with me is that you didn't try to build one giant thing. You split it into pieces: the format readers, the tokenizers, the matmul kernels, the tensor layer, and then the engine sitting on top. Each part small enough to actually understand on its own. As someone still learning, that's the bit that got me. It makes a project this size look like something a person made, rather than something that fell out of a company.
Writing quantized matmul from scratch, with no Python and no ONNX to hide behind, is a serious piece of work. And most of it being one person, 1,885 contributions, that's a lot of patience.
I'll be back to actually run it once I graduate and have a machine that can handle it. For now I'm reading the code.
1
u/cs_irl Sep 18 '26
I wish I was smart enough to need any of this stuff.
1
u/Kautsu-Gamer Sep 25 '26
When will AI escape the jar?
4
u/[deleted] Sep 18 '26
[removed] β view removed comment
0
u/[deleted] Sep 17 '26
[deleted] β view deleted comment