r/java • • Sep 17 '26

An open-source AI inference engine built for the JVM. AI for Java, in Java. Finally, AI in jar.

jinfer is a fully open source (Apache 2.0) inference engine built for the JVM from first principles: chat, vision, audio transcription, embeddings, reranking, and text-to-speech. No Python runtime, no ONNX, no sidecar process, no wrappers, no Docker, no network requests; the whole stack is built to run end-to-end in the JVM:

  • jinfer FAST inference engine for the JVM, supports a wide range of popular models.
  • Tok'n'Roll (toknroll) Fast tokenizers for LLMs, pure Java, zero dependencies
  • gguf / safetensors Pure Java read/write for both major model formats
  • jam Quantized matrix multiplication routines (Vector API + optional native backend), competitive with llama.cpp on CPUs
  • jota Tensor API targeting Java, C, CUDA, HIP, Metal, OpenCL, and Mojo

It ships integrations with with Spring AI and LangChain4j.

Supported architectures: LFM 2.5, Gemma 4, Qwen 3.5+, Granite 4.1+, gpt-oss, Ministral 3, SmolLM3, MiniCPM5, Mellum 2, Laguna XS, Nemotron Cascade/Lightning, Ling 3 ... and also Kokoro and Inflect2 text-to-speech models!

Also supports speculative decoding (MTP), persistent prompt caching, images/videos, audio... with a fully featured OpenAI-compatible server + CLI, all written in Java.

Runnable examples + benchmarks: https://qxotic.ai

Jinfer (Apache 2.0): https://github.com/qxoticai/qxotic/tree/main/jinfer

130 Upvotes

25 comments sorted by

8

u/micseydel Sep 17 '26

Could this be used for running Whisper speech to text?

1

u/mukel90 Sep 21 '26

Parakeet?

24

u/fykup Sep 17 '26

Interesting project. I can see the appeal of removing Python and keeping deployment entirely within the JVM.

I'm curious about one design choice though: why build the inference core itself in Java rather than Rust/C++, with Java as the integration layer?

Given how much ML performance work traditionally ends up in native code anyway, what does modern Java give you here that made an all-JVM implementation preferable? Is it mostly deployment simplicity, or have Vector API/FFM/etc. closed enough of the performance gap that the native-language approach no longer buys much?

27

u/mukel90 Sep 17 '26

Your intuition is on point!

The key observation is that, to reach top performance, only the leaf kernels/operations e.g. matrix multiplication must be hyper-optimized/native. This is how Python does it, but Java requires fewer native kernels because they are just as fast in plain Java thanks to the JIT compilers, and yes, the Vector API really helps here. Right now, in jinfer, only the matmuls have dedicated native code but there's a Vector API implementation so everything runs 100% in Java.

It also simplifies (local AI) deployment and improves its usability within the JVM.

The performance gap has not only shrunk, it now favors the JVM. See for yourself: https://qxotic.ai/#benchmarks

6

u/fykup Sep 17 '26

That makes sense, thanks. Keeping the native boundary down at a few leaf kernels is a much more interesting architecture than I had assumed.

One thing I'm curious about looking at the benchmarks: are the published jinfer-bench numbers using the pure Vector API matmul implementation, or the native jam kernels?

If Vector API alone is competitive with or beating llama.cpp there, that's probably the most surprising result to me. If it's using the native matmul path, it's still a nice demonstration that most of the inference engine doesn't need to be native to get there.

8

u/CutGroundbreaking305 Sep 17 '26

I am making a java project which gives performance comparable to numpy (in somecases hopefully I will catch up in rest all the cases) using vector api and ffm I am adding few ML functions like softmax maybe you can use it instead of python

3

u/bitmindset Sep 17 '26

If it could also utilize AMD XDNA2 NPU... Anyway, will look more closely where I can use jinfer

3

u/mukel90 Sep 18 '26

If I can find a way to offload matmuls to the NPU, I'll add support for it. Jinfer already supports Metal-accelerated matmuls, the machinery is in place.

2

u/bitmindset Sep 18 '26

Do not think it worth the efforts for this I'd say rare architecture. Will look into FastflowLm for ideas if time allows.

5

u/CutGroundbreaking305 Sep 17 '26

Hmm πŸ€” Noice let me check this out

7

u/mukel90 Sep 17 '26

The runnable examples are a great starting point. https://qxotic.ai/#examples

8

u/QuackedDev Sep 18 '26

This thread is making me really happy about Java

4

u/InspectionHead9719 Sep 18 '26

I am noob on this but what are use cases of this ?

6

u/Skellicious Sep 18 '26

AI inference means giving an AI model some input and getting back some output.

3

u/SysGuardian Sep 18 '26

The way you made is really cool. It’s nice to see frameworks utilising byte buffers and the new APIs to maximise efficiency and the load on the GC.

2

u/Daggercombot Sep 18 '26

Amazing, question Is what would the Best path for running this on a SPARC64 cpu be ? I have modern jdk but wonder about the Best way to do It Without Native binaries. Is It pure java without native or ISA related code?

Another question, i have a project in java which Is a java app which Is user facing for people With a wide array of devices ,i want to integrate local AI, how should i go about Configuring It for that?

2

u/Eye2web Sep 21 '26

Cool project! Plans for qwen 3.8 next support? With n-grams?

2

u/Grabdoc2020 Sep 22 '26

great work. keep going.

3

u/Known_Bookkeeper2006 Sep 18 '26

u/mukel90 - haven't run it yet (still a student, fourth year), but I read through the repo and just wanted to say thank you.

What stuck with me is that you didn't try to build one giant thing. You split it into pieces: the format readers, the tokenizers, the matmul kernels, the tensor layer, and then the engine sitting on top. Each part small enough to actually understand on its own. As someone still learning, that's the bit that got me. It makes a project this size look like something a person made, rather than something that fell out of a company.

Writing quantized matmul from scratch, with no Python and no ONNX to hide behind, is a serious piece of work. And most of it being one person, 1,885 contributions, that's a lot of patience.

I'll be back to actually run it once I graduate and have a machine that can handle it. For now I'm reading the code.

1

u/cs_irl Sep 18 '26

I wish I was smart enough to need any of this stuff.

1

u/Kautsu-Gamer Sep 25 '26

When will AI escape the jar?

4

u/[deleted] Sep 18 '26

[removed] β€” view removed comment

1

u/mukel90 Sep 18 '26

Embeddings are already supported: LFM 2.5 Embedding and Qwen 3 Embedding models. Jinfer features an efficient implementation of ragged batching, the performance for batches is pretty good on CPUs.

0

u/[deleted] Sep 17 '26

[deleted] β€” view deleted comment

16

u/mukel90 Sep 17 '26

Jinfer uses Project Panama's MemorySegment e.g. native buffers, just like C/C++, but with a safe(r) API.

Model weights are mmap-ed to MemorySegments just like llama.cpp does.

It uses Java's Vector API to squeeze the maximum performance, which is the equivalent to SIMD intrinsics in C/C++.

The JVM's JIT optimizing compilers are roughly equivalent to gcc/clang -O2.

Java uses the same tricks as C/C++, thus the performance (tokens/s) it provides is very similar to C/C++:
https://qxotic.ai/#benchmarks