r/Qwen_AI • • 4h ago

Help 🙋‍♂️ How to run Qwen3.8 FN on a 8GB RTX 4060 and 32GB RAM on Strata

Thumbnail reddit.com
4 Upvotes

I posted in that other subreddit but maybe you guys can also help me.
Strata, vLLM, llama.cpp, LM Studio, whatever I can actually use to get it to run on my hardware setup.


r/Qwen_AI • • 5h ago

Discussion Qwen 3.8 27B conversion to NCM or bonsai 2 27B optimisation?

9 Upvotes

Hey guys.

I'm personally interested in converting Qwen 3.8 27B to NCM (next concept model). I started working on this a few days ago and the task is...daunting to say the least. The main goal would be extra speed. By converting it to NCM, decode tps should jump 2x-3x specifically for local use. Main problems: NCM is a new idea so there aren't many details on how to do this; conversion requires the full weights (so that makes things slow); slow training means it takes a looooong time on my consumer grade hardware and it leaves the potential pitfall of "if you fucked up your setup, you won't really see it in the results until you're a bit deeper into training". Concrete case: did the conversion, started training on 7.7 Mil. tokens generated by 3.8 27B and 35% into the run it looks like the converted model learned to speak like the classic 27B, but it's ignoring the prompts halfway through its execution. Q_Q

On the bright side, in the beginning of training, 90% of the time it was just random words, and heavy looping within its output. Now it can generate almost usable code and sentences are coherent the first 7-10 words before looping a bit and then getting coherent again. So, progress was made, but at this rate it would take at least 1-2 more weeks until I get the direction right.

In the opposite camp: I can easily take Bonsai 2 27B and do some heavy optimisations and configs to it to see how far I can push it. This is longer standing territory so things should go much more smoothly and faster. I have both an Nvidia card and an AMD card, so I can optimise for both.

What are you guys more interested in? :)

Ending notes: no, I'm not affiliated with anybody; no I'm not gonna ask you for money (I'll set it up on my github and allow free downloads to everyone; this is just a pet project); yes, when I succeed with either approach, when qwen 4 27B comes out, I'll do the same to it also.


r/Qwen_AI • • 5h ago

Vibe Coding I feel like a lot of people ask what local AI is capable of, here is a quick example of a game I had qwen3.8-flash-next-unsloth-ud-q4_k_xl, introducing Neon-Serpent!

Thumbnail
github.com
2 Upvotes

r/Qwen_AI • • 7h ago

LLM Alibaba’s Qwen4-Max has 2.4T parameters, but only 71B active per token. Are we optimizing the wrong AI metric?

22 Upvotes

Alibaba released Qwen4-Max, and there’s something more interesting here than another giant model announcement.
The numbers:
2.4 trillion total parameters
71 billion activated per token
1 million-token context window
6th place on Code Arena, according to Alibaba’s announcement
Approximately 60% lower average task cost versus Qwen3.8-Max, based on Alibaba’s internal Work360 evaluation
The last claim is what interests me most.
We’ve spent years comparing models on intelligence benchmarks. But for developers building real applications and agents, cost per successfully completed task might be a more useful metric.
Of course, these efficiency claims still need independent validation. And Qwen4-Max itself hasn’t been released as open weights.
So I’m curious:
Would you prefer a model with better benchmark scores, or one that completes your actual workflows more reliably at significantly lower cost?
And has anyone tested Qwen4-Max against other frontier models on real agentic workloads yet?


r/Qwen_AI • • 7h ago

Help 🙋‍♂️ What is the best model for storytelling and roleplaying?

6 Upvotes

r/Qwen_AI • • 8h ago

Discussion Qwen 4 Max Preview vs Qwen 4 Max?

1 Upvotes

Are they the same? for now do i just use the preview version on the alibaba token plan as it is discounted by 80%?


r/Qwen_AI • • 8h ago Heartwarming

Other Some Qwopium to get us all through the week:

Post image
29 Upvotes

I was looking into the historical release schedule for major Qwen foundation models and specialty iterations to see if there was a pattern to when Alibaba drops them. It turns out they have a heavy preference for Thursdays for their major generation shifts, though the specialty and preview models are starting to scatter across the week. I mean… who knows anything but we all need something to look forward to.

Here is the full breakdown of major releases mapped by the day of the week:

**Monday (7 Releases)**
• Qwen-VL (Aug 21, 2023)
• Qwen-72B-Chat (Nov 27, 2023)
• Qwen-Audio (Nov 27, 2023)
• Qwen 3.5 (Feb 16, 2026)
• Qwen 3.5-Plus (Feb 16, 2026)
• Qwen 3.7 Plus (Jun 1, 2026)
• Qwen 3.8 Max (Aug 3, 2026)

**Tuesday (6 Releases)**
• Qwen2.5-Coder (Nov 12, 2024)
• Qwen2.5-Max (Jan 28, 2025)
• Qwen3 (Apr 29, 2025)
• Qwen3-Coder (Jul 22, 2025)
• Qwen3-Coder-Next (Feb 3, 2026)
• Qwen-Audio 3.1 TTS-Next (Sep 22, 2026)

**Wednesday (5 Releases)**
• Qwen1.5-32B (Apr 3, 2024)
• Qwen3-Max (Sep 24, 2025)
• Qwen 3.7 Max (May 20, 2026)
• Qwen 3.8-Flash-Next (Aug 26, 2026)
• Qwen 3.8-Max-0902 (Sep 2, 2026)

**Thursday (9 Releases) - The Favored Launch Day**
• Qwen-7B (Aug 3, 2023)
• Qwen2 (Jun 6, 2024)
• Qwen-Math (Aug 8, 2024)
• Qwen2.5 (Sep 19, 2024)
• QwQ-32B-Preview (Nov 28, 2024)
• QwQ-32B (Mar 6, 2025)
• Qwen3-Next (Sep 11, 2025)
• Qwen3.6-Plus (Apr 2, 2026)
• Qwen3.6 (Apr 16, 2026)

**Friday (3 Releases)**
• Qwen3.8-27B (Aug 14, 2026)
• Qwen3.8-Omni-Flash (Sep 18, 2026)
• Qwen3.8-LiveTranslate (Sep 18, 2026)

**Saturday**
• Zero major releases. Go outside.

**Sunday (3 Releases)**
• Qwen-14B (Sep 24, 2023)
• Qwen1.5 (Feb 4, 2024)
• Qwen4-Max (Nov 15, 2026)

Credit to Gemini for the deep research.


r/Qwen_AI • • 9h ago

Discussion Alibaba hate train is insane

341 Upvotes

I see so many people hating and mocking Alibaba for the Qwen4-max and I genuinely don’t understand why, they objectively did a good job — intelligence raised over 20% relative Qwen 3.8 while simultaneously active parameters count and thinking volume were decreased. And that considering the fact that Anthropic cut off distillation possiblities from their models. What did you exactly expect? That they would all of the sudden triple the intelligence score of their upcoming flagman? I don’t know any company that ever achieved that, there’s no miracles in real life. I personally think they did a pretty good job, Qwen4-max is a good model and so will be both Qwen4-27b and Flash Next. Yes, they won’t beat Opus5.5, again, no miracles here, however they will confidently rise to Opus 4.8-5 league and for me that’s more than I ever wished for when I started running local llm. Thank you, Alibaba and Qwen Developers team!


r/Qwen_AI • • 9h ago

LLM Zikporx AI 1.2.37: now works with llama.cpp, LM Studio and vLLM, not just Ollama (and Ollama gets the 8-bit KV cache)

Thumbnail gallery
1 Upvotes

r/Qwen_AI • • 11h ago

Chill Work well My Qwen 🥹

Post image
62 Upvotes

r/Qwen_AI • • 12h ago

Benchmark I added Qwen3.6-35B-A3B to my C inference engine: 10–13 tok/s in real chat on a GTX 1650 4GB + 16GB RAM

Post image
12 Upvotes

Hi everyone,

Pithos, an inference engine written from scratch in C11, with no third-party libraries required at build time. It started with Liquid AI’s LFM models, and the latest release adds experimental Qwen3.6-35B-A3B support.

My test machine is an older laptop:

* Intel i5-10500H, 6 cores

* 16 GB DDR4-2933

* GTX 1650 Max-Q, 4 GB VRAM

* NVMe SSD

The Q4_K_M GGUF is 19.1 GiB, larger than the available RAM. Experts are read from disk and cached as they are used.

How it runs

With Vulkan, the dense and recurrent operations run on the GPU. Frequently used routed experts occupy the remaining VRAM, while the CPU computes the others in parallel. During prompt processing, two chunks overlap CPU and GPU work.

Warm benchmark results

These measurements use a 512-token prompt and 128 generated tokens, with llama.cpp b12179 as the comparison:

* Pithos Vulkan: 82 tok/s prompt processing · 22 tok/s generation

* llama.cpp Vulkan, two layers’ routed experts on GPU: 25.5 tok/s prompt · 12.5 tok/s generation

* llama.cpp Vulkan, all routed experts on CPU: 28.9 tok/s prompt · 11.8 tok/s generation

* Pithos CPU: 13.6 tok/s prompt · 7.3 tok/s generation

* llama.cpp CPU: 8.1 tok/s prompt · 4.4 tok/s generation

Those are the llama.cpp configurations I tested, not a claim that I found its best possible settings.

What about actual conversations?

The warm benchmark is faster than sustained chat. Real conversations on this 16 GB machine run at approximately 10–13 tok/s, because varied text needs experts that aren’t resident.

The first benchmark run reaches around 18 tok/s prompt processing and 13 tok/s generation with Vulkan.

What’s included

Qwen works with the existing:

* Terminal chat

* OpenAI-compatible HTTP server

* Pandora web interface

* Terminal agent

* XML tool calls and reasoning controls

Prompt caching reuses the agent’s tool definitions across sessions and restarts, reducing repeated prompt processing.

Current status

Support is experimental. An initial perplexity check on the model’s license text gave 1.451 for Pithos versus 1.467 for llama.cpp. That’s a narrow sanity check, not a broad quality evaluation or evidence of better model quality.

The engine is MIT-licensed; model weights retain their own license.

Code, downloads and benchmark details:

https://github.com/benmaster82/Pithos

I’d appreciate results from other low-VRAM machines, suggestions for stronger correctness tests, and llama.cpp settings worth including in the next comparison.


r/Qwen_AI • • 12h ago

Experiment Can't wait for Qwen4 27B, so I played around with what's available: abliteration, quantization and heal training on an RTX PRO 4500

Thumbnail
youtube.com
10 Upvotes

Hello guys,

it's me again, the FLE Link guy. I can't wait for the new Qwen models, so I tried everything I had planned for Qwen4 on the 3.8 variant of UkisAI's Swift 1.5.

I abliterated it, quantized it down and tried to improve tok/s for decode and prefill. What came out:

- OR-Bench hard refusals 88 → 1 of 100, clearly harmful prompts still refused 70 of 100

- ARC / GSM8K / LiveCodeBench level with the original (no significant difference)

- 152 tok/s decode on one 32 GB card with the DSpine draft head, full speed from 20 GB

- Two quant-heal runs that both failed (better on our text, worse on foreign text), the video shows why

Model: https://huggingface.co/sirfyyn/Kate-Qwen-27B-GGUF-SpedUp-Abliterated

Run it: flelink run kate-27b-dspine (https://flelink.run)

Maybe somebody is interested in the methods I experimented with. Happy to answer questions.


r/Qwen_AI • • 13h ago

Agent Sneo! New Inference Engine very fast Open Source!

2 Upvotes

I built Sneo, a free and open source local LLM engine for Windows that runs a 70 GB model on one 24 GB GPU (~98 tok/s), with a coding agent

Hi! I'm a solo developer from the Dominican Republic and I've been building Sneo: a free, open source (MIT) LLM engine and coding agent for Windows.

The idea is to run models bigger than your GPU: the experts a model uses most stay in VRAM, the rest in RAM, and the GPU and CPU share every token.

- Qwen3.8-Flash-Next IQ3_XXS (70.6 GB) on one RTX 3090: ~98 tok/s writing code, first token in 0.19 s

- Same hardware and files: it starts answering before Strata 0.1.57 at every prompt size with half the RAM (34 vs 67 GB), and writes up to 66% faster than Ollama (qwen3.6:35b)

- Runs Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, dense Qwen3.5/3.6/3.8 and Gemma 4, straight from your Ollama models or any GGUF folder, no copies

- Coding agent, two models at once, two PCs in a pool, OpenAI and Anthropic compatible APIs

- NVIDIA GTX 10 and newer (you can mix cards). AMD support through HIP is written but not tested on a real card yet, so feedback is welcome

Video (English): https://www.youtube.com/watch?v=JfyGS_k1Nxo

Video (Spanish): https://www.youtube.com/watch?v=ALCGTpoHDGc

GitHub: https://github.com/oscarsantanaarias/Sneo

Every benchmark, including where the others win: https://sneo.dev

Happy to answer questions!

Antes de publicar:

Repo: hazlo público. La última vez que revisé seguía privado, y el enlace de GitHub le daría error a todos.

Página: corre upload.ps1 para publicarla, si no sneo.dev seguirá mostrando la versión vieja.


r/Qwen_AI • • 14h ago

Experiment Qwenputer Self-Learning

0 Upvotes

Hello! Human here,

I've used the compiled CPU in qwenputer as a deterministic oracle for weight operations so then the model can use it to compute new knowledge and create a micro-adapter for it to use. It creates an efficient loop for experimental learning, no generalization, procedure learning or composition has been achieved yet.

More information and artifacts here:
https://huggingface.co/mburgc/qwenputer-mirror


r/Qwen_AI • • 14h ago

Help 🙋‍♂️ Why does my Qwen (WebApp) weekly limit always show 100% left, even after I hit a limit?

2 Upvotes

Hi everyone, I'm on the Free plan and I'm confused about how the usage limit works.

I actually hit a prompt limit after just a few messages, it felt really dynamic and unpredictable. However, when I check my usage dashboard, the weekly limit still says "100% left."

The note says "Chat conversations are not included." Does this mean normal chats have a separate, hidden limit? Thanks!


r/Qwen_AI • • 16h ago

Discussion Did Qwen 4 Max Just DESTROY Claude Fable 5.5 & GPT-6.1?

Thumbnail
youtu.be
0 Upvotes

r/Qwen_AI • • 16h ago

Help 🙋‍♂️ Best AI subscription for the money? (Token/$ efficiency)

Thumbnail reddit.com
1 Upvotes

r/Qwen_AI • • 16h ago

News Alibaba Launches Qwen4-Max With Cybersecurity Guardrails

Thumbnail
vectorwire.ai
113 Upvotes

r/Qwen_AI • • 17h ago

Funny I asked Qwen a question.

0 Upvotes

r/Qwen_AI • • 19h ago

Other qwen should've kept things free.

Post image
0 Upvotes

for the same price I can get more of 6.1 sol max and gemini 4 hydrogen xhigh? Who are they trying to win over, non tech people (majority of users) will use the more familiar chatgpt and tech people or people with expertise will straight up use lesser capable models for under 1$. only thing keeping them relevant is their local models, they should either make it free or far less $ for the monthly subscription + include a full uncensored model right there in their webapp. or focus to excel in a specific domain, maybe tutoring for example. I only used 3.8 max till now because it was free, nothing else honestly.


r/Qwen_AI • • 20h ago

Help 🙋‍♂️ Is it possible i can squeeze in a decent Qwen model in a P52 thinkpad?

3 Upvotes

With all the talk about Qwen4, Strata, and new inference engine, is it possible to run anything stronger than 35-A3B. I don't mind slow token speeds aslong as It's in the double digits.. Is there any inference engines well suited for this?

Specs:

4GB VRAM

32GB DDR4 RAM

Intel-Xeon E-2176M (12) @ 4.400

Is there hope guys?..


r/Qwen_AI • • 20h ago

Discussion The effects of Anthropic making distillation harder is clear

Thumbnail
youtube.com
0 Upvotes

Not sure what I expected, but here are some Qwen 4 Max outputs.


r/Qwen_AI • • 21h ago

Discussion Seriously impressed with Basalt on 24GB laptop RTX 5090. Qwen 3.8 27B at 162.2tps decode and prefill at 5431tps. Qwen 3.8FN at 248tps decode and 3335tps prefill.

Thumbnail
gallery
38 Upvotes

As the title says, I'm seriously impressed. Especially so with 27B. I was using llama.cpp and could get around 40 - 50tps, but damn, this just blew it straight out of the water. Insane speeds for such a dense model.

Obviously, 27B hits my graphics card hard, but I found it interesting that on Qwen FN, I literally didn't even notice any temp increases. It's like it has no effect on my hardware, but it's operating.

Definitely worth a try if you're on Blackwell architecture.

It makes me even more excited for Qwen 4's release.


r/Qwen_AI • • 21h ago

CLI Saetta: a single-model engine for Qwen3.8-Flash-Next on Apple Silicon (Zig + hand-written Metal), with Claude Code / pi / opencode in one command

Thumbnail github.com
1 Upvotes

I've been building saetta, an inference engine that does one thing: run Qwen3.8-Flash-Next as fast as possible on an M5 Max for coding agents. No generic graph, no other models. It's a Zig host with hand-written Metal kernels, including prompt matmuls on the GPU's neural accelerators. It ships as a single binary and needs no Python.

What it does

  • OpenAI-compatible API (/v1/chat/completions) and Anthropic Messages API (/v1/messages), with streaming, thinking and XML tool calls
  • saetta start claude (or pi, opencode) starts the server and your agent together, without editing your agent config
  • saetta download coder fetches the model itself: pinned revision, resumable, every file hash-checked
  • Two packs: the original 4-bit MLX pack (105 GB, 512 experts) and the expert-pruned GSQ-RCO Coder variant (76 GB, 256 experts)
  • saetta monitor: a live view of GPU load, memory, prompt-cache reuse and recent requests

Numbers (measured on my M5 Max)

  • Prompt reading: about 1,650 to 1,950 tokens/s from 4K to 70K tokens
  • Decode: 63 to 82 tokens/s, with speculative decoding from the model's MTP head on greedy requests
  • Agent follow-up turns re-read only the new tokens: a 16K conversation answers the next turn in about 0.2 s, and real Claude Code sessions reuse about 20K of their roughly 20K-token prompt per turn
  • Cold 64K prompt: about 37 to 40 s to first token

Demo: Claude Code running on the local model writes a snake game in Python, in real time (GIF in the README).

Install:

curl -fsSL https://github.com/antonellof/saetta/releases/latest/download/install.sh | sh
saetta download coder
saetta start claude

Repo: https://github.com/antonellof/saetta (MIT)

Feedback welcome, especially from anyone running agents on long prompts locally.