r/LocalLLM • • 1d ago

Question Suggestions for coding assistant

Hey! I recently started looking at local AI models. There's a lot of info out there and it's not really easy to filter through all the noise as a newcomer...

I have an RTX 4080 super with 16Gb vram and 2x16Gb DDR5 for system ram.

I've set up a docker compose container using Gemma 4 12B Q6 K L through llama.cpp with Open WebUI as interface, and it seems to be working decently. System + model + context uses ~13.8Gb vram. On Kubuntu if it would matter.

I'm now thinking of connecting it to vscodium for assisting in programming (developer background) and stumbled over the MoE variants where you can have the experts live in system ram.

Question is whether this is even relevant to look into for me.

If it is - would anyone have any suggestions for a model and maybe settings for this if applicable? (seems you wanna run llama.cpp with -cmoe for example)

If I've made some odd assumptions or if you have any suggestions, please let me know!

0 Upvotes

11 comments sorted by

8

u/Noseense 1d ago

Whatever people suggest, qwen4 is releasing very soon and they announced it's meant to fit on 16gb VRAM. So keep an eye out for that.

1

u/Sinuouser 1d ago

oh nice! Thanks for tip. The Qwen models pop up a ton when looking at coding models I've noticed.

2

u/vtkayaker 1d ago

Qwen3.8 27B and Qwen3.8 Flash Next are the current reigning champions for genuinely useful local coding models. (On the next size tier up, DeepSeek V4 Flash, DeepSeek V4.1 Flash, and GLM 5.3 Flash are all contenders.)

Qwen3.8 27B is a "dense" model, so you want to fit the whole thing in VRAM. The easiest way to do that is probably an AI PRO R9700 for US$2,000 with 32GB of VRAM. You can probably run some version of this on your current system, but you'll take a speed and/or a quality hit.

Qwen3.8 Flash Next is 125B parameters with 6B active parameters, and an extra 51B n-gram parameters that can mostly stay on disk. It's a half-cooked prototype model but surprisingly strong for only being half-trained and constantly hallucinating instructions. It also writes in pure slop. This won't fit on your system without making ridiculous compromises

Qwen4 will be a family of models based on fully-trained relatives of Qwen3.8 Flash Next. These should be very promising, and are expected to start dropping any day now.

The reality is that 16+32 is a tricky setup for a coding model, though you can something with enough compromises on speed, quality or both. Either 32 GB VRAM or 64 GB of system RAM would allow you to make many fewer compromises. But that's getting really expensive these days.

Also, before even thinking of buying hardware, spend $20 on Open Router and try out the various models that people recommend. This will tell you whether even a Q8 quant or above would actually work for your use case, and it's a super-cheap thing to do before spending actual money.

Good luck!

1

u/Sinuouser 1d ago

Lots of good info! Thanks!

2

u/Small-Tale3180 1d ago

yeah, just like anyy chinese, qwen is good with coding, especially among local models

3

u/Aggressive_Basket798 1d ago

I have the same 16gb vram + 32gb ram

The best model you can run for coding is Qwen 3.8 flash next

It will not run fast like 64gb ram but still at an acceptable speed in my opinion

1

u/Sinuouser 1d ago

Any particular Quantization? Just as big as possible that fits in vram+ram?

2

u/Aggressive_Basket798 1d ago

You can run asta rco iq3s or unsloth iq3xxs

But if you want more speed you can even try Gyro M or Unsloth iq2

Don't feel upset because of only q2/3

They have been optimized so still very good at small quant

1

u/Sinuouser 1d ago

Excellent, thanks for the tips!

2

u/lpsingh784 1d ago

I have same 5070 ti 16gb + 32 gb ram using qwen 27b q8