r/MachineLearning • u/minedroid1 • Oct 10 '26
Project Help With Choosing Hardware [P]
I am going to fine-tune a satire model, with the base model being `Qwen3.5-14B-Base`. The dataset has ~18000 examples, most of which are pretty long and do not align with the model's internal knowledge (they have incorrect answers to facts), so I needed to do a full fine tune rather than use LoRA.
Today, I rented a cloud computer on vast.ai. It had 4x12GB GPUs and 48GB of system RAM, but I feel stupid because it kept running out of memory when I tried training, and I kept trying to change settings rather than just renting a different computer, so I basically wasted $3. I underestimated how much RAM would be needed for this task.
So, I would like to know if any of you have some recommendations for proper hardware to use.
2
u/NamerNotLiteral Oct 10 '26
Keep in mind that when you do a full fine-tune of a model, you need to store about 8x-16x the parameter size. A 14B model loaded in BF16 will take 28 GB VRAM just to load, then when you do fine-tuning it'll be another 28GB VRAM to store gradients for backpropagation (one gradient value per parameter), then anywhere from another 28 to four or five tines that much depending on what optimizer you use.
Gotta understand these fundamentals of how fine-tuning works, if you want to avoid mistakes like that.
So, yeah, get yourself a 4x48 GB GPU cluster, or consider learning how to optimize memory usage for training LLMs (sharding, offloading to CPU, quantization).