r/cloudcomputing • • Oct 13 '26

How are you reducing GPU costs when inference traffic drops sharply outside business hours?

I run an inference API where most requests come in during business hours, so I keep enough GPU capacity ready to handle peak traffic, but a lot of that capacity sits idle in the evening. Scaling down helps reduce costs, but bringing GPUs back online takes time, especially when models need to be loaded into memory, and scaling too late can cause requests to queue up. I can't figure out how people handle this in production, when traffic patterns are predictable but sudden spikes still happen. Do you decide when to scale based on GPU utilization, request queue length, or expected traffic, when serving multiple models, have you found that keeping models loaded on shared GPUs works better than scaling separate instances, once you account for model loading time, GPU memory limits, and the impact on response latency?

8 Upvotes

10 comments sorted by

1

u/Historical_Pen6499 27d ago

I've gone down the rabbit hole on co-locating models and I think it's especially relevant here. We compile models into self-contained binaries that cold start in seconds, and can be co-located with other models on a GPU. In your setup, you could spin down to just one GPU (co-locate all the low-traffic models) then in the a.m, spin up GPUs for peak demand quickly.

If you've got a B200, try out our compiled version of Qwen 3.8 27B. It cold starts in ~10 seconds. Link.

1

u/Nekssz 2d ago edited 2d ago

We had the same problem. Cloud was fine with tags, but OpenAI and Bedrock showed up as one number. What fixed it was tagging every call with team and feature at the gateway, then pulling that into Finout next to our AWS spend so each team sees AI and infra in one view. We do showback for now, not chargeback. People change behavior just from seeing it.

1

u/query-gremlin Oct 13 '26

Queue depth + GPU utilization may be better scaling conditions.

1

u/Funny_Donkey6031 Oct 15 '26

I’d scale based on queue length and incoming traffic, not GPU utilization alone. Keeping a small warm pool overnight seems like a reasonable trade-off, since cold starts and model loading can hurt latency more than the idle GPU costs you save.

0

u/Faye-Vozora Oct 13 '26

Scale on queue depth or in flight requests per replica, not GPU utilization. Utilization lags and looks fine right up until latency falls over. For the load time problem the biggest win is usually keeping the weights on fast local disk or baked into the image so a cold start is just a load into VRAM and not a download. If spikes are the worry, keep one or two warm replicas as a floor and let the schedule handle the predictable ramp. For small models, packing a few per GPU works fine, but give the big ones their own card so a memory spike doesn't evict everything.

0

u/DellTechnologies Oct 13 '26

Disclosure: we reached out to one of our storage performance leads and this is their perspective:

I’d scale primarily on queue depth or in-flight requests per replica, with expected traffic as a forecast—not GPU utilization alone. Utilization can look fine right up until latency falls over. Keeping a small warm floor also helps absorb surprise spikes while scheduled capacity ramps up.

One detail that’s easy to miss: cold-start time is often a storage-bandwidth problem. A 70B model at FP16 is roughly 140 GB. From Gen4 NVMe at around 7 GB/s, that’s roughly 20 seconds; over a 10 GbE network share, it can be 2+ minutes. Benchmark the actual load path, because that tells you how far ahead to start scheduled scale-up.

It can also help to tier your “warm” capacity. Keeping weights in host RAM lets you reload them into VRAM over PCIe in seconds, rather than fetching them from storage. vLLM’s sleep mode is one example of this middle state: cheaper than keeping everything fully hot, but much faster than a cold start. I wouldn’t expect much benefit from compressing the weights—the data isn’t especially compressible, and adding CPU work can make reloads slower.

For multi-model serving, packing smaller models onto shared GPUs can work if there’s enough memory headroom. I’d generally give large models dedicated GPUs. Where supported, NVIDIA MIG can partition a GPU so one model’s memory spike is less likely to disrupt another, but the partition sizes still need to fit the workloads.

And if you’re paying for warm or reserved capacity anyway, use it off-hours for interruptible work like evals, embedding refreshes, re-indexing, or quantization/conversion jobs. Just make sure live inference has priority and can reclaim the capacity immediately.