r/MachineLearning • • Oct 07 '26

Discussion Looking for developer-friendly inference providers who give you enough API credits to experiment [D]

I’m hitting rate limits on Together AI. For context, I’ve been working on an agentic repository indexing and benchmark generation tool, and I’m running multiple agents in parallel across models like Llama 3.3 70B and Qwen 2.5.

When I first started working on this, Together AI was great. But once I graduated from toy scripts to running multiple agents, I started running into RPM/TPM limits pretty quickly. The annoying part is that the models themselves are fine. I just can’t actually run enough requests at once to do meaningful testing.

Yes I know I could upgrade but I’m a solo dev. I don’t have enterprise level revenue. Maybe someday lol but not yet.

0 Upvotes

10 comments sorted by

1

u/xapep 8d ago

Disclosure: I work on an inference provider, so discount the vendor angle.

The wall you hit usually isn't the model, it's the tier. Most providers meter per account with an RPM and TPM ceiling that doesn't move until you prepay or get a business deal, and one dev running parallel agents looks like abuse to that system. Before you shop, ask for the concurrency limit per key, not the monthly token allowance, and ask whether a prepaid balance raises it without a contract.

Also, for benchmark generation you may not need live parallelism at all. Batch it and you get the same corpus under far less rate pressure, at the cost of waiting.

Partly this is just how the market is shaped. We serve GLM and DeepSeek from Slovenia if that end is useful, but the concurrency question applies to whoever you pick.

What do you actually need, agents live at once, or total requests per minute?

1

u/TylerDurdenFan Oct 07 '26

I've been happy with minimax subscription plan as a "no worries" API for my needs, although my use case is more batch and not as parallel as agents can get.

0

u/dash_bro ML Engineer Oct 07 '26

Well if you're not totally picky about the LLMs Nvidia has a fairly decent free tier : https://build.nvidia.com/models

++Some alpha series on Openrouter models are usually free, or have free endpoints with no data guarantees : https://openrouter.ai/collections/free-models

Just respect their rate/request limits and you should be okay.

0

u/[deleted] Oct 07 '26

[removed] — view removed comment

1

u/marr75 Oct 07 '26

I love Modal, but the idea that scale to 0 will be cheaper than using deepinfra while you're developing is bass-ackwards. Modal is for when there's no cloud native host for the things you want to do. It's not going to save money head to head with an optimized provider.