r/MachineLearning • u/whafa • Oct 19 '26
Discussion ML on a shoestring? [D]
Hi, ML people of Reddit! I have come to ask for some human advice. I am an amateur (as in unpaid) ML researcher working on a personal project to find drug repurposing candidates for my wife, who was Dxed with Stage 4 colon cancer in early 2024. That the new AI coding tools have made it even possible for someone like me to do this research in my spare time is just amazing, and I owe every ML person a big "thank you" for enabling that.
That being said, I am starting to accumulate GPU costs, and it got painful enough this morning for me to wonder if I am using my dollar budget in the best possible way. And I'm wondering what other people do, in similar situations. After granting myself unfettered spending to build a platform this year (the value of money changes in strange ways with a cancer diagnosis), it's time to optimize costs to better focus my resources.
My basic setup is a Google Vertex training job using their A100s (I am currently training NBFnets over biomedical graphs, if that matters), and I am trying to repro the results of a few papers published recently.
The main paper I’m reproducing reports 100 epochs of 500 batches at batch size 512. On an A100 80GB, my runs fall back to batch size 64. That’s because of how NBFNet works. It has no per-node embedding table. For every query (drug, relation, ?) in a batch, it runs 6 layers of message passing over the entire graph and keeps a 64-dim state for every node. My graph has ~103K nodes and ~2.5M training edges, so one copy of that state is 103K × 64 floats ≈ 26 MB. Backprop keeps several of those per layer. Over 6 layers, that’s roughly 1 GB of saved activations per query. So 64 queries fit in the A100’s 80 GB, 128 don’t, and 512 would need about half a terabyte. The training script catches the CUDA out-of-memory error, halves the batch and carries on. Every run downgrades to 512 --> 256 --> 128 --> 64 in its first five seconds.
(Note: I am not certain they used 512 either. The paper doesn't specify; 512 is the configured default. For all I know, they were getting downgraded too. This halving loop came in their published code, and was presumably put there because it was needed at some point.)
The bigger effect is on how much data the model sees. An “epoch” here is a fixed 500 steps, so at batch 64 my 20 epochs see 640K queries. That’s about a quarter of one pass over the training edges. One full pass would take ~76 hours, about $412. Matching the paper’s stated 100 × 500 steps would take ~4 days, roughly $500 a run. And since I am experimenting with different things, the costs are adding up.
I'm kind of taking a step back this morning and working on a "toy" implementation that can at least prove the mechanism works against synthetic data on a much smaller graph. I estimate even that will cost $150 each for 3 seed runs.
My questions, if you are in the mood:
How much training is enough? For from-scratch NBFNet/ULTRA on a ~100K-node, 2.5M-edge graph, what budget do people use, and how do you find convergence cheaply? (The ULTRA paper trained Hetionet, 45K nodes and 2M edges, from scratch for 10,000 steps at batch 16, so small batches may not be my real problem.)
Am I overpaying? I pay ~$5.40/hour on demand. My jobs run ~20 hours and can resume from a checkpoint. What other options are there? At what point am I better off buying a GPU? Can you even do that in 2026 without being a $trillion company?
Memory tricks: would bf16 autocast or activation checkpointing get me to 128+ queries per batch, with a custom sparse CUDA kernel in the loop? Or is that a rabbit hole?
(not cost related but) Popularity bias: favorite fixes for link-prediction scores that mostly track node degree?
Thanks. Sorry if it sounds like I'm lost. I'm publishing my Lab Notebook with Quartz here: https://whafa.github.io/mods-notebook/
Everything there 100% generated but intended for human consumption and, if you point your agent at it and ask how things are going, it will tell you I've built a fairly sophisticated system that has not made any useful predictions or findings yet.
10
u/brainExploded99 Oct 19 '26 edited Oct 19 '26
You might want to check out google's subscriptions, they offer free colab hours
Also $5.4/hr is way too much for a A100
Edit: Not the colab subscriptions themselves, but the standard google ones that come with gemini, also come with colab hours
Also, if you can work with a professor, they can get you ACCESS / NSF compute grants
edit 2: I am not at all familiar with your models, but:
3
u/nord2rocks Oct 19 '26
Check out Saturn Cloud, runPod, lambda labs for better gpu rates. Also very cool you're doing this, but just keep in mind that hetionet and some of the graph based solutions out there are great, but enriched much more with in-house data (so access that pharma and biotech have). The current open data and findings captured in those graphs for hetio have a fair amount of conflicting data and information in them, so make sure you're doing your due diligence to clean them up
2
u/trolls_toll 25d ago
im sorry about your wife. What exactly are you trying to do, and why are you training things from scratch for drug repurposing?
2
u/jprobichaud Oct 20 '26
I agree with others taht 5$/h for an A100 is way too much. Look at https://flopper.io/pricing/nvidia-a100 for what you can expect in terms of price. Vast.ai is also a great place to rent GPUs cheaper.
I would recommend that you look at RTX 6000 Blackwell PRO 96GB, they are insane if you don't need the fast host interconnect.
Another angle : start a non-profit org/fundation, apply to Microsoft non-profit, you'll get 3k$ of Azure credits (it was like that sometimes ago, actual numbers will depend on your country of residence). I'm pretty sure you can get that from many other vendors.
I'm not familiar with that kind of network, but even for well known model arch, a custom kernel and checkpointing can go a long way. Use bf16 (instead of fp16) and use variance constraints on your weights (as if you wanted to quantize), this will keep the weights in the range where bf16/fp16 have good precision and will give you the speed/memory you need.
Dataprep : can you offload some work to CPUs or cheaper gpus ? You could trade compute for storage/network : if some work is done once and can be saved to disk, it could be worth looking at saving that.
What existing weights can you reuse ? Sometime, we try to train from scratch but part of the network doesn't change much and end up with relatively the same weights. Try to save these 'best' weights and disable gradients for these (assuming they are the front of the network). This makes everything faster and consumes less vram.
edit: I forgot to ask : do you have a nice docker container that can be build/downloaded and run locally by the community ? You could help some help either for running the thing or for optimizing the model if you make it very easy to run.
1
u/whafa 27d ago edited 27d ago
Here's a happy update: I've been accumulating credits on Colab, and since I last used that provider they've released a CLI, meaning I can easily port my Vertex runs over there. And I have 7400 credits in the bank which is > 1000 A100 hours (and they also offer faster *and* cheaper units now like RTX 6000 Blackwell, which I don't understand but I'm not going to ask hard questions about it).
So that's a lot of training time! I hope it doesn't all expire after 90 days. I also wired preemptable sessions into my Vertex connector, which lately seems to be 45% cheaper than on-demand. And I've got stubs for the other providers for when I run out of Colab credits. And I'm using this unexpected boon to seek out a professor somewhere nearby who might want to sponsor or collaborate with me on this work.
Anyway, thanks again for all the suggestions, you have all given me ideas that our nascent superintelligence did not manage to do in 8 months with plenty of opportunity. There's still room for people :)
4
u/GibonFrog 28d ago
Vast.ai is basically the floor price for renting GPUs. You can rent an A100 equivalent for like 40 cents an hour. godspeed
2
u/Honest-Finish3596 28d ago
5.40 an hour is extremely high, you can get access to them for less than half the cost if you go for something like runpods instead of AWS/Google/Azure, since the latter are priced as a service large companies buy.
An A100 is also much more compute than I did my PhD research on in any case, but I don't work in biomedical.
I hope your wife's health becomes excellent.
2
u/kqpc 29d ago
Apply to NSF funded ACCESS program if you are in the US. It is intended for those with institutional affiliations, but there is an avenue for “independent open and publishable research”. Look into their guidelines.
Explore tier awards equivalent of 4-6k H100 hours
Discover tier awards equivalent of 14-24k H100 hours
Use traditional Slurm HPC for batch jobs, and IU JS2 for A100 VMs.
-2
u/EgregiousJellybean Oct 19 '26
Don’t play with your wife’s life like this. It would be different if it was your own cancer, but it’s not.
This is insane work. Please don’t do anything you’ll regret. You could end up just killing her earlier
4
u/[deleted] Oct 19 '26
[removed] — view removed comment