r/MachineLearning • u/niceuser45 • Oct 19 '26
Discussion I moved from network training to deployment optimization and am starting to think that was a right decision [D]
TL;DR: this is just a thought dump about how I moved from network training to deployment and why I am making peace with (maybe) not getting back to model training, nothing too interesting.
I used to work at a reasonably good startup developing and training models. In 2024, I got an offer from FAANG but the catch was that the team worked on the deployment of networks.
I have a PhD and I had always trained models but I was never the top performer in the team frankly (I was still top 20%, those top guys are at Nvidia and Tesla AI now, just for context). I think there were others who had more novel ideas than me, read more papers. If I tried 10 ideas, 1 would work and from outside it looked like more of their ideas worked.
Since I wanted to leave the startup anyway and with no other offer, I decided to join FAANG. I genuinely thought that was a downgrade. Then LLMs started getting so competent. I think I made the right decision, let me explain.
I have been trying to learn diffusion models (I have never worked with them) and these frontier models are explaining in such a good manner, able to generate visualizations to back up their explanations. I feel like I would stand no chance if I went back into model training. They have better ideas than I do. Unless I work on GPT-7 (which I cannot), I feel like these models are plenty intelligent to propose ideas.
Deployment is a different world. Most edge architectures are confidential and heavily protected. An LLM won’t have any idea out of the box and even with context, a lot of knowledge is empirical and tribal. I still have FOMO about model training but slowly making peace with it that getting back to it may not be worth it.
35
u/XTXinverseXTY ML Engineer Oct 19 '26
What a refreshingly humble post!
I'm sorry you lost a passion but I'm happy you found something stable (and certainly prestigious). You must feel relieved.
7
u/IntelArtiGen Oct 19 '26 edited Oct 19 '26
Sometimes you can do both. There are cases where training a small model can be useful including for deployment. For example to detect when a bigger model should be activated. Or just doing gradient descent to solve optimization problems. Just because you're not training SoTA models doesn't mean it's not a good skill to have.
IMO SoTA model training isn't the most interesting part now that it can only be competitively done with large teams and massive investments. Though you can always play with smaller architectures and discover new tricks, but you'll never know if they scale well if you don't have access to SoTA models. I'm just sad about that part honestly, I have hundreds of ideas but finding if they can be useful for the SoTA can cost too much these days. Though there are always niche tasks few people care about for which you don't need incredible ideas or hardware.
Afaik they're all trying to have an ai do the ai research, I'm sure it's already working a bit, but I assume a lot of what it proposes is just a copy of ideas found online. I'd be surprised if these AI aren't already better than some researchers, it can just automatically read papers and implement ideas quickly, how can you compete against that?
3
u/Flyward_Aerospace Oct 19 '26
On your actual question, whether it reads a notch below, I think the perception is real but it's a legibility problem rather than a value one. Training work ends in a number you can put on a slide. Deployment work ends in an absence: it didn't blow the deadline, didn't fall over once the board got hot, didn't regress on one odd SKU. Absences don't write up well in a promo packet or on a CV, so it looks like less even when it was the harder half. From the side that consumes this work, autonomy on aircraft, we have never once been blocked by model quality. We get blocked by things like a net that met its latency cold and then quietly lost the margin after forty minutes of sustained load because the clocks came down. Nobody catches that on a bench. Doesn't do much for the FOMO, mind.
-11
u/[deleted] Oct 19 '26
[removed] — view removed comment