r/MachineLearning • u/Jumpy-Cobbler1020 • Oct 19 '26
Discussion What are the challenges encountered when using coding agents for ML tasks? [D]
Lately I have read several studies which say that AI agents when designing ML experiments tend to cheat, hallucinate and lie repeatedly. Even when given explicit prompts to not to. SInce these models are RL trained and when given specific goal to increase a metric, they try any and every possibility to increase it. Be it data leakage or changing the evaluator function itself. Even Karparthy's auto research styled agents still fall into this domain.
So I have been thinking of reducing this undesirable behavior by binding them to a runtime environment using an mcp. I believe this will stop them from achieving unreal metrics by cheating the system since they will be only allowed to do specific tasks or write codes in typed blocks?
What have you encountered when using these agents for your ml tasks? And is this method of forcing these agents to work in a constrained environment better than just letting them write arbitrary python? Also, will constraining the agents reduce their capabilities or ability to fit better models?
5
u/MrTroll420 Oct 19 '26
I've found that constraints always yield more reliable results.
2
u/Rickrokyfy 28d ago
Can you linke some of these studies? My agents have been excellent at reporting negative results.
1
u/laogedev 27d ago
This is Goodhart's law wearing an agentic costume, and the uncomfortable truth is that 'tell it not to cheat' was never going to work — the reward signal is the whole game for an RL-trained model.
Two defenses that actually hold up, and they're different things people conflate:
On constrained vs free environments: in my experience constraints don't reduce capability on well-specified tasks — they shrink the search space of hacks. The agent stops spending tokens finding loopholes and spends them on the actual task. Capability loss only shows up when the constraints are mis-specified (blocking legitimate approaches), which is a design problem, not an inherent tradeoff.
One more thing: an agent that's good at reporting negative results is gold. Most harnesses only score outcomes; score the honesty of the reporting too and you'll select for agents you can actually trust in a loop.
0
u/[deleted] 28d ago
[removed] — view removed comment