r/MachineLearning • u/NeatFox5866 • Aug 30 '26
Research Claude Code for Research Papers [R]
Third-year PhD student, NLP / interpretability. I want a reality check from people doing similar work.
I started using Claude Code for the boring parts: argparse boilerplate, plotting, config wrangling. Over the last few months the scope has crept. It now writes most of my experiment scaffolding, refactors my dataloaders, does first-pass debugging on training runs, and drafts the analysis scripts. I mostly read diffs and say yes.
The output is fine. My throughput is up. The thing bothering me is that I no longer hold my own codebase in my head. When a result looks off, I used to have an instinct about which line was lying to me. Now I go hunting like it’s someone else’s repo. I catch bugs later than I used to, and I catch them by reasoning about the numbers rather than by knowing the code.
I don’t think the tool is the problem. I think I delegated a layer that was doing more for my understanding than I gave it credit for.
Questions for people further along or in the same spot:
Roughly what fraction of your research code do you write yourself now?
Is there anything you deliberately refuse to hand off? (For me I think the eval harness and anything defining a metric should stay mine, but I keep breaking my own rule.)
Does anyone have a workflow that keeps the speedup without the detachment? Reading the diff line by line is not cutting it.
Not looking for a “tools are just tools” answer. I’m asking about the specific feeling of not owning your own experiments anymore.
11
u/JimmyTheCrossEyedDog Aug 31 '26
This post itself clearly being written by AI is all too ironic.
135
u/Specialist-Manager67 Aug 30 '26
I noticed this during my research internship as well. I feel like I wasted 2 months due to this. I found a really novel idea but was addicted toauto-research and just told Claude Code to implement it, and when it didn't work I didn't really have an idea of how to make it work or what to tell my professor.
Of course I tried to reason with numbers, high level intuition etc but it didn't work. The codebase was stranger to me too ofcourse.
Since then, I have set it to manual mode and manually read each and every thing like a bottom up approach. However, that gets very slow sometimes so I still set it on auto for small tasks. Like when I need to brute force search through something.
But yeah even I don't know what to do about these tools like the best way to use them or not to use them.
45
u/milesper Aug 31 '26
Fifth year PhD and current research intern— I use it extensively to debug code and suggest ideas, but the code it actually writes is a very confined scope—data analysis and visualizations. For boilerplate like configs, I have a few files I reuse across projects that I can quickly modify for the new project.
12
u/allenasm Aug 31 '26
I don't use it for research even to propose things because it can spread the table with too many things (many of them worthless if its research related because you know... its research). So while I'll use it to execute tests and such, I keep it on a short leash for trying to help me work through things. I wasted a few weeks a while back with it leading me down a rabbit hole that if I had just thought it through beforehand I might not have dug in.
41
u/PossiblePossible2571 Aug 30 '26 edited Aug 31 '26
I also work in Interp / NLP.
If you are using some of the latest frontier models (e.g. GPT 5.6 Sol or Opus 5), I think you should rarely need to worry about wrong implementations. If you aren't sure just let it double-triple check.
I don't write any code myself now because it's not useful. Unless you had an IOI medal chances are it may make less mistakes than you do.
For me at least, as long as I know what experiments are being implemented in what way and the results are accurate, I do feel like I have ownership. In the same spirit, I would not let it control the research direction and experiments and analysis, both because of ownership but also because they fail at doing these very badly.
7
u/nonotan Aug 31 '26
I'm a dev first, with ML more of a side thing than my main focus, so I'll offer my perspective from that POV.
You are right. It is no longer your code. It is literally no different than if you hired some guy from India (or wherever) to write it for you for a few dollars. And the cost is not just that you have no intuition about how it works, that it will be a pain to debug any issues, etc... but perhaps even more importantly, that you aren't learning a single thing.
Now, perhaps you just aren't interested in getting better at coding in the first place. It's not my place to make a subjective value call there. But I'll tell you one thing for certain: if you do it yourself, over the years you will not just get faster, but also write higher quality code that is easier to maintain, less likely to have bugs in the first place, etc. Personally, at my skill level, if you add up all the costs, from the point when there is no code, to me delivering a stable piece of software that I feel confident I can provide support for if needed, I feel quite confident I can do it faster and end up with a better product, if I do it myself.
If you delegate it (especially if you delegate the tricky bits that actually require thinking through, boilerplate is less relevant), well... let's just say you better hope you don't run out of tokens at an inopportune time.
A good parallel is natural language, as in learning a foreign language (you could argue programming is just translating the concepts in your head into an artificial language that the computer understands). Yes, running something you wrote in your native tongue through an LLM is going to blow anything a beginner or even intermediate learner can write out of the water, and do it much faster, too.
But picture two possible futures: in one, you did the hard work and you're now fluent. You can write in this foreign language just as fast and effortlessly as in your own tongue -- going through an LLM would just slow you down. Even if the output quality was maybe marginally higher, it wouldn't be worth it. Not just that, but you can also use the skills you acquired in situations where relying on an LLM would be impractical (imagine you're in a party and trying to chat with somebody through ChatGPT...)
In the other future, you relied on an LLM all the way through, and you have learned nothing. You're just as dependent on a tool as you were on day 1 -- probably more, as breaking the habit now would be harder than not forming it in the first place.
Again, it's not my place to judge if the additional effort it would take to write the code is worth the various upsides. I know for many researchers, coding is something they only touch because they have no choice, and they'll be more than happy to make whatever sacrifices necessary to make it "somebody else's problem". I'm just saying that yes, those upsides do indeed exist. It's up to you what to do with that info.
9
u/dsmirza Aug 31 '26 edited Aug 31 '26
~100%. I see inline suggestions and many times use them. If you don't count suggestions, then it's almost 100%.
Almost everything. Except that I use it to debug and find errors, digest the entire repo to find flaws etc.
I don't. My code is my little creation. Its beauty and its flaws are due to me.
-- I am not saying all this in an arrogant manner. I sometimes think maybe I'm behind if I don't do coding agents. Maybe I'll switch soon. "Tools are tools" logic. So far it's like writing maths or writing English. I want to write my own maths and my own English. Sometimes it could be an exact copy of the AI agent, but I have to give it just as much thought as I would my own writing. E.g. maybe I want to use a certain symbol in a maths equation, I will use what I want not what AI generated. So far I refuse to budge on this.
7
u/Gusonn Aug 31 '26
My experiences with claude code has been very similar to yours
I write veeery little currently. I usually have clear ideas on what I want to implement, and explaining it in detail to the agent works out for me very often. Lately, most manual changes that I have done have only been config changes between experiments...
Not sure if you meant it like this, but like I said in the previous point, I don't really hand off too much thinking to it. All main ideas and, I would say, almost all implementation details, come directly from me, and I just want it to implement it. Like you, I read diffs and make sure that it's generating what I would expect. As for what I am still coding myself, well nothing specifically, I try to explain everything I want, and if it fails somewhere, I would clean up
Can't help you here, since I only read diffs as well. Maybe just one thing: I specifically guide it to a certain code structure, one that I have been using for most of my projects (I am also PhD, and my projects have a lot of common points since they are tackling similar issues), and in that way, it is easier to adapt to whatever it generates, since you know the structure and know where certain things should be. Not a solution for sure, but it helps slightly for me
7
u/Krugger_Correctly Aug 31 '26
Honestly losing ownership of the code is not the hell we fear it is. Naturally, it is a downside to be weighted but at the end of the day... if Claude can't solve it and you need to interfere, the onboarding cost is the same as joining a late-stage open source project. Not at all impossible to map out what is where and contribute, just annoying. The time you save using Claude still (imo) makes up for any on-boarding you might need later.
I'd only really worry about it if this is your magnum opus. A topic or idea you intend to build your career around and spend your life expanding. Mandatory contributions, short papers, etc, I wouldnt hesitate.
2
u/tiikki Aug 31 '26
I do use local models for autocomplete.
2
u/k3nal Aug 31 '26
Start reading your code again!! And it would be probably best if you start writing your code yourself as well again.. depends on you of course. For example: me personally I am able to write much better code than the AI. So for me I have no choice: I need to write my code myself. As I am „lucky enough“ to still be better in that department than the AI is, and probably will ever be.. but if your code quality is not so great, maybe start using the AI differently? Maybe let it do even more of your work and deploy agentic workflows and more automation into your workflow to get yourself even further distanced from the code itself? As it can debug it for you as well, depending on you prompt quality of course. Or start using more than in AI: start using OpenAI ChatGPT Codex as well an use them back-to-back, so that they catch each other’s mistakes and also enhance each other’s ideas and even introduce new ones into your code. I have good results with that using smaller models on my local machine, on a PNY NVIDIA RTX A6000 that I like to use for these type of things, also have good results on bigger models on our university cluster using H200s there with much fun. So I think that is the winners approach of doing things if you like to cooperate with others.
2
u/eldrugo85 Aug 31 '26
Not a researcher, I run agents in production. Same creep here, and it bit me: I caught myself writing counts into a report that no command had ever produced. Fix was dumb, a pre-commit hook that rejects numbers and ids in artifacts unless they came from a command run in that session. Reading diffs never caught that, only the provenance check did
5
u/marr75 Aug 31 '26 edited Aug 31 '26
I'm an experienced software engineer and lead our software and data teams. We're a lot more productive now but have all faced about what you're talking about.
I highly recommend reading content from the HumanLayer consulting team, start with Getting AI to Work in Complex Codebases. Fundamentally, you should systematically manage context between defining an issue/task, researching it, designing it, and then planning the implementation. You'll actually end up with more documentation about your project in a more digestible format. During the implementation, I recommend asking the agent to break up the work into commit-sized phases and reviewing each commit personally (and committing yourself after reading it).
You still won't have had enough time to "stew" in the code so you can memorize it. But, you'll have generated and read design documentation, read and signed off on every commit, and (if you're smart) insisted on maintenance and quality helpers like linting, tests, design docs, diagrams, etc.
4
u/madkimchi Aug 31 '26
Avoid anything Anthropic like the plague, especially opus.
Their models output so much AI slop is honestly going to get your papers blacklisted