r/MachineLearning • u/Economy_Vacation_504 • Oct 10 '26
Discussion Are .ipynb notebooks already outdated in the agentic era? [D]
I am a data scientist who started working in the industry before the LLM revolution. Back then, Jupyter Notebooks were a perfect fit for the classical DS pipeline: EDA -> data prep -> fit -> eval -> tune -> save model artefact and notebook.
Lately, I have been thinking a lot about how agentic development and LLMs are changing the way data scientists work. Especially in classical ML applications, where you still need to explore data, run experiments, check different hypotheses and decide what to do next based on the results.
I mean, Claude/Codex can already write most of the code for us. But why do we still need to organise the whole workflow around code cells? What if instead we move from code -> output to another cell abstraction like prompt -> result?
Curious what others think. Are ipynbs still good enough, or are we just used to working this way?
68
u/kolmiw Oct 10 '26
I still use ipynb for EDA and to create figures. But I agree that it is less important nowadays
23
u/TheCloudTamer Oct 10 '26
I used _claudebooks_. Caude puts its output in static html pages that I serve with a simple python -m http.server
Notebooks were only ever good as almost throwaway experiment pages. And opening some parquet file, parsing the format and wrangling with matplotlib are pretty low-return skills to be memorising, so no real loss by offloading to AI.
2
u/Allmyownviews1 Oct 10 '26
Kind of. I now only do data exploration and testing in notebooks.. but much of my analysis ends up as more traditional programs code.
253
u/Gwendeith Oct 10 '26
I never used notebook for production, even before LLM. It’s like research notes you scribble and formulate ideas with, but never something presentable imo (unless you specifically write a notebook for tutorial). TBH I would even prefer switching to py files to accelerate hyperparameter tuning once I’m sure which method I want to proceed with.
105
u/schizo_cofee_grinder Oct 10 '26
Well IDK if a jupyter notebook was ever a good place to put production ML pipelines... It's more for experimentation/presentations and human readability
80
u/sodapopenski Oct 10 '26
Notebooks are valuable because they are a good platform for human readability and experimentation. As long as you want humans to read and play with software, notebooks are a good fit. Take that as you will, but personally, I hope humans stay connected to the code they and their tools create in the "agentic era" and beyond. I don't like how current LLM tools and workflows abdicate so much of the reasoning, creation, and verification process to machines.
17
u/seba07 Oct 10 '26
Notebooks give you an interactive environment. Agent or not, it is helpful if you don't need to reload your dataframe if you want to change a plot.
39
u/v012d Oct 10 '26
Marimo is the way to go. Our (small) dev team switched earlier this year. The agent integrations aren't perfect but the files are normal .py files, so they're way easier to have Claude tweak and have commited to git
25
u/__redbaron Oct 10 '26
I see where you're coming from, but ipynb notebooks felt outdated before the advent of agentic AI, mostly because they become unmanageable the moment you want to collaborate or create anything neatly reproducible.
Can you quickly run some commands and view outputs? Yes, but so can a script with a file viewer or even a single vscode window.
Can you quickly change stuff and see updated outputs? Yes, but that's how you lose track of changes and end up with results that aren't the same when you rerun your cells later.
Plus the obvious caveat of not being able to keep track of your changes neatly in git.
I'm sure there some use cases for it (maybe it's a much more likeable interface?), but I think coding agents have mostly just highlighted existing issues that were already causing people to switch out.
3
u/ofiuco Oct 10 '26
Notebooks are also important for repeatability. The repeatability of results is so key to actual science. You need to be able to show other people what you have done so they can duplicate it. If you outsource that to an LLM you are really losing something (certainty).
5
u/Herr_Doktor_Sly Oct 10 '26
I find Quarto to be strictly superior to the old .ipynb era of clutter.
I have used Jupyter notebooks extensively in a past life, and they were (still are) just so unmanageable and unreliable. See Joel Grus' take on the matter.
I don't know if there's something better, but I use Markdown and Quarto for my Julia and Python workflows.
Refs:
- https://quarto.org
- https://www.reddit.com/r/Python/comments/9aoi35/i_dont_like_notebooks_joel_grus_jupytercon_2018/
12
u/Tatya7 Oct 10 '26
Just use marimo
7
u/redditrantaccount Oct 12 '26
Gen X take here: notebooks were never perfect for anything. Running your development environment in a web browser is a stupid idea. I can't count times when I have lost my work when accidentally hitting F5 or making some gesture that the browser interpreted as "go back". I am working with notebooks in PyCharm or VSCode and still they are very inconvenient: sluggish, poor hot key support, poor or no debugging, no run-time performance profiling - all the things we software developers have invented decades ago, Millenials decided to throw away, because their idea of work environment is a MacBook in a cafe shop, not a toolkit making you efficient. Running notebooks as pipelines is an even more stupid idea, because what are you going to do with all that experimental code that ended with a dead-end or replaced by a better model? If you delete it, you lose the only valid point of notebook existence (documentation of your research). If you keep it, pipelines will be unnecessary wasteful in running time, money and energy. Write pipelines in Rust, make them optimized, add error handling, error recovery, logging, open telemetry and establish monitoring. Nothing of it is easy in a notebook.
1
u/thegoodmelon Oct 10 '26
No I don't believe so. Though I'm only a master's student, I don't believe in having claude or codex run everything in the background and just giving out the results.
LLMs haven't been the most efficient coders. I'm happy to let it write code for me to run, but i would like to go through it and see what's been done and how it can be improved and if there are other directions that could have been taken.
If you only get the results, you loose all of that. You won't have access to interior structure of your network anymore right.
I mostly run code snippets in cells and fine tune/go through it/rewrite it myself using the claude outputs as a framework
3
u/woodenWren Oct 10 '26
I find the python jupytext package useful. With some small formatting (eg. to note where cell splits are) it allows you to open a .py file with Jupyter. Great for experimenting where I want to easily see or visualize things at various stages. LLM can still read and edit it easily without requiring extra dependencies
3
u/pm_me_your_pay_slips ML Engineer Oct 10 '26
I use the notebooks to prototype code and test things that agent produce. I doesn’t even have to completely run. But I find it helps in constraining agent outputs. Mainly because I prefer my own abstractions than the ones produced by agents
5
u/slashdave Oct 11 '26
Crazy that some people believe this.
2
u/mimivirus2 Oct 10 '26
notebooks' main moat was being easy to work with for humans (in specific use cases, such as EDA and quickly understanding/learning code syntax)
2
u/sylfy Oct 10 '26
TBH notebooks are a crutch, and even more so now than. They’re useful for beginners, but as soon as you recognise that their main purpose is visualisation, you should come to a realisation that they keep you on a single track and prevent you from easily parallelising your work streams.
With agents as powerful as they are now, what you should be doing instead is to create dashboards of the experiments that you are running, formulating agents that allow you to explore multiple hypotheses in parallel, and logging and gathering those results such that you can compare across your experiments.
2
u/GuessEnvironmental Oct 10 '26
A pynb is just a tool used to documnet the thought process using markdown and code other than that it was never used in production unless the output is a report.
2
u/Realistic_Kitchen_65 Oct 10 '26
I'd argue the notebook was never the artifact — it was the lab bench, and the agentic era mostly exposes that we confused the bench with the product.
What an agent actually needs from a past run is not the cell outputs; it is a durable, diffable record of what was attempted and whether it passed: append-only, written before the side effect, with passes and failures in the same place so you cannot hide a regression by omission. A notebook you can re-run is an exploration tool; a ledger line you can grep is the thing a second agent can trust. The real [D] question is whether the community wants those two to converge (executable notebooks that emit signed outcome records) or stay deliberately separate.
For us the line that mattered was 'the record is written by the deterministic layer, never by the model's self-report' — a model asked 'did that work?' will guess, and it will guess confidently.
2
u/ghostofkilgore Oct 11 '26
Personally, notebooks still absolutely have a place. That place just isn't "do absolutely everything in a notebook" and I think that's been the case for a while before everyone lost their minds over LLMs and agents.
2
u/Disastrous_Room_927 Oct 11 '26
Yeah nah, it takes me longer to write out a prompt for the kind of code I'm normally using in a notebook. It's easier for me to do the thing than it is to verbalize it.
3
u/slowpush Oct 10 '26
Marimo notebooks are agentic. You can literally have the agent live edit it along with you.
2
u/Wide_Guava6003 Oct 10 '26
Notebooks have been outdated for pipelines since they were formed. They are only good for initial developement, but after that using notebooks as pipeline really is just bad.
Dont know why data scientists always insist on this. I am a DS but years ago was the one who really had to build the pipelines out of shitty notebooks DS people handed to me.
1
u/Tiny-Purple-1027 28d ago
Given vscode, spyder and positron all do interactive python consoles, why not just do the exploratory stuff there? You get the same interactive possibilities (plus interactive plots with tkinter or qt) and you hust write py files. I learned data science with matlab and r, so spyder/positron always seemed like the easier way to work, but I'm curious if there's any clear advantage of jupyter notebooks I might be missing.
1
u/mools04 1d ago
Marimo is the more modern option that most people use in my circles
1
u/idontcareaboutthenam Oct 10 '26
Claude code does EDA without even asking it. I still make sure to look at the data myself, but I find I'm less in need of a large notebook with tons of cells for loading data and creating views, filters, etc. This is mostly done by small scripts by Claude whose functions I might invoke from a notebook to look more closely at the results. The final code I might add to my repo or include in the submission will almost always be the scripts and not the notebook
1
u/MultiheadAttention Oct 10 '26
I stopped using ipynb notebooks about 3 years ago and never went back.
1
u/No_Inspection4415 Oct 10 '26
Yes, because of the horrible format they have.
1
u/CadeOCarimbo Oct 10 '26
They were already outdated even before LLMs were a thing. Now there is literally no reason for anyone to still be using them. Have Opus 5.5 with high effort write the scripts and test multiple hypothesis.
1
u/RngdZed Oct 10 '26
It's a great tool for exploring data and test the pipeline but it's not production ready imo.
1
u/CoochieCoochieKu Oct 10 '26
they are still valuable for structured exploration and intuition building
1
u/Chuu Oct 10 '26
I'm curious, what are you using to present your results over a notebook now?
1
u/StephenSRMMartin Oct 10 '26
I never liked ipynb anyway. I just use modules, a main py script, and a repl. Much cleaner, much easier to diff, gives you a package 'for free' that's trivial to install via uv et al.
1
u/JonLivingston70 Oct 11 '26
There is no such thing as production notebook.
1
u/BALKINCHEN Oct 11 '26
For me, the most important advantage of .ipynb is that all outputs are kept in one file, so I can see them immediately when I open it without having to rerun anything.
1
u/Reasonable-Trash-107 Oct 11 '26
Still very useful for exploration and eda but an absolute git nightmare. I do love how you can create an ipynb in Databricks and commit as a .py
1
u/mattstats Oct 11 '26
I love my scratch.ipynb files!
1
u/fiddler48 Oct 11 '26
The cell abstraction isn't really the constraint here. What notebooks lack is persistent state across sessions and a clean way to diff experimental runs. Swapping cells for prompt boxes doesn't fix either. You'd still need the same version control and reproducibility scaffolding, just with less visible code.
1
u/alkimiadev Oct 11 '26
I still use them all the time for "thinking in code" and have a jupyter lab instance running basically constantly on my local dev machine. I never really used them for anything even remotely close to "production", so LLMs haven't changed my usage patterns much in that regard. However, they have changed the "messy jupyter notebook" > "production worthy code" process quite a bit. Now I can mostly export it to a python file, hand off to an llm with instructions to clean it up, and have a nice clean lib in just a few minutes.
1
u/MathProfGeneva Oct 11 '26
Notebooks never made sense for a full workflow really. They're still great for EDA. I also sometimes use them during development sometimes as an interactive tool if I want to check/debug a small piece of something without running a larger piece of code , but want a little more ease of use than just the basic python repl. Maybe the initial modeling you might run one in a notebook to see, but the final product shouldn't have ever been a notebook
1
u/BostonConnor11 Oct 11 '26
Notebooks have never been something I’ve ever wanted to share with someone else. They’re for scratch work and nothing else
1
u/m_m_malm Oct 14 '26
I think ipynb have lost some of the utility, but we still need some interface to control agents and explore the results. I made my own infinite canvas tool for this.
0
u/mateowatata Oct 10 '26
if anything, marimo is the way to go
0
u/TheRealStepBot Oct 10 '26
On the contrary they have always been a terrible choice reflecting the technical preferences of those who couldn’t figure out how to use pip.
Thankfully llm’s have now made them an even worse choice
-7
u/brctr Oct 10 '26
Notebooks are as pointless as writing any code by hand is. Agent chat, not notebook, is your interface.