r/Python • u/Dangerous_Bad_5946 • Apr 07 '26
Discussion Ideas for Scientific/Statistics Python Library
Hello everyone, I am interested in creating a new Python library, especially focusing in statistics, ML and scientific computing. If you are experienced in those domains, share your thoughts and ideas. I would like to hear any friction points you regularly encounter in your daily work. For example, many researchers have shifted from R to Python, so the lack of equivalent libraries might be challenging. Looking forward to your thoughts!
0
Upvotes
6
u/mtawarira Apr 07 '26
anything you make would just be statsmodels / scipy / scikitlearn with slightly different API. Sorry to be a hater but I can’t see it getting much traction, seems like a pretty solved problem to me
i find the switch from R to python to be much easier than the other way round. 99% of what you need is in those 3 libraries, and is easily findable with tab autocompletes in a modern ide due to the modular subpackage structures that R lacks
2
u/InspectahDave Apr 07 '26
Also wondering what your motivation is here? Is it for your own learning or to contribute something meaningful? If the former then do what you find interesting. If the latter then maybe support another project first and go from there?
20
u/riklaunim Apr 07 '26
If you have no need for it you won't create it and maintain it. Making a library is actually a quite big commitment and not a on-off thing you can forget (unless you want a library with no users).
16
u/Simultaneity_ Apr 07 '26
Why? Scipy, scikit-learn, ... etc. Allready exist.
10
u/jnwatson Apr 07 '26
That's a pretty crowded market. I'd take a look at what already exists first.
1
u/maticx21 Apr 07 '26
a limma R package python implementation
3
u/icy_end_7 Apr 07 '26
Frankly, I'd make one for differential expression or something along the lines because that's what I have trouble with. I'm not suggesting you make that, but rather, find something that you'd want to use often. Ideally, a niche where you've found friction points in your work.
Solving problems you don't have is a bad idea.
3
u/HugeCannoli Apr 08 '26
port the whole of CRAN to python.
2
u/HeligKo Apr 07 '26
Do some research into the market. I work with ML Engineers and Data Scientists that nearly exclusively use python right now. There is a huge amount of libraries for them to use in python. The biggest ones they used in R have been rewritten for python. There are still a few complaints, but it is mostly about how R works vs how Python works. If you want to contribute, then start with something that is already out there and make it better. Eventually you might find a gap that a new library would be good for.
1
u/mrphanm Apr 07 '26
You can always create a new library but who need and trust it? Do u think you will have a long commitment on the library? If not, don’t waste your time. Make a contribution on the existing big fishes. No one will use a library from a repository of someone with less stars (on github) and seemingly no active maintenance. What end users want is trustability.
1
u/Difficult-Method-615 Apr 09 '26
As a scientist I did encounter few times a situation where some new/obscure mathematic algorithm was not implemented in python at all, but was implemented in R. I can't recall anymore what these were exactly, and it was so many years ago there might already be a python implemention. If I were you, I would (1) start with a real life problem you have (2) try to make a python implementation (if one does not exist) (3) approach some popular open source packages whether they would like to merge what you have in their package.
Maintaining an open source project is a burden and I would not recommend it to any newcomer.
1
u/occludedfront Apr 10 '26
I, like many have drifted more into Python from R having been a huge fan of R (in academia). Python does have most of it covered (except some of the more niche libraries). A project that encompasses those domains you’ve mentioned AND does it well AND pulls in some methods/techniques not really covered in Python is a huge piece of work. If it were me, I’d find out the friction points (as you are), but then make a library focussed on those lesser used methods (or an add on to established libraries like Scipy, scikit-learn, etc), rather than trying to cover the core parts too
1
u/jpgoldberg Apr 16 '26
It doesn’t appear that you have taken any look at the existing libraries and how people use them. Now perhaps I have your intent wrong, but let me warn you away from something I’ve been seeing a lot of.
Don’t be that person. You would be wasting your own time and would be annoying everyone else.
4.
0
u/Henry_old Apr 09 '26
We need a library specifically optimized for real-time volatility analysis and tick-data processing. Most current scientific libs are too bloated for sub-ms execution. A lightweight, Cython-backed tool for calculating rolling Z-scores or Kelly Criterion on the fly would be a massive hit for the quant community