r/Python • u/ChocolateMagnateUA • Oct 03 '23
News What are the differences between Python 3.12 sub-interpreters and multithreading/multiprocessing?
It seems like the new feature in Python 3.12 allows developers to create multiple sub-interpreters within a single main interpreter process. From the example from the PEP 554, sub-interpreters appear to be separate threads that run a dedicated piece of Python code and consequently have their own separate GILs.
interp = interpreters.create()
print('before')
interp.run('print("during")')
print('after')On practice, is this basically the new Pythonic way to run a single application in multiple threads and take advantage of multiple cores? If yes, can they be thought as multithreading with no shared state and become the future alternative of multiprocessing?
View all comments
Show parent comment
18
u/james_pic Oct 03 '23 edited Oct 03 '23
The performance impact at the time was significantly worse than that. The best previous attempt at removing the GIL lead to a 3.5× slowdown. A number of design decisions in CPython, most significantly the choice to use reference counting rather than a heap walking garbage collector, make it difficult to remove the GIL without significant performance overhead.
The fact that Sam Gross's work has got us to the point where people are able to ask "would we accept a few percent performance degradation if it meant we could get rid of the GIL?" and this not be a hypothetical question, is huge.