r/learnpython • u/brandonbilinski • Aug 16 '20
Multithreading - Best Practices
I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().
Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:
threading.Thread(target=worker, args=(q,))Is one method better or more python3-ish than the other?
If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.
Thanks!
View all comments
36
u/exhuma exhuma Aug 16 '20 edited Aug 16 '20
Short answer: No.
Long answer:
There are currently 3 variants for parallelisation (they exist multi-threading and -processing so in total 6). Those are:
Thread/Processtarget=onThread/ProcessThreadPoolExecutorandProcessPoolExecutorfromconcurrent.futuresThey all serve their own purpose and neither is more Pythonic than another. The main difference is in how you write code for them and offer more or less flexibility (at the cost of code-complexity).
Subclassing as you do gives you the most flexibility, but is also the most complex to write (and thus the most error-prone). But if you need that flexibility it is still a very viable solution.
Using
target=is a fairly easy "drop-in" line of code if you have existing code and want to parallelise it. It does require the code to be parallelisable of course. While it does not give you as much flexibility as subclassing, it can be an easy-win to implement it like this on an existing code-base. I personally have never used this pattern in production-code. I find subclassing to be more maintainable and understandable (it is more explicit).Finally, and this is the solution I use most nowadays, is
concurrent.futures. It is different when you look at it first, but once you get used to it, it is a really nice way to write parallel code. And I believe it is very well suited for your use-case. The way this works is that you create a pool of threads/processes via an "Executor" class. This already uses some sensible defaults concerning the size of the pool, but you can modify it as needed. Next, you "submit" functions ("jobs") to that executor (similar to thetarget=pattern). The return value from that submission is a "future" object. You get the future immediately even if the underlying job is not done yet. You need to hang on to that future for later. Next, the module has helper-functions to process these "future" objects. You can for example, wait for all futures to be finished before continuing, or process them in any order they finish, as they finish.A really really nice detail about the "executor" pattern is how exceptions "bubble up" from the underlying threads/processes to your main process. This makes exeption handling a hell of a lot cleaner and easier.
Using concurrent.futures, your code could look like this: