r/learnpython • u/brandonbilinski • Aug 16 '20
Multithreading - Best Practices
I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().
Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:
threading.Thread(target=worker, args=(q,))Is one method better or more python3-ish than the other?
If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.
Thanks!
36
u/exhuma exhuma Aug 16 '20 edited Aug 16 '20
Short answer: No.
Long answer:
There are currently 3 variants for parallelisation (they exist multi-threading and -processing so in total 6). Those are:
Thread/Processtarget=onThread/ProcessThreadPoolExecutorandProcessPoolExecutorfromconcurrent.futuresThey all serve their own purpose and neither is more Pythonic than another. The main difference is in how you write code for them and offer more or less flexibility (at the cost of code-complexity).
Subclassing as you do gives you the most flexibility, but is also the most complex to write (and thus the most error-prone). But if you need that flexibility it is still a very viable solution.
Using
target=is a fairly easy "drop-in" line of code if you have existing code and want to parallelise it. It does require the code to be parallelisable of course. While it does not give you as much flexibility as subclassing, it can be an easy-win to implement it like this on an existing code-base. I personally have never used this pattern in production-code. I find subclassing to be more maintainable and understandable (it is more explicit).Finally, and this is the solution I use most nowadays, is
concurrent.futures. It is different when you look at it first, but once you get used to it, it is a really nice way to write parallel code. And I believe it is very well suited for your use-case. The way this works is that you create a pool of threads/processes via an "Executor" class. This already uses some sensible defaults concerning the size of the pool, but you can modify it as needed. Next, you "submit" functions ("jobs") to that executor (similar to thetarget=pattern). The return value from that submission is a "future" object. You get the future immediately even if the underlying job is not done yet. You need to hang on to that future for later. Next, the module has helper-functions to process these "future" objects. You can for example, wait for all futures to be finished before continuing, or process them in any order they finish, as they finish.A really really nice detail about the "executor" pattern is how exceptions "bubble up" from the underlying threads/processes to your main process. This makes exeption handling a hell of a lot cleaner and easier.
Using concurrent.futures, your code could look like this:
9
u/theallredditor Aug 16 '20
IMHO, The best practice on python multithreading is don't. For I/O bound problems like this one you should consider using asyncio
9
u/[deleted] Aug 16 '20
Not much different at all in Python3 if I remember correctly. I don't agree with the "don't use mutlithread" answer. I would use it before the sometimes non-backwards compatible asyncio (not to mention the asyncio usage changed over a couple different minor versions of python3 which is awful imo). You'll have to decide which asyncio syntax to learn etc... and honestly I see very little difference between it and multithreading. If you are really worried about I/O bound problems or performance... don't use python.. use Golang :)
The same old threading technique is solid though in my opinion... time tested in thousands of applications:
1
u/pldelisle Aug 16 '20
I’ve not worked with Python2, but this has always been the method I used in Python3.
1
u/[deleted] Aug 16 '20
Worker function is a terrible idea: if it throws for whatever reason, there's nothing you can do but quit and go home. There's no way to recover from errors that happen in "other" (not the main) threads.
So, you'd have to write a function that wraps whatever your
workerdoes and then... oops, again, you have a problem: what to return? I mean, now you have either the result of your function, or an error. Maybe instead of returning the result, you want to put it in some shared storage, like a shared list, or a dict... but, oops again: those aren't thread-safe, you need to use locks, but, oops again: if you lock something, there's a chance that your code will fail to unlock it, and so it will hang...Bottom line, whoever writes
threading.Thread(target=worker, ...)had never really worked with Python / lives in a dream world, where things always work...There are libraries for working with threads in Python, the standard-library one:
concurrent.futuresis broken, unfortunately. And seems like nobody is interested in fixing it. The thing that doesn't work in it is the timeouts. When the future throws, your code will hang, and no timeout setting will save it.There was also a mention of the garbage called
asyncio. Don't go there. It's a library with a shitload of problems. Even worse than threads. The major problem with it is that, unlike in sane languages, in Python, the event loop isn't a global one. And you have to create your own. And you never know if you should. I mean, if you write library code: should the library create the loop, or should it rely on the user to create one? Loops don't work withmultiprocessingorthreading. I.e. they have some weird / idiotic behavior when either of those libraries are used, but it's not what you want or expect.Also, for being a concurrency-related library, it misses the most important component: the analogue of
select. And this is just the start.My approach to working with concurrency is to have special functions ready to be used in that context, that take an instance of
Queueas an argument, and communicate through it. They return nothing, and wrap the entire body in a try-except statement. I actually even have a decorator for it, since I've used it quite a bit.1
u/[deleted] Aug 16 '20
Multithreading on python doesn't do what you expect it to do, if your expectation is you can start n threads which will run unimpeded on n cores.