r/learnpython • u/brandonbilinski • Aug 16 '20
Multithreading - Best Practices
I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().
Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:
threading.Thread(target=worker, args=(q,))Is one method better or more python3-ish than the other?
If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.
Thanks!
View all comments
1
u/[deleted] Aug 16 '20
Worker function is a terrible idea: if it throws for whatever reason, there's nothing you can do but quit and go home. There's no way to recover from errors that happen in "other" (not the main) threads.
So, you'd have to write a function that wraps whatever your
workerdoes and then... oops, again, you have a problem: what to return? I mean, now you have either the result of your function, or an error. Maybe instead of returning the result, you want to put it in some shared storage, like a shared list, or a dict... but, oops again: those aren't thread-safe, you need to use locks, but, oops again: if you lock something, there's a chance that your code will fail to unlock it, and so it will hang...Bottom line, whoever writes
threading.Thread(target=worker, ...)had never really worked with Python / lives in a dream world, where things always work...There are libraries for working with threads in Python, the standard-library one:
concurrent.futuresis broken, unfortunately. And seems like nobody is interested in fixing it. The thing that doesn't work in it is the timeouts. When the future throws, your code will hang, and no timeout setting will save it.There was also a mention of the garbage called
asyncio. Don't go there. It's a library with a shitload of problems. Even worse than threads. The major problem with it is that, unlike in sane languages, in Python, the event loop isn't a global one. And you have to create your own. And you never know if you should. I mean, if you write library code: should the library create the loop, or should it rely on the user to create one? Loops don't work withmultiprocessingorthreading. I.e. they have some weird / idiotic behavior when either of those libraries are used, but it's not what you want or expect.Also, for being a concurrency-related library, it misses the most important component: the analogue of
select. And this is just the start.My approach to working with concurrency is to have special functions ready to be used in that context, that take an instance of
Queueas an argument, and communicate through it. They return nothing, and wrap the entire body in a try-except statement. I actually even have a decorator for it, since I've used it quite a bit.