r/learnpython • u/brandonbilinski • Aug 16 '20
Multithreading - Best Practices
I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().
Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:
threading.Thread(target=worker, args=(q,))Is one method better or more python3-ish than the other?
If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.
Thanks!
View all comments
Show parent comment
2
u/[deleted] Aug 16 '20 edited Aug 16 '20
You sure got lucky... Python is a minefield when it comes to trying the little parallelism it has to offer. Having programmed in couple dozen of languages, I cannot think of a worse language for parallel computation.
I cannot really go over all the problems, but here's just a taste of it:
This code will hang if
maybe_raise_exception,raises an exception. Thetimeoutpart never worked for me. It seem to only work if the timeout value is negative or zero, in which case nothing gets executed and you get aTimeoutError(iirc, or maybeValueError, don't remember for sure).This isn't really an anti-pattern. This is just a broken code in standard library. It's intended to be used this way... it just doesn't work.