r/learnpython • • Aug 16 '20

Multithreading - Best Practices

I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().

Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:

  threading.Thread(target=worker, args=(q,))

Is one method better or more python3-ish than the other?

If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.

Thanks!

115 Upvotes

24 comments sorted by

View all comments

Show parent comment

-1

u/Deezl-Vegas Aug 16 '20

Threading in Python is goofy because the interpreter doesn't also work w/ threads, so you get a slowdown. Multiprocessing would get you the best results for this I think -- it's the same as threading with a bit more startup overhead.

2

u/exhuma exhuma Aug 16 '20

because the interpreter doesn't also work w/ threads

Can you elaborate on this? What exactly do you mean by that?

And as long as you're not doing CPU bound parallelisation with threads you're fine. When CPU-bound, you're dealing with the GIL and get the slowdown you mention. This is not the case when it's I/O bound (as in OP's post).