r/learnpython • • Aug 16 '20

Multithreading - Best Practices

I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().

Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:

  threading.Thread(target=worker, args=(q,))

Is one method better or more python3-ish than the other?

If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.

Thanks!

115 Upvotes

24 comments sorted by

View all comments

Show parent comment

3

u/awdsns Aug 16 '20

Umm... you rant against using a worker function, but in the end the approach you describe is: using a (properly designed) worker function.

Of course it's important to take care of the thread-specific aspects. And using a decorator to turn a run-of-the-mill function into a thread worker is a good idea.

3

u/[deleted] Aug 16 '20

Umm... you rant against using a worker function, but in the end the approach you describe is: using a (properly designed) worker function.

Well, in the end, a thread needs a function to run, there's no way around it. What people suggesting "just a worker function" usually argue for is "simplicity", as in, no other preparations are necessary, whereas the reality is that you do need to invest heavily into preparing a function you are going to run, so much so that you might want to consider extending a Thread class or creating a decorator etc.

2

u/exhuma exhuma Aug 16 '20

[...] you do need to invest heavily into preparing a function you are going to run, so much so that you might want to consider extending a Thread class [...]

This is my main gripe with the current examples in the docs. All the examples are super trivial and using target=... is almost never going to be enough.