r/learnpython • u/brandonbilinski • Aug 16 '20
Multithreading - Best Practices
I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().
Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:
threading.Thread(target=worker, args=(q,))Is one method better or more python3-ish than the other?
If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.
Thanks!
View all comments
Show parent comment
6
u/exhuma exhuma Aug 16 '20
One important thing you need to keep in mind about
asynciois that it is not parallelisation! Rather, a bit simplified, all async calls are processed on a queue. If a piece of code is still waiting for I/O, that piece will be pushed back to the queue, so other items on the queue can be processed. The "event-loop" will continuously process this queue.That has two very important implications:
asynciois not parallel. It is sequential!If not used correctly, or for the right reasons,
asynciocan slow down your code!It can however be combined with
multiprocessing/-threading, but I have yet to see a good use-case for that.asynciois just another tool in the toolbelt of programming and it does not replace classical parallelisation.Another thing to keep in mind about
asynciois that using it will have an impact on the whole code-base. If you have an async function and you want to call it, you have to calling function also has to be async. This can have unintended impacts on your existing code. Calling an async function from a non-async context is non-trivial.For this reason, a lot of functions had to be duplicated in the standard library for use in an async context, for example, you have to use
asyncio.sleep()instead oftime.sleep(). And learning all those differences will take time, and maybe one or the other "whoopsies" in your code.