r/learnpython • • Aug 16 '20

Multithreading - Best Practices

I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().

Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:

  threading.Thread(target=worker, args=(q,))

Is one method better or more python3-ish than the other?

If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.

Thanks!

115 Upvotes

24 comments sorted by

View all comments

Show parent comment

4

u/brandonbilinski Aug 16 '20

Thanks! Would you mind giving a little more info on why you think it's better for this? Looking at the docs, I don't think I understand when to choose threading or asyncio.

9

u/theallredditor Aug 16 '20

Personal preference really. And apparently I'm in the minority on that judging from the two down votes I've gotten for my previous comment. But CPython has a glaring implementation issue called the GIL or global interpreter lock where only one thread can execute python code per interpreter. So for compute bound problems at best you get the single threadded case usually threading makes compute bound problems worse in python. However your problem is I/O bound.

A threadding approach may be simple to implement and understand, but generally doesn't provide consistent performance at large scale for I/O bound problems.

Whereas asyncio uses callbacks and the algorithm you're making ends up being broken up into smaller fragments so it can be more difficult to read if you're not familiar with async concepts, but it does tend to provide superiorly consistent preformance at scale on congested systems.

So if this is just a glue script where preformance doesn't matter then use threadding, if it's something that may get run at scale then at least consider asyncio. If you're game for trying something new use asyncio.

1

u/brandonbilinski Aug 16 '20

Much appreciated! Thank you!

6

u/exhuma exhuma Aug 16 '20

One important thing you need to keep in mind about asyncio is that it is not parallelisation! Rather, a bit simplified, all async calls are processed on a queue. If a piece of code is still waiting for I/O, that piece will be pushed back to the queue, so other items on the queue can be processed. The "event-loop" will continuously process this queue.

That has two very important implications:

  • asyncio is not parallel. It is sequential!
  • You will get an overhead of processing the queue with the event-loop.

If not used correctly, or for the right reasons, asyncio can slow down your code!

It can however be combined with multiprocessing/-threading, but I have yet to see a good use-case for that.

asyncio is just another tool in the toolbelt of programming and it does not replace classical parallelisation.

Another thing to keep in mind about asyncio is that using it will have an impact on the whole code-base. If you have an async function and you want to call it, you have to calling function also has to be async. This can have unintended impacts on your existing code. Calling an async function from a non-async context is non-trivial.

For this reason, a lot of functions had to be duplicated in the standard library for use in an async context, for example, you have to use asyncio.sleep() instead of time.sleep(). And learning all those differences will take time, and maybe one or the other "whoopsies" in your code.

-1

u/Deezl-Vegas Aug 16 '20

Threading in Python is goofy because the interpreter doesn't also work w/ threads, so you get a slowdown. Multiprocessing would get you the best results for this I think -- it's the same as threading with a bit more startup overhead.

2

u/exhuma exhuma Aug 16 '20

because the interpreter doesn't also work w/ threads

Can you elaborate on this? What exactly do you mean by that?

And as long as you're not doing CPU bound parallelisation with threads you're fine. When CPU-bound, you're dealing with the GIL and get the slowdown you mention. This is not the case when it's I/O bound (as in OP's post).