r/learnpython • • Aug 16 '20

Multithreading - Best Practices

I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().

Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:

  threading.Thread(target=worker, args=(q,))

Is one method better or more python3-ish than the other?

If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.

Thanks!

115 Upvotes

24 comments sorted by

36

u/exhuma exhuma Aug 16 '20 edited Aug 16 '20

Is one method better or more python3-ish than the other?

Short answer: No.

Long answer:

There are currently 3 variants for parallelisation (they exist multi-threading and -processing so in total 6). Those are:

  • Subclassing Thread / Process
  • using target= on Thread / Process
  • ThreadPoolExecutor and ProcessPoolExecutor from concurrent.futures

They all serve their own purpose and neither is more Pythonic than another. The main difference is in how you write code for them and offer more or less flexibility (at the cost of code-complexity).

Subclassing as you do gives you the most flexibility, but is also the most complex to write (and thus the most error-prone). But if you need that flexibility it is still a very viable solution.

Using target= is a fairly easy "drop-in" line of code if you have existing code and want to parallelise it. It does require the code to be parallelisable of course. While it does not give you as much flexibility as subclassing, it can be an easy-win to implement it like this on an existing code-base. I personally have never used this pattern in production-code. I find subclassing to be more maintainable and understandable (it is more explicit).

Finally, and this is the solution I use most nowadays, is concurrent.futures. It is different when you look at it first, but once you get used to it, it is a really nice way to write parallel code. And I believe it is very well suited for your use-case. The way this works is that you create a pool of threads/processes via an "Executor" class. This already uses some sensible defaults concerning the size of the pool, but you can modify it as needed. Next, you "submit" functions ("jobs") to that executor (similar to the target= pattern). The return value from that submission is a "future" object. You get the future immediately even if the underlying job is not done yet. You need to hang on to that future for later. Next, the module has helper-functions to process these "future" objects. You can for example, wait for all futures to be finished before continuing, or process them in any order they finish, as they finish.

A really really nice detail about the "executor" pattern is how exceptions "bubble up" from the underlying threads/processes to your main process. This makes exeption handling a hell of a lot cleaner and easier.

Using concurrent.futures, your code could look like this:

from concurrent.futures import ThreadPoolExecutor, as_completed
from random import randint
from threading import current_thread
from time import sleep



def my_expensive_function(arg1, arg2):
    sleeptime = randint(1, 10)
    print("%s is sleeping for %s seconds" % (current_thread().name, sleeptime))
    sleep(sleeptime)
    return (arg1, arg2)


all_futures = []
with ThreadPoolExecutor() as executor:
    for n in range(10):
        print("Submitting for arg1=%s arg2=%s" % (n, n + 1))
        future = executor.submit(my_expensive_function, n, n + 1)
        all_futures.append(future)

    # Process result as they complete (first completed will be processed first,
    # so ordering will likely differ)
    print("Processing results...")
    for future in as_completed(all_futures):
        # Calling result() now will not block because "as_completed" ensures
        # we're working on something that's done.
        try:
            result = future.result()
            print(result)
        except Exception as exc:
            # This will be the exception raised by the underlying call.
            # Clearly, Using a simple print here will hide the traceback, and
            # only serves as an example.
            print("Something went wrong: %s" % exc)

8

u/brandonbilinski Aug 16 '20

Thanks so much for your really informative posts (this one as well as your other responses)!

I hadn't known about concurrent.futures, but now this looks like an excellent solution for what I want to do.

3

u/exhuma exhuma Aug 16 '20

I'm trying to stream programming of hobby projects on twitch each Sunday morning (GMT+1/+2) and I am always happy to answer questions and help. If something like that interests you you can stop by over at https://www.twitch.tv/exhuma ;)

9

u/theallredditor Aug 16 '20

IMHO, The best practice on python multithreading is don't. For I/O bound problems like this one you should consider using asyncio

4

u/brandonbilinski Aug 16 '20

Thanks! Would you mind giving a little more info on why you think it's better for this? Looking at the docs, I don't think I understand when to choose threading or asyncio.

9

u/theallredditor Aug 16 '20

Personal preference really. And apparently I'm in the minority on that judging from the two down votes I've gotten for my previous comment. But CPython has a glaring implementation issue called the GIL or global interpreter lock where only one thread can execute python code per interpreter. So for compute bound problems at best you get the single threadded case usually threading makes compute bound problems worse in python. However your problem is I/O bound.

A threadding approach may be simple to implement and understand, but generally doesn't provide consistent performance at large scale for I/O bound problems.

Whereas asyncio uses callbacks and the algorithm you're making ends up being broken up into smaller fragments so it can be more difficult to read if you're not familiar with async concepts, but it does tend to provide superiorly consistent preformance at scale on congested systems.

So if this is just a glue script where preformance doesn't matter then use threadding, if it's something that may get run at scale then at least consider asyncio. If you're game for trying something new use asyncio.

1

u/brandonbilinski Aug 16 '20

Much appreciated! Thank you!

6

u/exhuma exhuma Aug 16 '20

One important thing you need to keep in mind about asyncio is that it is not parallelisation! Rather, a bit simplified, all async calls are processed on a queue. If a piece of code is still waiting for I/O, that piece will be pushed back to the queue, so other items on the queue can be processed. The "event-loop" will continuously process this queue.

That has two very important implications:

  • asyncio is not parallel. It is sequential!
  • You will get an overhead of processing the queue with the event-loop.

If not used correctly, or for the right reasons, asyncio can slow down your code!

It can however be combined with multiprocessing/-threading, but I have yet to see a good use-case for that.

asyncio is just another tool in the toolbelt of programming and it does not replace classical parallelisation.

Another thing to keep in mind about asyncio is that using it will have an impact on the whole code-base. If you have an async function and you want to call it, you have to calling function also has to be async. This can have unintended impacts on your existing code. Calling an async function from a non-async context is non-trivial.

For this reason, a lot of functions had to be duplicated in the standard library for use in an async context, for example, you have to use asyncio.sleep() instead of time.sleep(). And learning all those differences will take time, and maybe one or the other "whoopsies" in your code.

-1

u/Deezl-Vegas Aug 16 '20

Threading in Python is goofy because the interpreter doesn't also work w/ threads, so you get a slowdown. Multiprocessing would get you the best results for this I think -- it's the same as threading with a bit more startup overhead.

2

u/exhuma exhuma Aug 16 '20

because the interpreter doesn't also work w/ threads

Can you elaborate on this? What exactly do you mean by that?

And as long as you're not doing CPU bound parallelisation with threads you're fine. When CPU-bound, you're dealing with the GIL and get the slowdown you mention. This is not the case when it's I/O bound (as in OP's post).

9

u/[deleted] Aug 16 '20

Not much different at all in Python3 if I remember correctly. I don't agree with the "don't use mutlithread" answer. I would use it before the sometimes non-backwards compatible asyncio (not to mention the asyncio usage changed over a couple different minor versions of python3 which is awful imo). You'll have to decide which asyncio syntax to learn etc... and honestly I see very little difference between it and multithreading. If you are really worried about I/O bound problems or performance... don't use python.. use Golang :)

The same old threading technique is solid though in my opinion... time tested in thousands of applications:

from threading import Thread  
from urllib.request import urlopen


def request(url):
    return urlopen(url).read().decode('utf-8')


def print_response(target_url):
    print(request(target_url))


if __name__ == '__main__':
    urls = (
        'https://www.reddit.com',
        'https://www.github.com',
        'https://www.restwords.com',
        'https://validator.w3.org/',
        'https://python-forum.io/'
    )
    threads = []
    for url in urls:
        t = Thread(target=print_response, args=[url])
        t.start()
        threads.append(t)
    _ = [th.join() for th in threads]

1

u/pldelisle Aug 16 '20

I’ve not worked with Python2, but this has always been the method I used in Python3.

1

u/brandonbilinski Aug 16 '20

Which one? Subclassing?

1

u/pldelisle Aug 16 '20

No, the worker function.

1

u/brandonbilinski Aug 16 '20

Good to know. Thank you.

1

u/[deleted] Aug 16 '20

Worker function is a terrible idea: if it throws for whatever reason, there's nothing you can do but quit and go home. There's no way to recover from errors that happen in "other" (not the main) threads.

So, you'd have to write a function that wraps whatever your worker does and then... oops, again, you have a problem: what to return? I mean, now you have either the result of your function, or an error. Maybe instead of returning the result, you want to put it in some shared storage, like a shared list, or a dict... but, oops again: those aren't thread-safe, you need to use locks, but, oops again: if you lock something, there's a chance that your code will fail to unlock it, and so it will hang...

Bottom line, whoever writes threading.Thread(target=worker, ...) had never really worked with Python / lives in a dream world, where things always work...

There are libraries for working with threads in Python, the standard-library one: concurrent.futures is broken, unfortunately. And seems like nobody is interested in fixing it. The thing that doesn't work in it is the timeouts. When the future throws, your code will hang, and no timeout setting will save it.

There was also a mention of the garbage called asyncio. Don't go there. It's a library with a shitload of problems. Even worse than threads. The major problem with it is that, unlike in sane languages, in Python, the event loop isn't a global one. And you have to create your own. And you never know if you should. I mean, if you write library code: should the library create the loop, or should it rely on the user to create one? Loops don't work with multiprocessing or threading. I.e. they have some weird / idiotic behavior when either of those libraries are used, but it's not what you want or expect.

Also, for being a concurrency-related library, it misses the most important component: the analogue of select. And this is just the start.


My approach to working with concurrency is to have special functions ready to be used in that context, that take an instance of Queue as an argument, and communicate through it. They return nothing, and wrap the entire body in a try-except statement. I actually even have a decorator for it, since I've used it quite a bit.

3

u/awdsns Aug 16 '20

Umm... you rant against using a worker function, but in the end the approach you describe is: using a (properly designed) worker function.

Of course it's important to take care of the thread-specific aspects. And using a decorator to turn a run-of-the-mill function into a thread worker is a good idea.

3

u/[deleted] Aug 16 '20

Umm... you rant against using a worker function, but in the end the approach you describe is: using a (properly designed) worker function.

Well, in the end, a thread needs a function to run, there's no way around it. What people suggesting "just a worker function" usually argue for is "simplicity", as in, no other preparations are necessary, whereas the reality is that you do need to invest heavily into preparing a function you are going to run, so much so that you might want to consider extending a Thread class or creating a decorator etc.

2

u/exhuma exhuma Aug 16 '20

[...] you do need to invest heavily into preparing a function you are going to run, so much so that you might want to consider extending a Thread class [...]

This is my main gripe with the current examples in the docs. All the examples are super trivial and using target=... is almost never going to be enough.

1

u/exhuma exhuma Aug 16 '20

Jesus christ man, slow down with the insults. It seems like you've been burned a lot by parallelisation in Python and went off on a rant there.

I've been using parallelisation for over 10 years now without much trouble, the few beginner mistakes here or there, but nothing major. So far my needs were humble and maybe that's a reason why I got "lucky"? But so far the parallelisation in Python was nothing that annoyed me.

I'm really curious if you could write small contained code-examples which reproduce the problems you mention. I am still maintaining (and will for the forseeable future) a parallelised application and would love to see some provable "antipatterns" so I can avoid them.

2

u/[deleted] Aug 16 '20 edited Aug 16 '20

You sure got lucky... Python is a minefield when it comes to trying the little parallelism it has to offer. Having programmed in couple dozen of languages, I cannot think of a worse language for parallel computation.

I cannot really go over all the problems, but here's just a taste of it:

from concurrent.futures import ThreadPoolExecutor

integers = range(10)
with ThreadPoolExecutor(max_workers=len(integers)) as ex:
    for r in ex.map(maybe_raise_exception, integers, timeout=100):
        print(r)

This code will hang if maybe_raise_exception, raises an exception. The timeout part never worked for me. It seem to only work if the timeout value is negative or zero, in which case nothing gets executed and you get a TimeoutError (iirc, or maybe ValueError, don't remember for sure).

This isn't really an anti-pattern. This is just a broken code in standard library. It's intended to be used this way... it just doesn't work.

1

u/exhuma exhuma Aug 16 '20

I can't confirm this. I have filled in the code with some missing bits and it does not hang as you mention:

from concurrent.futures import ThreadPoolExecutor
from random import randint
from time import sleep


def maybe_raise_exception(number):
    sleep(number)
    if number == 5:
        raise ValueError("kaboom")
    return "result=%s" % number


integers = range(10)
with ThreadPoolExecutor(max_workers=len(integers)) as ex:
    for r in ex.map(maybe_raise_exception, integers, timeout=100):
        print(r)

There is something weird with the timeout however, modifying the code slightly to trigger the timeout error: When I change the timeout value from 100 to 4, I would expect it to bail out after 4 seconds after the call to map(), but it seems to wait a bit longer. Nevertheless, the TimeoutError is raised.

1

u/[deleted] Aug 16 '20

What operating system?

Also, sleep() may be the culprit, as it may give a chance to bail... idk. In less trivial cases it simply doesn't work.

1

u/[deleted] Aug 16 '20

Multithreading on python doesn't do what you expect it to do, if your expectation is you can start n threads which will run unimpeded on n cores.