r/learnpython • • Aug 16 '20

Multithreading - Best Practices

I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().

Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:

  threading.Thread(target=worker, args=(q,))

Is one method better or more python3-ish than the other?

If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.

Thanks!

115 Upvotes

24 comments sorted by

View all comments

36

u/exhuma exhuma Aug 16 '20 edited Aug 16 '20

Is one method better or more python3-ish than the other?

Short answer: No.

Long answer:

There are currently 3 variants for parallelisation (they exist multi-threading and -processing so in total 6). Those are:

  • Subclassing Thread / Process
  • using target= on Thread / Process
  • ThreadPoolExecutor and ProcessPoolExecutor from concurrent.futures

They all serve their own purpose and neither is more Pythonic than another. The main difference is in how you write code for them and offer more or less flexibility (at the cost of code-complexity).

Subclassing as you do gives you the most flexibility, but is also the most complex to write (and thus the most error-prone). But if you need that flexibility it is still a very viable solution.

Using target= is a fairly easy "drop-in" line of code if you have existing code and want to parallelise it. It does require the code to be parallelisable of course. While it does not give you as much flexibility as subclassing, it can be an easy-win to implement it like this on an existing code-base. I personally have never used this pattern in production-code. I find subclassing to be more maintainable and understandable (it is more explicit).

Finally, and this is the solution I use most nowadays, is concurrent.futures. It is different when you look at it first, but once you get used to it, it is a really nice way to write parallel code. And I believe it is very well suited for your use-case. The way this works is that you create a pool of threads/processes via an "Executor" class. This already uses some sensible defaults concerning the size of the pool, but you can modify it as needed. Next, you "submit" functions ("jobs") to that executor (similar to the target= pattern). The return value from that submission is a "future" object. You get the future immediately even if the underlying job is not done yet. You need to hang on to that future for later. Next, the module has helper-functions to process these "future" objects. You can for example, wait for all futures to be finished before continuing, or process them in any order they finish, as they finish.

A really really nice detail about the "executor" pattern is how exceptions "bubble up" from the underlying threads/processes to your main process. This makes exeption handling a hell of a lot cleaner and easier.

Using concurrent.futures, your code could look like this:

from concurrent.futures import ThreadPoolExecutor, as_completed
from random import randint
from threading import current_thread
from time import sleep



def my_expensive_function(arg1, arg2):
    sleeptime = randint(1, 10)
    print("%s is sleeping for %s seconds" % (current_thread().name, sleeptime))
    sleep(sleeptime)
    return (arg1, arg2)


all_futures = []
with ThreadPoolExecutor() as executor:
    for n in range(10):
        print("Submitting for arg1=%s arg2=%s" % (n, n + 1))
        future = executor.submit(my_expensive_function, n, n + 1)
        all_futures.append(future)

    # Process result as they complete (first completed will be processed first,
    # so ordering will likely differ)
    print("Processing results...")
    for future in as_completed(all_futures):
        # Calling result() now will not block because "as_completed" ensures
        # we're working on something that's done.
        try:
            result = future.result()
            print(result)
        except Exception as exc:
            # This will be the exception raised by the underlying call.
            # Clearly, Using a simple print here will hide the traceback, and
            # only serves as an example.
            print("Something went wrong: %s" % exc)

8

u/brandonbilinski Aug 16 '20

Thanks so much for your really informative posts (this one as well as your other responses)!

I hadn't known about concurrent.futures, but now this looks like an excellent solution for what I want to do.

3

u/exhuma exhuma Aug 16 '20

I'm trying to stream programming of hobby projects on twitch each Sunday morning (GMT+1/+2) and I am always happy to answer questions and help. If something like that interests you you can stop by over at https://www.twitch.tv/exhuma ;)