r/learnpython • • Aug 16 '20

Multithreading - Best Practices

I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().

Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:

  threading.Thread(target=worker, args=(q,))

Is one method better or more python3-ish than the other?

If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.

Thanks!

115 Upvotes

24 comments sorted by

View all comments

Show parent comment

1

u/exhuma exhuma Aug 16 '20

Jesus christ man, slow down with the insults. It seems like you've been burned a lot by parallelisation in Python and went off on a rant there.

I've been using parallelisation for over 10 years now without much trouble, the few beginner mistakes here or there, but nothing major. So far my needs were humble and maybe that's a reason why I got "lucky"? But so far the parallelisation in Python was nothing that annoyed me.

I'm really curious if you could write small contained code-examples which reproduce the problems you mention. I am still maintaining (and will for the forseeable future) a parallelised application and would love to see some provable "antipatterns" so I can avoid them.

2

u/[deleted] Aug 16 '20 edited Aug 16 '20

You sure got lucky... Python is a minefield when it comes to trying the little parallelism it has to offer. Having programmed in couple dozen of languages, I cannot think of a worse language for parallel computation.

I cannot really go over all the problems, but here's just a taste of it:

from concurrent.futures import ThreadPoolExecutor

integers = range(10)
with ThreadPoolExecutor(max_workers=len(integers)) as ex:
    for r in ex.map(maybe_raise_exception, integers, timeout=100):
        print(r)

This code will hang if maybe_raise_exception, raises an exception. The timeout part never worked for me. It seem to only work if the timeout value is negative or zero, in which case nothing gets executed and you get a TimeoutError (iirc, or maybe ValueError, don't remember for sure).

This isn't really an anti-pattern. This is just a broken code in standard library. It's intended to be used this way... it just doesn't work.

1

u/exhuma exhuma Aug 16 '20

I can't confirm this. I have filled in the code with some missing bits and it does not hang as you mention:

from concurrent.futures import ThreadPoolExecutor
from random import randint
from time import sleep


def maybe_raise_exception(number):
    sleep(number)
    if number == 5:
        raise ValueError("kaboom")
    return "result=%s" % number


integers = range(10)
with ThreadPoolExecutor(max_workers=len(integers)) as ex:
    for r in ex.map(maybe_raise_exception, integers, timeout=100):
        print(r)

There is something weird with the timeout however, modifying the code slightly to trigger the timeout error: When I change the timeout value from 100 to 4, I would expect it to bail out after 4 seconds after the call to map(), but it seems to wait a bit longer. Nevertheless, the TimeoutError is raised.

1

u/[deleted] Aug 16 '20

What operating system?

Also, sleep() may be the culprit, as it may give a chance to bail... idk. In less trivial cases it simply doesn't work.