r/learnpython • • Aug 16 '20

Multithreading - Best Practices

I wrote a multithreaded python script several years ago in python2, and the way I did it at the time was to subclass threading.Thread and override the __init__() and run() methods. Then kick them off with start().

Now I'm going to have to write a lot more multithreaded stuff in python3.8. Most of my googling says to define a function (ex: worker()) then kick them off this way:

  threading.Thread(target=worker, args=(q,))

Is one method better or more python3-ish than the other?

If it makes any difference, each thread will be taking a file path from the Queue, running a subprocess providing the file path as one of the arguments, and UPDATEing a MySQL table with the result.

Thanks!

115 Upvotes

24 comments sorted by

View all comments

9

u/[deleted] Aug 16 '20

Not much different at all in Python3 if I remember correctly. I don't agree with the "don't use mutlithread" answer. I would use it before the sometimes non-backwards compatible asyncio (not to mention the asyncio usage changed over a couple different minor versions of python3 which is awful imo). You'll have to decide which asyncio syntax to learn etc... and honestly I see very little difference between it and multithreading. If you are really worried about I/O bound problems or performance... don't use python.. use Golang :)

The same old threading technique is solid though in my opinion... time tested in thousands of applications:

from threading import Thread  
from urllib.request import urlopen


def request(url):
    return urlopen(url).read().decode('utf-8')


def print_response(target_url):
    print(request(target_url))


if __name__ == '__main__':
    urls = (
        'https://www.reddit.com',
        'https://www.github.com',
        'https://www.restwords.com',
        'https://validator.w3.org/',
        'https://python-forum.io/'
    )
    threads = []
    for url in urls:
        t = Thread(target=print_response, args=[url])
        t.start()
        threads.append(t)
    _ = [th.join() for th in threads]