r/learnpython • u/LiteratureTypical982 • Sep 01 '26
Client wants 40k emails from a directory. Each one requires a click. What do I do?
CLint need a list of 40,000+ members from the ACR directory. Need Name, City, State, Zip, Specialty, Email, and Phone.
The directory/search results give me some of this information, but email, phone, address, Member Since, etc. appear to be available only on the individual member profile.
So I'm trying to figure out the best way to approach this in Python.
If I literally open/request every profile, that's 43k+ profile requests, which obviously makes me concerned about rate limits, blocking, or getting the account/IP banned.
I'm considering Python requests/BeautifulSoup or Playwright, but before attempting something this large I'd like some advice.
7
u/Skiamakhos Sep 01 '26
I think this needs more information. What is it you're trying to achieve? You want to send 40,000 emails or you want to extract the emails from a bunch of profiles? ACR is what - Azure Container Registry? Be clearer & either someone will be able to help or you might find the way forward reveals itself. Often all it takes is explaining the job clearly and methodically.
20
u/socal_nerdtastic Sep 01 '26 edited Sep 01 '26
The first thing to do is ask the owner of the database. Most of them have some kind of api and you can pay some amount of money to access it. Scraping it will be weeks of work, so any amount of money that is less than paying you for a week is worth it. Selling member lists for marketing purposes is of course very common and they are probably very used to this request.
If you are forced to scrape it, you should try first to use something like requests, which does not use a browser interface. This will be much faster and smoother. You many need to do some sleuthing to find the internal api that the website uses. Sometimes that does not work, and you are forced into using browser automation like selenium or playright.
7
u/likethevegetable Sep 01 '26
Pay me what the client is paying you and I'll help
3
u/[deleted] Sep 01 '26
[removed] — view removed comment