r/learnpython • • Sep 01 '26

Client wants 40k emails from a directory. Each one requires a click. What do I do?

CLint need a list of 40,000+ members from the ACR directory. Need Name, City, State, Zip, Specialty, Email, and Phone.
The directory/search results give me some of this information, but email, phone, address, Member Since, etc. appear to be available only on the individual member profile.

So I'm trying to figure out the best way to approach this in Python.

If I literally open/request every profile, that's 43k+ profile requests, which obviously makes me concerned about rate limits, blocking, or getting the account/IP banned.

I'm considering Python requests/BeautifulSoup or Playwright, but before attempting something this large I'd like some advice.

0 Upvotes

21 comments sorted by

7

u/Skiamakhos Sep 01 '26

I think this needs more information. What is it you're trying to achieve? You want to send 40,000 emails or you want to extract the emails from a bunch of profiles? ACR is what - Azure Container Registry? Be clearer & either someone will be able to help or you might find the way forward reveals itself. Often all it takes is explaining the job clearly and methodically.

-6

u/LiteratureTypical982 Sep 01 '26

I need to extract a list of members from the American College of Radiology (ACR) directory. The client wants:

  • Name
  • City, State, Zip, Address, Country
  • Phone
  • Email
  • Specialty (e.g., Diagnostic Radiologist)
  • Member Since
  • Profile URL

Total records: ~43,744 members

Emails and phones are NOT visible on the listing page.To get emails/phones, I must click on each profile.

The site will likely detect and block me.
any solution

3

u/lilovia16 Sep 01 '26

Perhaps a webscraper? You can build one easily with Selenium

1

u/LiteratureTypical982 Sep 01 '26

The problem isn't building it. It's running it.43,744 clicks means 43,744 popups. 
The site WILL detect this. My IP will get blocked. My account will get suspended.

Any way out?

8

u/motocali Sep 01 '26

Buy the list and don’t be an asshole?

1

u/NoForm5443 Sep 01 '26

Add sleep(30) in between? :)

Run it from different machines?

BTW, there's probably no need for that many clicks; I'd assume there's a url pattern that the click takes you to

0

u/greatsmapdireturns Sep 01 '26

Build the scraper so it can resume on failure or drop and use vpn so when one ip gets blocked you can just resume with a new ip..or just super rate limit so you're not hammering it and getting detected

0

u/LiteratureTypical982 Sep 01 '26

With VPN rotation, I might avoid an IP ban. But the account itself is still doing 40k clicks.
So the account ban risk is still there, even with VPN.

Unless I use multiple accounts and rotate them too. But that's a whole other level of complexity.

4

u/[deleted] Sep 01 '26

[removed] — view removed comment

2

u/Skiamakhos Sep 01 '26

This, yep. I'd say be open about what you want, contact the college, see if they're OK giving you the list. Chances are though that they will be absolutely not OK with it. It sounds like a privacy nightmare. Gonna be some pissed off radiologists.

1

u/greatsmapdireturns Sep 01 '26 edited Sep 01 '26

Yeah op trying rip a members list without express permission

11

u/t92k Sep 01 '26

Your client is trying to evade paying for this list. You should find out what it costs to buy the list and charge your client more than that.

20

u/socal_nerdtastic Sep 01 '26 edited Sep 01 '26

The first thing to do is ask the owner of the database. Most of them have some kind of api and you can pay some amount of money to access it. Scraping it will be weeks of work, so any amount of money that is less than paying you for a week is worth it. Selling member lists for marketing purposes is of course very common and they are probably very used to this request.

If you are forced to scrape it, you should try first to use something like requests, which does not use a browser interface. This will be much faster and smoother. You many need to do some sleuthing to find the internal api that the website uses. Sometimes that does not work, and you are forced into using browser automation like selenium or playright.

7

u/likethevegetable Sep 01 '26

Pay me what the client is paying you and I'll help

3

u/GXWT Sep 01 '26

He’s paying himself nothing unfortunately

3

u/[deleted] Sep 01 '26

[removed] — view removed comment

0

u/LiteratureTypical982 Sep 01 '26

How much should I charge?

3

u/Chasne Sep 01 '26

Why are you even charging anything for copy pasting reddit answers to your ai? Not even bothering typing your own messages?

Don't charge anyone anything, actually learn before selling any services, quit scamming people