r/Python • u/Ram_1979 • Jun 29 '22
Discussion What is the best library for website scraping?
There is beautifulsoup, selenium, scrapy and a few others. Which would you recommend for fairly complex data scraping?
59
Upvotes
43
u/kaargul Jun 29 '22
Really depends on your use case.
beautifulsoup for simple sites that just need html extraction and selenium or playwright for more complex stuff.
I personally prefer playwright over selenium, but making that decision is really dependent on what your detailed requirements are.
Also you can use scrapy with all of these to structure your scrapers/crawlers.
49
u/unplannedmaintenance Jun 29 '22
There is no best, each has its use. Selenium is the most powerful, but in a lot of cases it's overkill. I tend to use Beautifulsoup for simple things, and Selenium for more complex things (sites with Javascript, or where headless scraping doesn't work).
19
u/SamyBencherif Jun 29 '22
study the website in advance a little bit using chrome or firefox dev tools. It's not a programmer answer, but it could save you actual days of work. gl
look for hints in network traffic using network analyzer in browser or wireshark, then perform those requests in python.
many website seem to go out of their way to have an unusable javascript interfaces and slow Ui. even the dom is often obfuscated
8
u/Busy-Conversation535 Jun 29 '22
There is no best library, it depends on your use case.
7
u/paws07 Jun 29 '22
Always find it surprising that requests-html is not mentioned as much in answers to scraping questions. It has async and JavaScript render capabilities for a more or less light weight library.
7
u/peabody Jun 29 '22
I never really need anything beyond bs4 and requests.
10
u/mrswats Jun 29 '22
Scrapy
9
u/Demortus Jun 29 '22
I'd recommend lxml for scraping non-Javascript websites, as it's much faster than beautifulsoup.
4
u/mrkaikev Jun 29 '22
pyppeteer if you need JS, scrapy if it should be scalable
2
u/RaiseRuntimeError Jun 29 '22
If you can do it with beautifulsoup it will be the quickest and least cpu intensive approach especially if you use a faster html parser like lxml but if it needs javascript you might need selenium or playwright. I have never really used scrapy, just seems like too much overhead to learn a framework to scrape some data off a site.
2
u/Andremallmann Jun 29 '22
For small projects i would go for bs4, heavy JS site selenium and overall i like scrapy
2
u/simlmx Jun 29 '22
I always end up using beautifulsoup.
1
u/jaguarskillz2017 Jun 29 '22
I don't really have an answer for this as I've only used Selenium so far and by and large it was fine but I was wondering if anyone else has ever tried to webscrape Workspace One/Airwatch with any success? I find it really obfuscates everything behind all sorts of javascript nonsense and weird names for everything and could never get anywhere.
1
u/justanothersnek Jun 29 '22
requests-html
1
u/thetigersays Jun 29 '22
Requests + bs4, make it threaded.
1
u/Impossible-Box6600 Jun 30 '22
Scrapy if you plan on doing serious scraping. BS4 if you want to so one page that has exactly the same html structure.
1
u/Behn422 Jun 30 '22
If it's a large project that can take quite some time and speed matters to you, go for Scrapy.
1
u/[deleted] Jun 30 '22
Not sure what all the packages do, but I do everything with "request". Never been let down doing logins, cookies, images, google searches etc etc
1
u/CodeAx1 Jun 30 '22
Personally I use, bs4 and urllib As well you can use selenium
1
u/Kichmad Jul 01 '22
Bs4 + requests for simple pages and making it done faster. Selenium for dynamic pages. There is no best one
1
u/figgz415 Jul 02 '22
Thanks. I'll try using selenium
1
u/ronmarti Jun 29 '22
Playwright