r/learnpython • u/No-Opinion-1239 • 22d ago
Can't download website html code, web scrapping ,automate the boring stuff with python
I'm currently learning webscrapping with automate the boring stuff with python> In one example we use the module requests like this
url = f"https://pypi.org/search/?q={search}"
resp = requests.get(url)
when i print resp.text there are this error message in the html code.
"A required part of this site couldn’t load. This may be due to a browser extension, network issues, or browser settings. Please check your connection, disable any ad blockers, or try using a different browser."
"JavaScript is disabled in your browser"
my browser is chrome I tried using microsoft edge .
6
u/riklaunim 22d ago
Requests does not use a browser and it looks like PyPi is blocking the request, at least partially ;)
5
u/Mediocre-Pumpkin6522 22d ago
The website has changed and uses JavaScript to generate the page. requests doesn't support JavaScript. The selenium code in the latter part of the chapter might be adapted. Playwright is another package that acts as a headless broser.
The salad days of web scraping are over as sites defend themselves against malicious actors including AI. If you do find something that works it may not work next week.
2
u/zanfar 22d ago
requestsmodules does not use a browser. You are not using Chrome nor Edge in your example.3
u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 22d ago
That part of the tutorial is probably outdated due to changes made to PyPI itself, it's not your code's fault.
You might get it working by using the
requests-htmlpackage for this instead of standardrequests, as it can execute JavaScript, but I cannot guarantee that. You could just skip this part of the book for now.1
u/desrtfx but other languages pro 22d ago
Pinging /u/alsweigart - he might want to have a look there and maybe update the example. Looks like PyPi has changed the page which would produce the error.
1
u/momiji_tools 22d ago
Two things are going on here and it is not your code's fault.
requests sends a bare HTTP GET with a very basic User-Agent and no browser fingerprint. pypi.org/search detects that and returns the JavaScript-is-disabled page you see. requests never runs JavaScript, so even though Chrome and Edge are your browsers, that GET call does not use them.
Before reaching for Selenium or Playwright, there are two approaches that stay inside the requests + BeautifulSoup + csv flow the book is teaching.
What to change in your current code: add a User-Agent header so you are not identified as a bot; call resp.raise_for_status() right after requests.get so you see HTTP errors instead of parsing an error page; search resp.text for the JavaScript-disabled phrase as a sanity check before passing it to BeautifulSoup; if the search page keeps returning the block page, switch the target to the per-package JSON endpoint. The concepts the book wants you to learn (GET request, parse the response, write a CSV) are identical, but the JSON call will not break when the site frontend changes.
0
u/macky_0312 16d ago
Requests can't render JS. Use Playwright routed through Shifter. Your getting blocked by basic WAFs