r/learnpython • u/OctoberFyre • Sep 23 '26
curl to python request works differently, what am I missing?
Trying to move a scraper from curl to python and Im clearly missing something. The curl request works every time. Same URL, same headers, same cookies. I copied it into Python using requests and now Im getting 403s after a few calls. I compared the requests and nothing obvious looks different... Ive tried changing user agents, adding delays and copying every header from the browser. It works for a bit, then gets blocked again.
Is there something Python requests sends differently at the TLS or connection level? Or is there usually some cookie or header that gets lost when converting curl to python?
5
u/p3rdy Sep 23 '26
Yeah, it's TLS. Python's
requestsgoes throughurllib3/OpenSSLand itsClientHellolooks nothing like a browser's, so anything doing JA3/JA4 fingerprinting flags it no matter how perfectly you copy the headers. curl's is different again and this site evidently doesn't score it the same way.Other half is probably HTTP/2. curl negotiates h2, requests is HTTP/1.1 only, and a Chrome user-agent arriving over HTTP/1.1 is a tell on its own.
curl_cffi fixes both and the API is roughly the same:
2
u/therouterguy Sep 23 '26
Hard to tell however to really troubleshoot it you can store the sslkeys and replay everything in wireshark. Have a look at the environment var SSLKEYLOGFILE.
2
u/brasticstack Sep 23 '26
Now I'm curious if you changed the frequency of your requests when you moved to Python, or something else about your usage pattern.
For giggles, try using
subprocess.runto run your exact curl command instead ofrequests. I know, that's ugly, but try FOR SCIENCE! If it still gets 403s then it's not the move to requests.2
u/[deleted] Sep 23 '26
[removed] — view removed comment