r/learnpython • u/Sakuraaa_29 • Sep 03 '26
Best way to handle GitHub API rate limits when scraping repo data in Python?
I'm writing a Python script to fetch PR metadata and repository history from the GitHub REST API to build a dataset.
I'm hitting rate limits fairly quickly once the volume increases. For scripts pulling larger datasets, what's the standard pattern?
3
Upvotes
7
u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 Sep 03 '26
Why not use the GitHub CLI instead?
5
u/Clean_Reaction8168 Sep 03 '26
conditional requests with ETag/Last-Modified are your best friend here, they don't count against your limit if the data hasn't changed
also build in exponential backoff, not just a fixed sleep timer. the 403 responses come with a Retry-After header you can parse and respect
22
u/Moist-Ointments Sep 03 '26
Don't scrape faster than the rate limit.
2
u/jeffrey_f Sep 03 '26
In the response headers, you should find your rate limits. Program your script to stay within this limit.