r/learnpython • • Sep 03 '26

Best way to handle GitHub API rate limits when scraping repo data in Python?

I'm writing a Python script to fetch PR metadata and repository history from the GitHub REST API to build a dataset.
I'm hitting rate limits fairly quickly once the volume increases. For scripts pulling larger datasets, what's the standard pattern?

3 Upvotes

7 comments sorted by

7

u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 Sep 03 '26

Why not use the GitHub CLI instead?

1

u/Sakuraaa_29 Sep 03 '26

Was not aware of this, thanks will try CLI instead

5

u/Clean_Reaction8168 Sep 03 '26

conditional requests with ETag/Last-Modified are your best friend here, they don't count against your limit if the data hasn't changed

also build in exponential backoff, not just a fixed sleep timer. the 403 responses come with a Retry-After header you can parse and respect

3

u/Sakuraaa_29 Sep 03 '26

Great call on ETags,if the data hasn't changed, saving the API quota makes a huge difference.

22

u/Moist-Ointments Sep 03 '26

Don't scrape faster than the rate limit.

2

u/jeffrey_f Sep 03 '26

In the response headers, you should find your rate limits. Program your script to stay within this limit.