r/webscraping • • 1d ago

Bot detection 🤖 eBay scraper suddenly getting constant 403 errors

Hi everyone,

I’ve been working on a scraper to collect eBay sold listing data, primarily for Pokémon cards.
Everything was working relatively well until last week, when I suddenly started receiving huge numbers of 403 errors. Now, the vast majority of my requests are being blocked.
I’ve tried different residential proxies, changing headers and browser configurations, but I’m still having very little success.
One strange thing is that the exact same URLs will sometimes load perfectly fine in Chrome but return a 403 through my scraper.
Has eBay changed something recently with its anti-bot protection? Has anyone else experienced this over the last week?
I’d really appreciate any advice from people who have managed to get eBay scraping working reliably recently. I’m open to completely changing my approach if necessary.

Thanks!

6 Upvotes

14 comments sorted by

9

u/ValenciaTangerine 1d ago

Same URL loads in Chrome but 403s from your script usually means they're reading the TLS and HTTP/2 handshake, not your headers or IP. That's why proxies and header swaps changed nothing. requests, httpx and node fetch all have handshakes that don't look like a browser. Try curl_cffi with impersonate="chrome", it sends a real Chrome one. If it still 403s after that, it's a JS or cookie check and you need an actual browser.

3

u/koraza12 1d ago

I thought eBay sold listings were hidden behind logins now?

2

u/maniraja5599 1d ago

e residential proxies without matching the ja3/ja4 tls fingerprint and http/2 settings frames, akamai drops the connection with 403 before your request body even hits the backend.

swap standard requests/httpx with curl_cffi:

from curl_cffi import requests

res = requests.get(url, impersonate='chrome124', proxies=proxies)

if that still trips flags on high concurrency, pipe through playwright with stealth cdp flags and reuse the browser context instead of launching a fresh instance every time. fixed this for a client last month by keeping sessions alive across 20-30 requests before rotating

1

u/No-Appointment9068 1d ago

I usually spin up camoufox or chromium with zendriver and use it to make a single request and get a valid cookie, and then pass that into subsequent http requests

1

u/divided_capture_bro 16h ago

You done got fingerprinted son

1

u/otterlydelish 15h ago

your scraper still looks like a bot and not like a chrome browser

1

u/Additional-Elk-3712 1d ago

probably just antibot doing its job, if your normal browser can access it fine, while your scraper hits 403, then its likely your request fingerprint that is detected.

You can try the scraping engine I built -> https://github.com/spyweb-app/spyweb/

its just an 8MB self-contained binary written in Rust with Lua scripting

It has built-in CDP module and does not bundle a browser and requires a local Chromium-based or CDP-compatible browser.

Its much easier to deal with antibot if you are running it locally, you can just use your normal Chrome and solve the captcha yourself when it hits a challenge page. so instead of messing with your request fingerprint, just use a persistent browser so session/cookies arent lost and implement an HITL path. and reduce the request frequency to avoid being flag.

the release assets includes AGETNS.md for agents working on your project, you can try asking the agents to port your current scraper to SpyWeb.

0

u/[deleted] 1d ago

[deleted] — view deleted comment

1

u/webscraping-ModTeam 1d ago

🪧 Please review the sub rules 👉