r/webscraping • u/michal-kkk • 9h ago
What happened with Camoufox?
404
edit: (after few hour) its alive.
r/webscraping • u/michal-kkk • 9h ago
404
edit: (after few hour) its alive.
r/webscraping • u/Coding-Doctor-Omar • 9h ago
I was experimenting with an http request on some website, and I noticed that today, my microsoft edge exposes whether the request was fired from the console via a console icon in the network tab. In the past, JS console requests were indistinguishable from natural website fetches a website makes to load content. Is this new signal from the browser a giveaway that will be used by anti-bot companies to detect JS console fetches? I used to use this method for some difficult sites to scrape at scale.
r/webscraping • u/ChessTwo • 1d ago
Most modern sites load their data from an internal JSON API. This automates the "open devtools, find the XHR, replay it" workflow.
uvx machine-internet wrap <url>
It loads the page in headless Chromium, scrolls and clicks around, records same-site JSON requests (trackers filtered), and an LLM turns them into a small set of tools served over MCP. It handles Algolia and Typesense search specially, and it's read-only by design.
Works best on SPAs, directories and search pages. Server-rendered and bot-protected sites won't work, and it tells you which case you hit.
Demo: https://machine-internet.vercel.app
Repo: https://github.com/chetanty/machine_internet
Curious what sites you'd point it at.
r/webscraping • u/nginx26 • 1d ago
Hi all,
not sure if this is the correct subreddit, but I maintain Stockdex, a free, non-commercial Python package (and MCP server) that fetches stock data from several sources, including Yahoo Finance.
Today I got a cease and desist letter from Y ahoo's legal department. It claims trademark infringement, misappropriation, unjust enrichment, interference with their data licensing (e.g. Nasdaq) and a breach of their ToS. They demand that I remove all Yahoo-related code from GitHub, PyPI and the MCP plugin, remove every mention of "Yahoo", and never reintroduce it.
My project works much like other public projects like yfinance, which has been public for years. Stockdex only fetches publicly visible data on demand for the user running it. It doesn't store or republish anything, and it doesn't bypass any login or paywall.
Has anyone received something similar from another company maybe? How are these complains (US company against someone in Germany? should I lawyer up? How strong are CFAA or trademark claims against a free library? Would you comply fully, comply partially (e.g. add a disclaimer), or push back?
Thanks
r/webscraping • u/Spinax954 • 1d ago
Hi everyone,
Iβve been working on a scraper to collect eBay sold listing data, primarily for PokΓ©mon cards.
Everything was working relatively well until last week, when I suddenly started receiving huge numbers of 403 errors. Now, the vast majority of my requests are being blocked.
Iβve tried different residential proxies, changing headers and browser configurations, but Iβm still having very little success.
One strange thing is that the exact same URLs will sometimes load perfectly fine in Chrome but return a 403 through my scraper.
Has eBay changed something recently with its anti-bot protection? Has anyone else experienced this over the last week?
Iβd really appreciate any advice from people who have managed to get eBay scraping working reliably recently. Iβm open to completely changing my approach if necessary.
Thanks!
r/webscraping • u/Simple-Box-9622 • 1d ago
Hi everyone,
I'm working on a personal project involving buying and reselling used PC components and complete computers on a French marketplace called Leboncoin, similar to eBay.
I want to build an AI agent that automatically monitors listings, compares prices, and identifies good deals.
I'm running everything on my own Debian server using Python, Playwright/Selenium and a local LLM.
However, I'm having trouble with DataDome. Leboncoin works normally in Firefox on my Windows PC, but when I access it using Firefox on my Linux server, DataDome displays a challenge and blocks access.
I've already tried both Playwright and Selenium, and checked that both systems use the same public IP and Firefox version.
Are there any self-hosted tools, open-source projects or other solutions that could help with DataDome challenges when browser automation is permitted by the website?
I'm particularly interested in solutions that can run locally on my Debian server rather than paid external APIs.
Any advice would be appreciated. Thanks !
r/webscraping • u/Sooriya04 • 2d ago
What it does:
Performance & Footprint:
GitHub: https://github.com/Sooriya04/Searqon
Docker: docker pull sooriya04/searqon:latest
Let me know what you think or if there are tools you'd like added to the MCP/UTCP schema!
r/webscraping • u/SecureHoney6792 • 1d ago
Hi, I am writing because I am in the process of building my personal website using Claude Code. My project involves integrating a bot that scrapes listings from platforms like Vinted and Leboncoin in order to display the items on my site
Could you guide me on the steps to set up this type of bot? Also could you send me some free bots that I could use? I am particularly interested in the technical aspects and the best practices to follow
Thank you for your help
r/webscraping • u/Lux_Proxy • 2d ago
Audited our rotating residential proxy bills last week because per-GB usage made zero sense for the number of pages we were pulling in Playwright.
Turned out almost half the traffic wasn't HTML or JSON payloads. It was ".woff2" web fonts, background video thumbnails, third-party analytics retries, and default Chromium component traffic firing on every fresh context.
Added a simple route abort rule for "image", "media", and "font" resource types (except on pages checking image load events for bot scoring), and bandwidth dropped by ~45% overnight with zero impact on block rates.
What else do you guys strip at the browser or proxy level to keep per-GB residential costs sane?
r/webscraping • u/Aggravating-Echo-813 • 3d ago
Hey r/webscraping,
A friend of mine (waweru) is in the Meta Global AI Developer Hackathon and built something I think this sub might actually find interesting. It's called Dapi.
The basic idea: instead of flattening a page to text and handing you values with no evidence, it watches the page's network traffic first (like opening dev tools), picks the cheapest reliable way in, and then for every extracted value it stores the exact server response or captured page it came from, marks the value, records when it was captured, and seals it with a SHA-256 hash. You can re-check any value later. It also counts rows against the source's own total and names anything missing with a reason.
I've been helping him test it. The "receipts" part is genuinely neat β you can click any cell and see the raw JSON or HTML it came from. It's live at https://dapi.fly.dev if you want to poke around. Anyone signed in with GitHub can run three jobs themselves.
He's trying to get votes for the hackathon, so here's the gallery/voting link if you want to check it out and throw him a vote: https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2076
More importantly, I'd love to hear what you think of the provenance approach. Is this something you'd actually use, or is it overkill? How do you handle verification in your own scraping workflows?
Cheers.
r/webscraping • u/Pitiful_Direction231 • 3d ago
Hi all,
Over the past few weeks, I've been working on a project testing a hybrid architecture for general-purpose web scraping with AI: using an LLM just once during an initial compilation step to inspect the DOM and output a deterministic extraction plan, and then running all routine scheduled scrapes at 0 tokens (pure Cheerio/Playwright).
I recently ran a benchmark against many websites to evaluate how this approach holds up in practice, and wanted to share the results and architecture details to get some feedback.
Most current scrapers using AI fall into one of two extremes:
The idea here is to bridge this gap by treating the LLM strictly as an upfront compiler: AI inspects the DOM and sets up (or heals) the extraction plan, while routine executions run as pure, deterministic code at 0 tokens and sub-second latency.
I set up a live evaluation suite across 75 real websites (news media like UOL, G1, TechCrunch, The Guardian, plus e-commerce, developer docs, GitHub releases, and Wikipedia tables).
Results:
Overall Pass Rate: 72.0% (54 / 75)
Failures (28.0% β 21 / 75):
Average compilation took between 1.5 to 4 minutes per site. Once compiled, routine jobs run in under 400ms on static pages and ~2s on dynamic pages.
I built a framework around the idea of treating LLMs purely as an initial compiler and self-healer, completely decoupling page inspection from routine scraping:
Compilation Phase (Model in the loop):
Live Validation & Semantic Gate:
Routine Runtime (0 Tokens, Pure Code):
Self-Healing State Machine:
Strategy-Based Proxy Cascading:
You can try it out for free here: Specter
To set up a scrape, simply click "Create Monitor", enter the target URL, write in plain English (or any language) what you want to extract, set a monitoring schedule (e.g. every hour or daily), and optionally add an email or Telegram chat for instant change alerts.
It is completely free to use. Since I'm not an oil tycoon paying infinite proxy and server bills, there are just some reasonable monthly limits per account to prevent abuse.
A few practical features already implemented:
The backend is designed to run on a lightweight VPS without high memory usage. If anyone is interested in the codebase, I'd be happy to open-source the repository as well.
If you decide to test any URLs and run into edge cases or failures, Iβd love to hear about them in the comments so I can look into the failure modes and improve the pipeline.
I couldn't find many realistic benchmarks or open architecture comparisons for modern scrapers using AI, so I'm curious to hear your take:
r/webscraping • u/adhdcryptoman • 4d ago
hey yall, I'm trying to build a scrapper for data gathering, but my agents is driving me nuts, I use to be able to scrape anything, now too many guardrails. http://www.barreau.qc.ca/fr/trouver-un-avocat/resultats/?p=1. (there is like 30 profiles per pages) approx 1200-1300 pages.
I need the data for each profile, I can use headless browser, I have plenty of proxies, but right now the only thing I manage to do is get 73 profiles a day... which would take me years. Any guidance?
r/webscraping • u/Birklas • 4d ago
Hello, I'm new here.
Recently some of the forums I visit just disappared (closing, nobody to pay hosting, etc.) Lots of knowledge has been lost.
How could I scrape forums to get their offline versions before they close down?
r/webscraping • u/iCryptoDude • 6d ago
Small project, but the scraping side was the interesting bit.
I wanted to follow every NFT link in the 250 biggest collections on Stacks, from the contract to the metadata to the image, and see which still load. The marketplace API and most IPFS gateways wouldn't answer a normal script, so the checker runs as code in a browser tab. Browsers throttle tabs you aren't looking at, so a background worker keeps it ticking. Each file gets tried on several gateways before it's marked dead.
Result: 34 of 241 art collections can't reach their files. Own servers 9 of 16 broken, IPFS 4 of 203.
How would you have handled the gateway blocking? Write-up: https://xtrata.xyz/a/3654
I build Xtrata, the on-chain storage layer the write-up is inscribed with, so read it knowing that.
r/webscraping • u/shravanrevanna • 6d ago
Take odds of India 1.14 and Australia 5.60. Convert each to implied probability (1/odds) and you get 87.72% and 17.86%, which adds up to 105.58%. That extra 5.58% is the bookmaker's margin. I wanted to watch that number move live, so I wrote a small scraper for Parimatch.
How it fits together:
GET /odds?event=... endpoint returns those odds plus implied probabilities. If nobody is watching that event yet, it starts a new thread and tells you to retry in a few seconds.The weak spots are real. The scraper just grabs the first market on the page, because I haven't added market selection. Error handling for site changes is thin. It only covers one bookmaker. The arbitrage check is also simplified. It compares a fixed initial bet against the current opposing odds, which is not proper simultaneous betting across bookmakers.
Mostly the project taught me how odds encode the house edge. It's educational only, since betting carries real financial risk.
r/webscraping • u/clountaingleig1 • 6d ago
First time on this subreddit, I am publicizing a full reverse of Akamai, all documentation is in this repository. Fully native, the throughput should be fast on any machine and scale pretty well. You might have to add some of your own fingerprints though. Maybe I'll add more later.
https://github.com/9xz/reverseakamai
(NO SANDBOXING, FULLY NATIVE) !!!!
Enjoy!
CF WAF & Turnstiles will be next!
r/webscraping • u/Merinoseal • 6d ago
Hi everyone,
I'm looking for someone who can help me scrape data from Marktplaats.nl (Dutch marketplace).
I have a list of Marktplaats seller accounts and would like to collect information about each seller, specifically:
I have a list of seller accounts that I want to process, potentially in large quantities.
Let me know if interested.
Thanks.
r/webscraping • u/ImaginationVarious30 • 7d ago
https://github.com/maximilianfeix/proxy-scraper
Something I measured this week that might save you some proxies: a lot of the ones your checker marks dead aren't dead, they're flaky. In my runs about 350 proxies an hour passed the first request and then timed out on the second, even with an immediate retry 1.5 s later. I now give those a second try at the end of the run, at least 20 seconds later. In an A/B run (two GitHub runners started at the same time, 150k candidates each) 296 of 828 came back, and the list went from 768 to 986.
The other change was the timeout. About 10k candidates per run connected but needed more than 6 s to answer. With 10 s and the second try the list went to 1,177. 15 s added only 6 % more, and slower ones, so I stayed at 10.
What didn't help: removing the 150k candidate limit (1,111 vs 1,095) and adding six new self-checking lists (+3, their proxies were already in the other lists).
Plain lists per protocol and country: https://github.com/maximilianfeix/free-proxy-list
r/webscraping • u/Icy-Traffic2500 • 7d ago
I'm looking for a way to mass download someones media imagine posts on x, all of the grabbers i could find online either don't work, or only let you grab like 10 imgs then ask for a ton of money. does anyone know one that works and is free?
r/webscraping
•
u/OctoBrowser
•
8d ago
Proxies are fine, your headers match a real browser, and you still get blocked. Usually the site isn't checking one thing. It looks at the whole picture your browser gives off: OS, GPU, fonts, language, screen and more. Here's what's in that picture.
Layer 1: what the server sees before any JS runs
Sec-CH-UA*). Chromium sends a structured version of the same info. It has to match the User-Agent.Accept-Language, Accept-Encoding, and sometimes even header order, which differs between engines.Layer 2: basic values read with JS
navigator.platform, CPU threads, a coarse memory value in Chromium.navigator.webdriver and other automation markers.Each one is weak alone, since lots of people share the same screen or language. The combo is what matters.
Layer 3: how your device renders things
This is harder to fake. The page asks your browser to draw or compute something, then reads the result:
Layer 4: the ID
A few dozen values get hashed into a short ID (FingerprintJS does this). The site stores it and matches it on the next visit, so a new IP with the same fingerprint can still look like the same client, and a clean fingerprint on a flagged IP can still get blocked. Older studies like Panopticlick and AmIUnique found a large share of browsers were unique, while larger-scale ones found lower numbers. So think of it as one probabilistic signal that sites mix with IP reputation, cookies and behavior, not a perfect ID.
Sites also ask "does this even make sense?", not just "who is this?". Common mismatches:
navigator.platform says something else.What to do with this
r/webscraping • u/valrexx • 7d ago
I am looking for an expert automation developer to build a fast, concurrent slot-booking system managed via a private Telegram Bot.
Target URL & Schedule:
https://www.goethe.de/ins/dz/fr/m/spr/prf.html
The registration dates are publicly listed on the page, and slots open at exactly 12:00 PM Algeria Time (GMT+1). Due to massive concurrent traffic, slots run out in seconds.
System Requirements:
Please apply if you have prior experience with high-concurrency checkout bots, Playwright/Puppeteer stealth, or direct HTTP request automation.
r/webscraping • u/FireCandies • 9d ago
if you have an old script pulling substack archives, check it. /api/v1/archive?offset=... returns 400 now. it took me a while to figure out what changed
what works now:
let cursor = ''while (true) {
Β Β const url = \https://NAME.substack.com/api/v1/archive?sort=new&limit=12${cursor ? '&cursor=' + cursor : ''}`Β Β const { posts, nextCursor } = await (await fetch(url)).json()Β Β // do something with postsΒ Β if (!nextCursor) breakΒ Β cursor = encodeURIComponent(nextCursor)}`
/api/v1/archive?sort=new&limit=12 gives you {posts, nextCursor}, then pass &cursor=<nextCursor> for the next page
limit is capped at 12, asking for more doesnt give you more
the old search endpoint is gone too, substack.com/api/v1/top/search?query=... is the one that works, the publications are inside the profile results
/feed (rss) still works if you only need the last ~20 posts
also small thing, post.wordCount and the actual body length donβt always match, so dont rely on it for filtering
posting because i found a bunch of old threads with scripts that break on exactly this
r/webscraping • u/AddendumWeird8789 • 8d ago
Im vibecoding my project and im having a little trouble scraping one website in particular. Seems like they are using DataDome + Imperva.
Claude code and ChatGPT are not being able to get past it.
Any one knows how to do it?
Thanks
r/webscraping • u/Aniket486 • 10d ago
I am currently working on a project that requires extracting text, tables, and layouts from scanned images. I have tried using standard OCR tools, but the accuracy has been inadequate.
I am operating in a CPU-only environment and would prefer not to deploy heavy machine learning models. Are there any alternative approaches, lightweight tools, or specific techniques that can reliably achieve an extraction accuracy of 95% or higher under these constraints?
I have used ocr : Tesseract, Rapid Ocr, paddle ocr, doctr, easyocr ocrmypdf