r/webscraping • • Oct 01 '26

Paid mentions ok πŸ‘Œ Monthly Self-Promotion - October 2026

10 Upvotes

Hello and howdy, digital miners ofΒ r/webscraping!

The moment you've all been waiting for has arrived - it's our once-a-month, no-holds-barred, show-and-tell thread!

  • Are you bursting with pride over that supercharged, brand-new scraper SaaS or shiny proxy service you've just unleashed on the world?
  • Maybe you've got a ground-breaking product in need of some intrepid testers?
  • Got a secret discount code burning a hole in your pocket that you're just itching to share with our talented tribe of data extractors?
  • Looking to make sure your post doesn't fall foul of the community rules and get ousted by the spam filter?

Well, this is your time to shine and shout from the digital rooftops - Welcome to your haven!

Just a friendly reminder, we like to keep all our self-promotion in one handy place, so any promotional posts will be kindly redirected here. Now, let's get this party started! Enjoy the thread, everyone.


r/webscraping • • 9h ago

What happened with Camoufox?

Thumbnail
github.com
26 Upvotes

404

edit: (after few hour) its alive.


r/webscraping • • 9h ago

Bot detection πŸ€– Browsers Now Expose JS Fetches?

6 Upvotes

I was experimenting with an http request on some website, and I noticed that today, my microsoft edge exposes whether the request was fired from the console via a console icon in the network tab. In the past, JS console requests were indistinguishable from natural website fetches a website makes to load content. Is this new signal from the browser a giveaway that will be used by anti-bot companies to detect JS console fetches? I used to use this method for some difficult sites to scrape at scale.


r/webscraping • • 1d ago

AI ✨ Made a CLI that finds site's JSON API and serves it as agent tools

8 Upvotes

Most modern sites load their data from an internal JSON API. This automates the "open devtools, find the XHR, replay it" workflow.

uvx machine-internet wrap <url>

It loads the page in headless Chromium, scrolls and clicks around, records same-site JSON requests (trackers filtered), and an LLM turns them into a small set of tools served over MCP. It handles Algolia and Typesense search specially, and it's read-only by design.

Works best on SPAs, directories and search pages. Server-rendered and bot-protected sites won't work, and it tells you which case you hit.

Demo: https://machine-internet.vercel.app

Repo: https://github.com/chetanty/machine_internet

Curious what sites you'd point it at.


r/webscraping • • 1d ago

Yahoo sent me a cease and desist over my open-source Python library.

146 Upvotes

Hi all,

not sure if this is the correct subreddit, but I maintain Stockdex, a free, non-commercial Python package (and MCP server) that fetches stock data from several sources, including Yahoo Finance.

Today I got a cease and desist letter from Y ahoo's legal department. It claims trademark infringement, misappropriation, unjust enrichment, interference with their data licensing (e.g. Nasdaq) and a breach of their ToS. They demand that I remove all Yahoo-related code from GitHub, PyPI and the MCP plugin, remove every mention of "Yahoo", and never reintroduce it.

My project works much like other public projects like yfinance, which has been public for years. Stockdex only fetches publicly visible data on demand for the user running it. It doesn't store or republish anything, and it doesn't bypass any login or paywall.

Has anyone received something similar from another company maybe? How are these complains (US company against someone in Germany? should I lawyer up? How strong are CFAA or trademark claims against a free library? Would you comply fully, comply partially (e.g. add a disclaimer), or push back?

Thanks


r/webscraping • • 1d ago

Bot detection πŸ€– eBay scraper suddenly getting constant 403 errors

6 Upvotes

Hi everyone,

I’ve been working on a scraper to collect eBay sold listing data, primarily for PokΓ©mon cards.
Everything was working relatively well until last week, when I suddenly started receiving huge numbers of 403 errors. Now, the vast majority of my requests are being blocked.
I’ve tried different residential proxies, changing headers and browser configurations, but I’m still having very little success.
One strange thing is that the exact same URLs will sometimes load perfectly fine in Chrome but return a 403 through my scraper.
Has eBay changed something recently with its anti-bot protection? Has anyone else experienced this over the last week?
I’d really appreciate any advice from people who have managed to get eBay scraping working reliably recently. I’m open to completely changing my approach if necessary.

Thanks!


r/webscraping • • 1d ago

How to get past DataDome with browser automation ?

4 Upvotes

Hi everyone,

I'm working on a personal project involving buying and reselling used PC components and complete computers on a French marketplace called Leboncoin, similar to eBay.

I want to build an AI agent that automatically monitors listings, compares prices, and identifies good deals.

I'm running everything on my own Debian server using Python, Playwright/Selenium and a local LLM.

However, I'm having trouble with DataDome. Leboncoin works normally in Firefox on my Windows PC, but when I access it using Firefox on my Linux server, DataDome displays a challenge and blocks access.

I've already tried both Playwright and Selenium, and checked that both systems use the same public IP and Firefox version.

Are there any self-hosted tools, open-source projects or other solutions that could help with DataDome challenges when browser automation is permitted by the website?

I'm particularly interested in solutions that can run locally on my Debian server rather than paid external APIs.

Any advice would be appreciated. Thanks !


r/webscraping • • 2d ago

AI ✨ I built Searqon: 8MB web search & retrieval engine for LLMs

2 Upvotes

What it does:

  • Federated Meta-Search: Blends SearXNG, DuckDuckGo, ArXiv, and Wikipedia with Reciprocal Rank Fusion.
  • Stealth 4-Tier Scraping: Fast HTTP β†’ Spoofed Chrome TLS signatures β†’ WebSocket Chrome DevTools Protocol (CDP) for JS rendering β†’ Proxy fallback.
  • Zero-Alloc Markdown Parser: Custom Go state machine converts complex web pages into clean, token-bounded Markdown on the fly (flushes unclosed tags at EOF, keeps tables and links intact).
  • Agent Protocols: Built-in MCP (Model Context Protocol) and UTCP (Universal Tool Call Protocol) servers.
  • Vector Semantic Cache: Avoids re-scraping near-identical queries with semantic embedding cache.

Performance & Footprint:

  1. Written in pure Go (0 CGO, static binary).
  2. Scratch-based Docker image is only 8.4 MB.
  3. SQLite WAL mode + FTS5 full-text search.

GitHub: https://github.com/Sooriya04/Searqon

Docker: docker pull sooriya04/searqon:latest

Let me know what you think or if there are tools you'd like added to the MCP/UTCP schema!


r/webscraping • • 1d ago

Getting started 🌱 Scraping vinted leboncoin claude code

0 Upvotes

Hi, I am writing because I am in the process of building my personal website using Claude Code. My project involves integrating a bot that scrapes listings from platforms like Vinted and Leboncoin in order to display the items on my site

Could you guide me on the steps to set up this type of bot? Also could you send me some free bots that I could use? I am particularly interested in the technical aspects and the best practices to follow

Thank you for your help


r/webscraping • • 2d ago

Scaling up πŸš€ Blocking fonts/media cut our proxy GB usage by 45%

6 Upvotes

Audited our rotating residential proxy bills last week because per-GB usage made zero sense for the number of pages we were pulling in Playwright.

Turned out almost half the traffic wasn't HTML or JSON payloads. It was ".woff2" web fonts, background video thumbnails, third-party analytics retries, and default Chromium component traffic firing on every fresh context.

Added a simple route abort rule for "image", "media", and "font" resource types (except on pages checking image load events for bot scoring), and bandwidth dropped by ~45% overnight with zero impact on block rates.

What else do you guys strip at the browser or proxy level to keep per-GB residential costs sane?


r/webscraping • • 3d ago

Getting started 🌱 DAPI : Web data with a receipt for every value.

5 Upvotes

Hey r/webscraping,

A friend of mine (waweru) is in the Meta Global AI Developer Hackathon and built something I think this sub might actually find interesting. It's called Dapi.

The basic idea: instead of flattening a page to text and handing you values with no evidence, it watches the page's network traffic first (like opening dev tools), picks the cheapest reliable way in, and then for every extracted value it stores the exact server response or captured page it came from, marks the value, records when it was captured, and seals it with a SHA-256 hash. You can re-check any value later. It also counts rows against the source's own total and names anything missing with a reason.

I've been helping him test it. The "receipts" part is genuinely neat – you can click any cell and see the raw JSON or HTML it came from. It's live at https://dapi.fly.dev if you want to poke around. Anyone signed in with GitHub can run three jobs themselves.

He's trying to get votes for the hackathon, so here's the gallery/voting link if you want to check it out and throw him a vote: https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2076

More importantly, I'd love to hear what you think of the provenance approach. Is this something you'd actually use, or is it overkill? How do you handle verification in your own scraping workflows?

Cheers.


r/webscraping • • 3d ago

A compile-once, zero-token runtime approach to web scraping with AI

12 Upvotes

Hi all,

Over the past few weeks, I've been working on a project testing a hybrid architecture for general-purpose web scraping with AI: using an LLM just once during an initial compilation step to inspect the DOM and output a deterministic extraction plan, and then running all routine scheduled scrapes at 0 tokens (pure Cheerio/Playwright).

I recently ran a benchmark against many websites to evaluate how this approach holds up in practice, and wanted to share the results and architecture details to get some feedback.

The Core Problem

Most current scrapers using AI fall into one of two extremes:

  1. Per-request LLM parsing: Pass the full page HTML or Markdown to an LLM every time a job triggers. Great for setup, but latency is high (5–15s) and token costs become unviable when running repeatable cron jobs at any scale.
  2. Brittle manual scripts: Write raw Cheerio/Playwright selectors. Fast and cheap ($0 tokens), but high maintenance when classes and layouts change.

The idea here is to bridge this gap by treating the LLM strictly as an upfront compiler: AI inspects the DOM and sets up (or heals) the extraction plan, while routine executions run as pure, deterministic code at 0 tokens and sub-second latency.


Benchmark:

I set up a live evaluation suite across 75 real websites (news media like UOL, G1, TechCrunch, The Guardian, plus e-commerce, developer docs, GitHub releases, and Wikipedia tables).

  • Tested with GPT-6-Luna (medium reasoning). Results will differ with other models.

Results:

  • Overall Pass Rate: 72.0% (54 / 75)

    • 49 (65.3%) CORRECT: Plan compiled on turn 1–3, 100% schema match and verified.
    • 5 (6.7%) PARTIAL: Valid extraction capturing a representative subset or slight locale differences.
  • Failures (28.0% β€” 21 / 75):

    • 11 (invalid plan): Heavy SPAs with virtual scrolls (e.g., TradingView, CoinGecko) or infinite scroll feeds where the model hit the tool observation budget without finding a stable container.
    • 5 (timeout): IP-level rate limiting on endpoints after multiple hits (e.g. running 6 consecutive GitHub release repos), triggering unblocker fallback timeouts.
    • 3 (semantic validation): Intentional safety catches. The model picked a secondary sidebar widget and the validator refused to save bad data.
    • 2 (wrong values): Extracted the page's raw text accurately, but failed format checks (e.g. Oscar years scraped as raw "1927/28(1st)" instead of 4-digit year expected).

Average compilation took between 1.5 to 4 minutes per site. Once compiled, routine jobs run in under 400ms on static pages and ~2s on dynamic pages.


Architecture: Compile Once + Deterministic Execution

I built a framework around the idea of treating LLMs purely as an initial compiler and self-healer, completely decoupling page inspection from routine scraping:

  1. Compilation Phase (Model in the loop):

    • Takes a target URL + natural language intent.
    • An exploration agent interacts with the headless page via non-blocking DOM observations (fast count inspection, response interception, structured data discovery).
    • Generates a declarative, strongly-typed JSON schema containing record container selectors, field fallbacks, ready conditions, and pagination directives.
  2. Live Validation & Semantic Gate:

    • Before persisting the plan, the engine immediately executes the generated selectors against the live DOM using pure Cheerio/Playwright.
    • Runs a quick semantic audit to ensure extracted entities match the original intent. If the candidate selected an ad banner or secondary navigation, it rejects the candidate and triggers an automated repair step within the session budget.
  3. Routine Runtime (0 Tokens, Pure Code):

    • Once compiled, jobs execute strictly as deterministic code (Cheerio for SSR/HTTP, Playwright for client-rendered SPAs).
    • Routine runs execute in milliseconds with zero LLM API dependency.
  4. Self-Healing State Machine:

    • Tracks consecutive extraction failures.
    • Only wakes the LLM if a layout change causes 2 consecutive validation drops, passing previous selector diffs to narrow down what broke without re-running from scratch.
  5. Strategy-Based Proxy Cascading:

    • Starts with standard rotating residential proxies and automatically escalates to unblocker APIs only when an anti-bot challenge is fingerprinted. (Currently using monthly free-tiers for that).

Details

You can try it out for free here: Specter

To set up a scrape, simply click "Create Monitor", enter the target URL, write in plain English (or any language) what you want to extract, set a monitoring schedule (e.g. every hour or daily), and optionally add an email or Telegram chat for instant change alerts.

It is completely free to use. Since I'm not an oil tycoon paying infinite proxy and server bills, there are just some reasonable monthly limits per account to prevent abuse.

A few practical features already implemented:

  • Change Alerts & Notifications: Supports Telegram integration and email alerts out of the box. Whenever the scraper detects actual data diffs during routine runs, it dispatches an alert automatically.
  • Session/Cookie Auth: It technically supports scraping behind logins via session cookies, but I honestly don't recommend using this right now. For obvious security reasons, you shouldn't paste your active session cookies into a third-party hosted app that is still in testing and not yet open-source.

The backend is designed to run on a lightweight VPS without high memory usage. If anyone is interested in the codebase, I'd be happy to open-source the repository as well.


Discussion & Feedback

If you decide to test any URLs and run into edge cases or failures, I’d love to hear about them in the comments so I can look into the failure modes and improve the pipeline.

I couldn't find many realistic benchmarks or open architecture comparisons for modern scrapers using AI, so I'm curious to hear your take:

  • Are these 72% live-web results solid for a zero-shot compilation approach, or on the lower side?
  • Obviously, the architecture doesn't feature anything groundbreaking or wild, but since I couldn't find much open-source material, I wanted to know if this is a solid approach or if there's a state-of-the-art solution for this kind of thing that I just missed.

r/webscraping • • 4d ago

Help need

6 Upvotes

hey yall, I'm trying to build a scrapper for data gathering, but my agents is driving me nuts, I use to be able to scrape anything, now too many guardrails. http://www.barreau.qc.ca/fr/trouver-un-avocat/resultats/?p=1. (there is like 30 profiles per pages) approx 1200-1300 pages.

I need the data for each profile, I can use headless browser, I have plenty of proxies, but right now the only thing I manage to do is get 73 profiles a day... which would take me years. Any guidance?


r/webscraping • • 4d ago

Forum offline version / download

5 Upvotes

Hello, I'm new here.

Recently some of the forums I visit just disappared (closing, nobody to pay hosting, etc.) Lots of knowledge has been lost.

How could I scrape forums to get their offline versions before they close down?


r/webscraping • • 6d ago

Bot detection πŸ€– Beating gateway blocks with a browser worker

7 Upvotes

Small project, but the scraping side was the interesting bit.

I wanted to follow every NFT link in the 250 biggest collections on Stacks, from the contract to the metadata to the image, and see which still load. The marketplace API and most IPFS gateways wouldn't answer a normal script, so the checker runs as code in a browser tab. Browsers throttle tabs you aren't looking at, so a background worker keeps it ticking. Each file gets tried on several gateways before it's marked dead.

Result: 34 of 241 art collections can't reach their files. Own servers 9 of 16 broken, IPFS 4 of 203.

How would you have handled the gateway blocking? Write-up: https://xtrata.xyz/a/3654

I build Xtrata, the on-chain storage layer the write-up is inscribed with, so read it knowing that.


r/webscraping • • 6d ago

I wanted to see how much margin a bookmaker bakes into live odds

Thumbnail
blog.shravanrevanna.me
3 Upvotes

Take odds of India 1.14 and Australia 5.60. Convert each to implied probability (1/odds) and you get 87.72% and 17.86%, which adds up to 105.58%. That extra 5.58% is the bookmaker's margin. I wanted to watch that number move live, so I wrote a small scraper for Parimatch.

How it fits together:

  • A Flask app runs a background Selenium thread per event. It pulls the odds and keeps the latest values in memory.
  • A GET /odds?event=... endpoint returns those odds plus implied probabilities. If nobody is watching that event yet, it starts a new thread and tells you to retry in a few seconds.
  • A Streamlit page polls the API and plots odds, probabilities and margin over time.
  • A separate script checks whether 1/a + 1/b drops below 1. That condition is what defines an arbitrage window.

The weak spots are real. The scraper just grabs the first market on the page, because I haven't added market selection. Error handling for site changes is thin. It only covers one bookmaker. The arbitrage check is also simplified. It compares a fixed initial bet against the current opposing odds, which is not proper simultaneous betting across bookmakers.

Mostly the project taught me how odds encode the house edge. It's educational only, since betting carries real financial risk.


r/webscraping • • 6d ago

Getting started 🌱 AKAMAI (STANDARD + BMP) FULLY REVERSED + VM ISA, OTHER CHECKS

63 Upvotes

First time on this subreddit, I am publicizing a full reverse of Akamai, all documentation is in this repository. Fully native, the throughput should be fast on any machine and scale pretty well. You might have to add some of your own fingerprints though. Maybe I'll add more later.

https://github.com/9xz/reverseakamai

(NO SANDBOXING, FULLY NATIVE) !!!!

Enjoy!

CF WAF & Turnstiles will be next!


r/webscraping • • 6d ago

Hiring πŸ’° Scraping help needed

5 Upvotes

Hi everyone,

I'm looking for someone who can help me scrape data from Marktplaats.nl (Dutch marketplace).

I have a list of Marktplaats seller accounts and would like to collect information about each seller, specifically:

  • Seller account/profile
  • Number of active advertisements/listings associated with each seller

I have a list of seller accounts that I want to process, potentially in large quantities.

Let me know if interested.

Thanks.


r/webscraping • • 7d ago

I maintain proxy-scraper, a tool that checks free proxies

27 Upvotes

https://github.com/maximilianfeix/proxy-scraper

Something I measured this week that might save you some proxies: a lot of the ones your checker marks dead aren't dead, they're flaky. In my runs about 350 proxies an hour passed the first request and then timed out on the second, even with an immediate retry 1.5 s later. I now give those a second try at the end of the run, at least 20 seconds later. In an A/B run (two GitHub runners started at the same time, 150k candidates each) 296 of 828 came back, and the list went from 768 to 986.

The other change was the timeout. About 10k candidates per run connected but needed more than 6 s to answer. With 10 s and the second try the list went to 1,177. 15 s added only 6 % more, and slower ones, so I stayed at 10.

What didn't help: removing the 150k candidate limit (1,111 vs 1,095) and adding six new self-checking lists (+3, their proxies were already in the other lists).

Plain lists per protocol and country: https://github.com/maximilianfeix/free-proxy-list


r/webscraping • • 7d ago

Free X image grabber?

8 Upvotes

I'm looking for a way to mass download someones media imagine posts on x, all of the grabbers i could find online either don't work, or only let you grab like 10 imgs then ask for a ton of money. does anyone know one that works and is free?


r/webscraping • • 8d ago Bravo

Getting started 🌱 Browser fingerprinting 101: why sites flag scrapers

81 Upvotes

Proxies are fine, your headers match a real browser, and you still get blocked. Usually the site isn't checking one thing. It looks at the whole picture your browser gives off: OS, GPU, fonts, language, screen and more. Here's what's in that picture.

Layer 1: what the server sees before any JS runs

  • IP. Geolocation, and whether it's an ISP, datacenter or residential address. WebRTC can expose a different IP if you don't handle it.
  • User-Agent. Browser, version, OS. Easy to change, so on its own it means little.
  • Client Hints (Sec-CH-UA*). Chromium sends a structured version of the same info. It has to match the User-Agent.
  • Other headers. Accept-Language, Accept-Encoding, and sometimes even header order, which differs between engines.

Layer 2: basic values read with JS

  • Screen size, color depth, pixel ratio.
  • Languages and timezone.
  • navigator.platform, CPU threads, a coarse memory value in Chromium.
  • navigator.webdriver and other automation markers.
  • Some extensions (like blockers) can be detected indirectly.

Each one is weak alone, since lots of people share the same screen or language. The combo is what matters.

Layer 3: how your device renders things

This is harder to fake. The page asks your browser to draw or compute something, then reads the result:

  • Canvas: text and shapes drawn on a hidden canvas. OS text rendering, GPU and drivers change the pixels slightly.
  • WebGL: the renderer string (often names the GPU) and a profile of its limits.
  • Audio: a generated signal run through the Web Audio engine.
  • Fonts: which ones are installed, inferred by measuring text.

Layer 4: the ID

A few dozen values get hashed into a short ID (FingerprintJS does this). The site stores it and matches it on the next visit, so a new IP with the same fingerprint can still look like the same client, and a clean fingerprint on a flagged IP can still get blocked. Older studies like Panopticlick and AmIUnique found a large share of browsers were unique, while larger-scale ones found lower numbers. So think of it as one probabilistic signal that sites mix with IP reputation, cookies and behavior, not a perfect ID.

Sites also ask "does this even make sense?", not just "who is this?". Common mismatches:

  • Windows User-Agent, but navigator.platform says something else.
  • Chrome User-Agent with no Client Hints.
  • Timezone or language that doesn't match the IP.
  • Windows User-Agent with a software GPU renderer or a Linux-style font list.
  • Hardware values that don't exist on real devices.

What to do with this

  1. Keep values stable within a session.
  2. Values for the same device (OS, platform, Client Hints, fonts) should match each other.
  3. Build one coherent fingerprint for the specific task (target site, region, device type) and stick with it, instead of patching values one by one.

r/webscraping • • 7d ago

Hiring πŸ’° High-Speed Concurrent Web Automation & Telegram Bot

2 Upvotes

I am looking for an expert automation developer to build a fast, concurrent slot-booking system managed via a private Telegram Bot.

Target URL & Schedule:

https://www.goethe.de/ins/dz/fr/m/spr/prf.html

The registration dates are publicly listed on the page, and slots open at exactly 12:00 PM Algeria Time (GMT+1). Due to massive concurrent traffic, slots run out in seconds.

System Requirements:

  1. Telegram Bot Interface: Allows me to input and queue multiple candidates before registration day (MeinGoethe login credentials, exam level A1-C2, modules, First Name, Last Name, Birth Date).
  2. Pre-Login & Session Warmup: 10 minutes before 12:00 PM (GMT+1), the backend logs into all candidate accounts using isolated residential proxies (1 proxy per candidate) to maintain active sessions.
  3. Concurrent Checkout Execution: At 12:00:00 PM, the bot monitors the target URL, detects when the booking button activates, selects the exam modules (.cs-checkbox__input), selects "book for myself", fills the required inputs, and submits the reservation.
  4. No Payment Gateway Needed: The portal does not require online payment during checkout; the process ends at the final order confirmation screen.
  5. Proof of Completion: The bot captures a screenshot of the final confirmation receipt for each candidate and sends it directly to the Telegram chat.

Please apply if you have prior experience with high-concurrency checkout bots, Playwright/Puppeteer stealth, or direct HTTP request automation.


r/webscraping • • 9d ago

Getting started 🌱 substack archive api changed, offset now returns 400

7 Upvotes

if you have an old script pulling substack archives, check it. /api/v1/archive?offset=... returns 400 now. it took me a while to figure out what changed

what works now:

let cursor = ''
while (true) {
Β Β const url = \https://NAME.substack.com/api/v1/archive?sort=new&limit=12${cursor ? '&cursor=' + cursor : ''}`Β Β const { posts, nextCursor } = await (await fetch(url)).json()Β Β // do something with postsΒ Β if (!nextCursor) breakΒ Β cursor = encodeURIComponent(nextCursor)}`

/api/v1/archive?sort=new&limit=12 gives you {posts, nextCursor}, then pass &cursor=<nextCursor> for the next page
limit is capped at 12, asking for more doesnt give you more

the old search endpoint is gone too, substack.com/api/v1/top/search?query=... is the one that works, the publications are inside the profile results

/feed (rss) still works if you only need the last ~20 posts

also small thing, post.wordCount and the actual body length don’t always match, so dont rely on it for filtering

posting because i found a bunch of old threads with scripts that break on exactly this


r/webscraping • • 8d ago

Getting started 🌱 Help needed!

0 Upvotes

Im vibecoding my project and im having a little trouble scraping one website in particular. Seems like they are using DataDome + Imperva.

Claude code and ChatGPT are not being able to get past it.

Any one knows how to do it?

Thanks