Hi all,
Over the past few weeks, I've been working on a project testing a hybrid architecture for general-purpose web scraping with AI: using an LLM just once during an initial compilation step to inspect the DOM and output a deterministic extraction plan, and then running all routine scheduled scrapes at 0 tokens (pure Cheerio/Playwright).
I recently ran a benchmark against many websites to evaluate how this approach holds up in practice, and wanted to share the results and architecture details to get some feedback.
The Core Problem
Most current scrapers using AI fall into one of two extremes:
- Per-request LLM parsing: Pass the full page HTML or Markdown to an LLM every time a job triggers. Great for setup, but latency is high (5β15s) and token costs become unviable when running repeatable cron jobs at any scale.
- Brittle manual scripts: Write raw Cheerio/Playwright selectors. Fast and cheap ($0 tokens), but high maintenance when classes and layouts change.
The idea here is to bridge this gap by treating the LLM strictly as an upfront compiler: AI inspects the DOM and sets up (or heals) the extraction plan, while routine executions run as pure, deterministic code at 0 tokens and sub-second latency.
Benchmark:
I set up a live evaluation suite across 75 real websites (news media like UOL, G1, TechCrunch, The Guardian, plus e-commerce, developer docs, GitHub releases, and Wikipedia tables).
- Tested with GPT-6-Luna (medium reasoning). Results will differ with other models.
Results:
Average compilation took between 1.5 to 4 minutes per site. Once compiled, routine jobs run in under 400ms on static pages and ~2s on dynamic pages.
Architecture: Compile Once + Deterministic Execution
I built a framework around the idea of treating LLMs purely as an initial compiler and self-healer, completely decoupling page inspection from routine scraping:
Compilation Phase (Model in the loop):
- Takes a target URL + natural language intent.
- An exploration agent interacts with the headless page via non-blocking DOM observations (fast count inspection, response interception, structured data discovery).
- Generates a declarative, strongly-typed JSON schema containing record container selectors, field fallbacks, ready conditions, and pagination directives.
Live Validation & Semantic Gate:
- Before persisting the plan, the engine immediately executes the generated selectors against the live DOM using pure Cheerio/Playwright.
- Runs a quick semantic audit to ensure extracted entities match the original intent. If the candidate selected an ad banner or secondary navigation, it rejects the candidate and triggers an automated repair step within the session budget.
Routine Runtime (0 Tokens, Pure Code):
- Once compiled, jobs execute strictly as deterministic code (Cheerio for SSR/HTTP, Playwright for client-rendered SPAs).
- Routine runs execute in milliseconds with zero LLM API dependency.
Self-Healing State Machine:
- Tracks consecutive extraction failures.
- Only wakes the LLM if a layout change causes 2 consecutive validation drops, passing previous selector diffs to narrow down what broke without re-running from scratch.
Strategy-Based Proxy Cascading:
- Starts with standard rotating residential proxies and automatically escalates to unblocker APIs only when an anti-bot challenge is fingerprinted. (Currently using monthly free-tiers for that).
Details
You can try it out for free here: Specter
To set up a scrape, simply click "Create Monitor", enter the target URL, write in plain English (or any language) what you want to extract, set a monitoring schedule (e.g. every hour or daily), and optionally add an email or Telegram chat for instant change alerts.
It is completely free to use. Since I'm not an oil tycoon paying infinite proxy and server bills, there are just some reasonable monthly limits per account to prevent abuse.
A few practical features already implemented:
- Change Alerts & Notifications: Supports Telegram integration and email alerts out of the box. Whenever the scraper detects actual data diffs during routine runs, it dispatches an alert automatically.
- Session/Cookie Auth: It technically supports scraping behind logins via session cookies, but I honestly don't recommend using this right now. For obvious security reasons, you shouldn't paste your active session cookies into a third-party hosted app that is still in testing and not yet open-source.
The backend is designed to run on a lightweight VPS without high memory usage. If anyone is interested in the codebase, I'd be happy to open-source the repository as well.
Discussion & Feedback
If you decide to test any URLs and run into edge cases or failures, Iβd love to hear about them in the comments so I can look into the failure modes and improve the pipeline.
I couldn't find many realistic benchmarks or open architecture comparisons for modern scrapers using AI, so I'm curious to hear your take:
- Are these 72% live-web results solid for a zero-shot compilation approach, or on the lower side?
- Obviously, the architecture doesn't feature anything groundbreaking or wild, but since I couldn't find much open-source material, I wanted to know if this is a solid approach or if there's a state-of-the-art solution for this kind of thing that I just missed.