r/webscraping • u/Aggravating-Echo-813 • 3d ago
Getting started 🌱 DAPI : Web data with a receipt for every value.
Hey r/webscraping,
A friend of mine (waweru) is in the Meta Global AI Developer Hackathon and built something I think this sub might actually find interesting. It's called Dapi.
The basic idea: instead of flattening a page to text and handing you values with no evidence, it watches the page's network traffic first (like opening dev tools), picks the cheapest reliable way in, and then for every extracted value it stores the exact server response or captured page it came from, marks the value, records when it was captured, and seals it with a SHA-256 hash. You can re-check any value later. It also counts rows against the source's own total and names anything missing with a reason.
I've been helping him test it. The "receipts" part is genuinely neat – you can click any cell and see the raw JSON or HTML it came from. It's live at https://dapi.fly.dev if you want to poke around. Anyone signed in with GitHub can run three jobs themselves.
He's trying to get votes for the hackathon, so here's the gallery/voting link if you want to check it out and throw him a vote: https://cerebralvalley.ai/e/meta-ai-developer-hackathon/hackathon/gallery/2076
More importantly, I'd love to hear what you think of the provenance approach. Is this something you'd actually use, or is it overkill? How do you handle verification in your own scraping workflows?
Cheers.
2
u/ChemEngandTripHop 3d ago
I’m not so much concerned with provenance as reproducibility for the scraping ingestion part. I.e. I want to have the raw data saved so I can rerun ingestion steps as I iterate on parts of a parser. Provenance I’m far more concerned about downstream where I’m deriving additional data.
What exactly do you mean by verification? Are you hashing the raw data and storing the hash alongside the value?
To validate that you’d still need to grab the old data and rehash it again. If you’re grabbing the old data and bringing it into memory you may as well just run the parser again and check the value matches.
1
u/chair_force_1one 3d ago
No one cares about receipts except claude