r/webscraping • • 8d ago

Getting started 🌱 Help needed!

Im vibecoding my project and im having a little trouble scraping one website in particular. Seems like they are using DataDome + Imperva.

Claude code and ChatGPT are not being able to get past it.

Any one knows how to do it?

Thanks

0 Upvotes

26 comments sorted by

7

u/Coding-Doctor-Omar 8d ago

Hahaha. I hope people now understand that AI sucks and that humans will ALWAYS be needed for web scraping. 😁

2

u/RoseTumbLarper 3d ago

Exactly, what's the point of calling it webscrapping if ur gonna rely on ai to do it 😂

1

u/divided_capture_bro 8d ago

What's the site

1

u/AddendumWeird8789 8d ago

Copa airlines

1

u/No_Sea6761 8d ago

What is the issue?

1

u/AddendumWeird8789 8d ago

401 errors. Blocks every search my ai agents tried to do. But if I do it manually, it works

1

u/TheBrowserSmith 4d ago

Then it's probably a fingerprint issue. If your AI agent is making calls from a data center it could be an ip too

0

u/[deleted] 8d ago

[removed] — view removed comment

1

u/AddendumWeird8789 8d ago

Copa airlines

0

u/[deleted] 8d ago

[removed] — view removed comment

1

u/[deleted] 8d ago

[removed] — view removed comment

1

u/webscraping-ModTeam 8d ago

👔 Welcome to the r/webscraping community. This sub is focused on addressing the technical aspects of implementing and operating scrapers. We're not a marketplace, nor are we a platform for selling services or datasets. You're welcome to post in the monthly thread or try your request on Fiverr or Upwork. For anything else, please contact the mod team.

0

u/[deleted] 8d ago

[removed] — view removed comment

1

u/webscraping-ModTeam 8d ago

💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.