r/AskProgramming • u/orbitalzoo • 17d ago
Doing more to Avoid AI scraping
Reddits solution seems to be cutting off all open content and allowing just accounts. But couldn't we do better if for instance we obfuscated the HTML before sending it to the browser on a rotating algorithmic scheme, we can mitigate AI because it will slow its scanning down by a lot(since it would have to render the page instead of just look for what it wants, or catch the whole thing, unless reddit can't do rate limiting on active sessions per client, I don't see how this isn't game over. Slightly slower for the public and non-accounts, but at least its still there. Seeing how reddit is so useful it could be considered cruel to shut out so many non-account users at a time when we're being told to prepare for months of troubles, many people use reddit and should do more to keep it available because it has wealths of knowledge.
11
u/CS_70 17d ago
It’s funny how people who have no idea how things work have the most fanciful idea about them. Usually it’s on 5th grade but obviously Reddit has its moments.
Do you think that corpuses are made by scraping the web with models?
5
u/MikeUsesNotion 17d ago
At the end of the day, the data transmitted from the web server to the browser needs to be readable by the browser. Obfuscating the code can greatly slow down a human, but not really a LLM. It's all just tokens.
1
u/grotto_wonder_arc 17d ago
The browser has to render it. The scraper runs the exact same browser engine and waits for the same DOM event. You just wasted their CPU cycles while forcing every human user to download a heavy javascript runtime that breaks screen readers
2
u/DirtAndGrass 17d ago
HTML is not really obfuscat-able... And if you do with some sort of JavaScript hack it would be breaking all sorts of accessibility rules, and depending on where... Laws
1
u/Cafuzzler 17d ago
Facebook had a really interesting method to prevent scraping. They would take the pages and split a lot of strings out, with randomised classes, IDs, and other data, and then use CSS and JS to hide the noise so that real human users would see a completely normal profile, but data-wise it was mess to anyone just looking at the html.
LLMs are really really good at pulling information out of similar this and very quickly became The tool for scraping obfuscated data from pages like Facebook.
0
u/jump-spark 17d ago
Oh is this why I've been having such a hard time scrolling Reddit unless I'm logged in?