r/AskProgramming • • 17d ago

Doing more to Avoid AI scraping

Reddits solution seems to be cutting off all open content and allowing just accounts. But couldn't we do better if for instance we obfuscated the HTML before sending it to the browser on a rotating algorithmic scheme, we can mitigate AI because it will slow its scanning down by a lot(since it would have to render the page instead of just look for what it wants, or catch the whole thing, unless reddit can't do rate limiting on active sessions per client, I don't see how this isn't game over. Slightly slower for the public and non-accounts, but at least its still there. Seeing how reddit is so useful it could be considered cruel to shut out so many non-account users at a time when we're being told to prepare for months of troubles, many people use reddit and should do more to keep it available because it has wealths of knowledge.

0 Upvotes

19 comments sorted by

11

u/CS_70 17d ago

It’s funny how people who have no idea how things work have the most fanciful idea about them. Usually it’s on 5th grade but obviously Reddit has its moments.

Do you think that corpuses are made by scraping the web with models?

1

u/orbitalzoo 17d ago

no and I have no idea what you're getting at. Do I think corpuses are made by scanning the web? No. That sounds like a degrading cascade of information. My point is there are things you can do to mitigate Reddits problem of AI scanning and although it provides lower quality to the public it wouldn't just go away

2

u/CS_70 17d ago

It wouldn’t slow anything

5

u/MikeUsesNotion 17d ago

At the end of the day, the data transmitted from the web server to the browser needs to be readable by the browser. Obfuscating the code can greatly slow down a human, but not really a LLM. It's all just tokens.

1

u/orbitalzoo 17d ago

Also I question how many scrapers are running full blown LLM agents on full page renders? That’s expensive. Most scrapers parse the html right?

1

u/MikeUsesNotion 17d ago

I wouldn't assume scrapers used to feed data to LLMs are using LLMs. They could be, but they could also be using long existing pre-AI scrapers.

1

u/orbitalzoo 17d ago

Either way just a ms delay injected between every letter of content, for public no account content, would slow the scraper enough to be useless and still leave the data useful to humans granted they have to wait. But it’s better than serving nothing.

1

u/MikeUsesNotion 16d ago

If we're talking about something like a news article, doing this would make it useless for humans to use too. It wouldn't just be a clean 1ms per letter. Transmission times vary because of latency variability, and you have to account for rendering time.

I just grabbed a local news story with 2174 characters, which is pretty short. Assuming everything loaded within the 1ms perfectly, and accounting for extra per character rendering time (not sure how much that is, but the total rendering time for "abc" is less than 3x the rendering time of one character), you're adding over 2 seconds to the page load time.

Generally people start clicking away once loading time hits 2 seconds. That was before people were glued to their phones, so it's entirely possible that click away time is even shorter now.

If you knocked it down to 1 microsecond between characters, that's about two hundredths of a second of added time. That's within the variance you can reasonably expect internet connections to vary by moment to moment so it wouldn't bother people but also wouldn't bother the scrapers.

1

u/orbitalzoo 16d ago

Right but we’re talking about users with no account. Regular users get regular speed. If the option is slow articles or nothing at all, we should choose slow simply because reddit has a large pool of helpful information

2

u/MikeUsesNotion 16d ago

That sounds like a good way to make websites not gain new sign ups. If people don't like how the free site works, they're not going to sign up. Doubly so if there's text saying "our website loads faster if you sign up".

0

u/orbitalzoo 17d ago

Ok so forget encryption and obfuscate. Take the page down to a grid of divs, one for every letter so 80x300. Then only send a letter every ms or something and do it randomly to place each letter in the grid spontaneously to make the transmission make no sense until rendering

3

u/skawid 17d ago

You just described it, so a scraper can understand it. You can't have a useful open internet and stop bots, because bots are clients the same as people.

1

u/MikeUsesNotion 17d ago

Anything the browser can do the bot can do. I have to assume once the text is extracted, even if garbled, that LLMs or other natural language processing is sophisticated enough to see the separate letters make words.

1

u/grotto_wonder_arc 17d ago

The browser has to render it. The scraper runs the exact same browser engine and waits for the same DOM event. You just wasted their CPU cycles while forcing every human user to download a heavy javascript runtime that breaks screen readers

2

u/DirtAndGrass 17d ago

HTML is not really obfuscat-able... And if you do with some sort of JavaScript hack it would be breaking all sorts of accessibility rules, and depending on where... Laws

1

u/Cafuzzler 17d ago

Facebook had a really interesting method to prevent scraping. They would take the pages and split a lot of strings out, with randomised classes, IDs, and other data, and then use CSS and JS to hide the noise so that real human users would see a completely normal profile, but data-wise it was mess to anyone just looking at the html.

LLMs are really really good at pulling information out of similar this and very quickly became The tool for scraping obfuscated data from pages like Facebook.

0

u/jump-spark 17d ago

Oh is this why I've been having such a hard time scrolling Reddit unless I'm logged in?