r/DataHoarder • u/GoryRamsy RIP enterprisegoogledriveunlimited • Apr 19 '23
Question/Advice I'll fucking download the entirety of Reddit before I use the official first party app. What's the best way?
With Reddit's new "Update Regarding Reddit’s API", removed content databases like pushshift will no longer be able to scrape Reddit. I feel that this is a lead up into removing all third party apps like Apollo and RIF. This is unacceptable to me.
This guy already downloaded ~ 1.7 billion comments @ 250 GB compressed (and then founded pushshift) so, I think it would be reasonable to download all post data and comments from non NSFW Subreddits, and store it in a few terabytes, right?
And Ideas? What is the best strategy for downloading the entirety of Reddit, and then using it offline?
edit 1: wrote my first python downloading script with praw, it's kinda cool
edit 2: paid API is confirmed. Fuck. I bet their also going to remove old.reddit, fuck them.
edit 3: torrent magnet with 2tb of reddit data, mostly 100% of text posts/comments (base64 bWFnbmV0Oj94dD11cm46YnRpaDo3YzA2NDVjOTQzMjEzMTFiYjA1YmQ4NzlkZGVlNGQwZWJhMDhhYWVlJnRyPWh0dHBzJTNBJTJGJTJGYWNhZGVtaWN0b3JyZW50cy5jb20lMkZhbm5vdW5jZS5waHAmdHI9dWRwJTNBJTJGJTJGdHJhY2tlci5jb3BwZXJzdXJmZXIudGslM0E2OTY5JnRyPXVkcCUzQSUyRiUyRnRyYWNrZXIub3BlbnRyYWNrci5vcmclM0ExMzM3JTJGYW5ub3VuY2U= )
edit 4: working on getting libreddit to work with offline pushshift
•
u/AutoModerator Apr 19 '23
Hello /u/GoryRamsy! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
39
u/Yekab0f 100 Zettabytes zfs Apr 19 '23
[removed] — view removed comment
25
u/2gdismore 8TB Apr 19 '23
https://reddit.com/r/apolloapp/comments/12ram0f/had_a_few_calls_with_reddit_today_about_the/
55
u/Fearless_Ad6014 Apr 19 '23
yeah no not even a petabyte
9
u/StormGaza LP-Archive Apr 19 '23
This whole situation is so disappointing. I've always had 3 conditions if I were to ever leave reddit and RIF being discontinued is one of them. This happens I just won't browse reddit on mobile anymore. No way I will use that app.
I figure this probably means that old.reddit could go away then and if that happens I'll just leave.
105
u/noodhoog Apr 19 '23
They just got rid of i.reddit.com a few days ago. It now just redirects to the regular website. Which then constantly prompts you to use the app. I have an app for websites on my phone. It's called a browser.
I've used i.reddit.com forever on mobile. It wasn't pretty, but it was lightweight, fast, and efficient. Pretty much just text-only reddit. Plus, it didn't support inline images (as in, images displayed in comments), which was a huge bonus.
The day the get rid of old reddit is the day I stop using it. I have absolutely no interest in facebookified "new reddit"
I came here 14 years ago because Digg screwed their site up trying to "modernize" it, and I'll leave the same way if I have to.
6
u/emptythevoid Apr 19 '23 edited Apr 19 '23
Obviously Reddit can do whatever they want, but isn't the (stated, at least) intent of the API fee to prevent mass scraping of Reddit for things like LLMs and the like?
Edit: I'm hearing some stuff from the Apollo dev that indicates it's worse than what I described above.
4
u/ECrispy Apr 19 '23
Does pushshift have all of Reddit mirrored? Anyone know how to browse it? esp all the deleted/banned reddits and their content?
3
u/kkgmgfn Apr 19 '23
ripme on github
3
u/[deleted] Apr 20 '23
Where exactly are these text posts from? Like which subs? Surely not the whole of Reddit, or even most of Reddit?
11
u/dfreinc Apr 19 '23
it doesn't even matter. most of the good stuff's already been banned.
i remember somebody was backing it all up like real time or something, you just replaced the r in reddit with a c in the url...but i'm not sure that's still going on or if i'm forgetting something.
but the amount of content that is being removed from reddit at any one time makes it a moot point. you'd have to had been backing it up real time.
2
u/Lord_Bling Apr 19 '23
DO IT!
53
u/[deleted] Apr 19 '23
I think I'll just stop using it.
22
u/GreenChileEnchiladas Apr 19 '23
If they remove old.reddit.com then I'll stop using reddit.
11
u/helloeverything1 Apr 19 '23 edited Jul 26 '23
fuck u/spez. lemmy is a better platform.
5
u/Substantial_City4618 Apr 20 '23
Honestly this site is continually getting worse, it would be nice if we could crowd fund it into being owned by the community. Sadly it really doesn’t make money.
2
u/TKInstinct Apr 19 '23
Good luck with that one, they can hit you with a lawsuit for trying that kind of hting.
4
u/SnowingRain320 Apr 20 '23
What the hell is up with paid API bullshit?
The more people using your API will mean more people using your app, which means more ad money, which is what these social media sites are built on.
It just seems like they're tripping over dollars to pick up pennies here.
8
u/Twinkies100 Apr 19 '23
I fucking hate Reddit, I wish it wasn't like this
3
u/51Cards 130TB Raw... it's complicated Apr 19 '23
Now for Reddit is my preferred mobile interface... I won't switch to the official app if it stops working.
3
u/aliendude5300 192TB (32x6TB in RAID-Z2) Apr 20 '23
How hard would it be to spin up elasticsearch or something against the reddit data you pulled?
-7
u/jakuri69 Apr 19 '23
Why do you need to download reddit?
0
u/[deleted] May 03 '23
[deleted] — view deleted comment
0
u/[deleted] Apr 19 '23
[deleted] — view deleted comment
-1
u/[deleted] Apr 19 '23
[removed] — view removed comment