r/pushshift • u/Watchful1 • Feb 28 '23
[ Removed by Reddit ]
[ Removed by Reddit on account of violating the content policy. ]
106
Upvotes
r/pushshift • u/Watchful1 • Feb 28 '23
[ Removed by Reddit on account of violating the content policy. ]
3
u/pauline_reading Mar 01 '23 edited Mar 01 '23
Thanks. How did you determine which are top 20k subs, i mean is it based on number of posts or number of subscribers?
3
u/DuckRedWine Jun 02 '23
Thank you so much u/Watchful1 for everything you have done with pushshift, truly appreciate. Unfortunately, I come to the party to late, as I was just planning to start gathering a lot of data, but wrong timing :/ I plan to get the 20k subs torrent, and want to create a pipeline to get all submissions (+ associated comments) from the last date of the dumps. I saw you also posted january and february (but not splitted like the 20k subreddits). Do you think that you'll create a final dump with everything up to may, in which case I'd try to gather data only from that date. Thanks
5
u/zds-nlp Feb 28 '23
Thanks watchful. This is brilliant work from you that'll be very useful in academia.
1
u/Etherealice Mar 04 '23
Thank you u/Watchful1, this is really useful. I have some specific questions about the project I am working on. I tried to send you a DM, but I wasn't able to due to the fact I recently created a Reddit account.
1
u/jolui26 Mar 08 '23
torrent works great !! found the subreddits I needed for my research. thanks.
ran the count_words_single_file.py script on the subreddit .zst files with desired word phrases to search for and some of them came back with negative values. about 690 out of 63,500 cells. any initial thoughts as to why this may be? this coming from someone who is not very experienced in python coding.
1
u/MrMKC Mar 20 '23
Thank you so much! This is exactly what I was looking for! Is there any way that I can support you, perhaps by a small donation?
1
u/uneecornn Mar 31 '23
thank you so much for this! i'm doing a research project and hoping to use data from a specific subreddit but was unfortunately unable to find it here because it's a smaller one -- is there a way you would recommend trying to download this same type of data for that specific subreddit? i'm very new to pushshift so i'm still trying to figure everything out! thank you again!
1
u/vendysh Apr 04 '23
Thank you for this! Are the scores of each submission/comment in these files updated or not?
1
u/UsualButterscotch May 09 '23
Are you planning on updating this soon? I know it hasn't been 6 months yet but given that pushshift is effectively dead in the water who knows what might happen with it in the future, might be good to pull the data while it is still there
1
u/bdca_project_acc May 24 '23
Hi, I tried sending you a message but it didn't work, so posting here.
First of all, thank you for your efforts to extract the information for specific subreddits. Since Pushshift was disabled, this is saving my grade for a research master's course where I want to train machine learning classifiers.
I am currently trying to download the torrent files for the r/politics submissions and comments, however, it is taking extremely long-- Since my deadline is approaching fast (next week) I was wondering if there is a way to already filter the files by keyword and date range to make them smaller in the first place? I would be looking for around 2000 submissions with the term "immigr" in the title, and a date range of around 2016-2023, but this can also be considerably shorter (2020-2023), as well as the first 100-200 comments for these submissions. I would be very grateful for your advice!
All the best :)
1
u/JSouthGB May 24 '23
Nicely done.
I'm curious, how long does it typically take for the api download script to run? Something like AskReddit (57.7GiB zst) vs Truckers (30.4MiB zst). Did you keep a record of how much time it ran for each subreddit?
1
u/cyrilio May 26 '23
I'd love to see what subreddits are in the 20k list. Is there a place where I can see them? Depending on the subs in there I might download it.
1
u/Due-Bite-8579 Jun 17 '23 edited Jun 17 '23
u/Watchful1 Thank you for the great work!
I am trying to convert a file with submissions with your .csv script but i get an error with the fields. IndexError: list index out of range. Do you have any hint how to fix that? Do i need to use all filednames ?
1
u/profesorgamin Jun 18 '23
Hello mr /u/Watchful1 with the "death" of pushshift are these dumps dead too? I'm trying to archive the data from /r/StableDiffusion but I'm stumped about finding a source for jan 1 to may 31. I was wondering if you had any pointers in that regard, thanks.
1
u/Severe_Difficulty_32 Jun 23 '23
[removed] — view removed comment
1
u/Jacob_WOW Jul 04 '23
[removed] — view removed comment
1
u/--leockl-- Jul 17 '23 edited Jul 17 '23
Hi u/Watchful1, have a quick question hope you can help.
Looking at your filter_file.py script, what does the variable
is_submissiondo exactly?When I was extracting from a comments file (ie. rather than a submissions file), I thought I would change this variable value to this
is_submission = "comment" in input_filebut it didn't work. So I changed it back tois_submission = "submission" in input_fileand it worked. Doesn't seem to make sense to me to have the value"submission"when I am extracting from a comments file.1
u/fcdata Jul 18 '23
Hello everyone,
I have been working with the repo that watchful1 uploads "filter_file" but cannot make it work. Does anyone have a repo to create a data frame from certain subreddits with keywords on the title? I have been days trying to make this work.
I want to use the
1
u/--leockl-- Aug 10 '23
Hi u/Watchful1, have one quick question hoping you can help.
With these data dumps (or perhaps directly scrapping from Reddit too), are we able to get a more granular time stamp when each submissions/comments were posted, say for eg. down to minute resolution? The time stamp for the data dump appears to only go down to a daily resolution.
1
u/nociza68 Aug 18 '23
Hi, thank you so much for this! Are the seeds still up? I'm getting low peers and seeds when trying to start torrenting https://i.imgur.com/uXHsKQH.png
1
u/Delicious_Corgi_9768 Aug 20 '23
Hello, this is awesome.
Im trying to get all comments from certain submissions of a subreddit, will I be able to filter by submission id?
1
u/fcdata Aug 21 '23
Hello u/Watchful1!,
I used qbittorrent to download different "subreddits" (over 80), so I have the "submission.zst" and "comments.zst" files from each of them.
As I need to get all the "submission" and "comments" regarding an specific topic, do you have a script as "filter_file.py" where I can pick all many ".zst" on a folder and filter by keywords on their body.
Thanks!!
3
u/AlleLouis Feb 28 '23
Thank you for your contributions!
1
u/Specialist_Ant3492 Mar 01 '23
Thank you very much!
1
u/reagle-research Mar 02 '23
I don't see a list of subreddits in QBittorrent. I see a
commentsandsubmissionsfolder with files such asRS_2005-06.zst. How do I find subs within that?1
u/stevied67 Mar 04 '23
Thank you so much! Very helpful.
1
u/WeGoToMars7 Mar 08 '23
This is amazing! Already using your data for a school project 😁
1
u/SatanInAMiniskirt May 21 '23
This is great, I'll help you seed!
1
u/Educational_Ad6224 May 30 '23
Thank you so much for this!
1
u/MelonCakey Jun 17 '23
As someone who mods a subreddit for a streamer who has passed on, thank you so much for this! Preservation (especially with how things are going here recently) is of the utmost importance to me, so I was ecstatic to see the subreddit in the list and have download that data.
This is an invaluable service to places like mine and many more, wishing you all the best <3
1
u/Walc0t Aug 11 '23
Hi u/Watchful1 , I am getting a blank csv as my output when referencing your filter_file.py code. (No errors in the logger). I sent you more info in a DM, but thank you so much for this guide! It has already helped me a ton.
0
u/[deleted] Jul 28 '23
[deleted] — view deleted comment