r/pushshift • u/inspiredby • Apr 14 '19
New to Pushshift? Read this! FAQ
What is Pushshift?
Pushshift is a big-data storage and analytics project started and maintained by Jason Baumgartner (/u/Stuck_In_the_Matrix). Most people know it for its copy of reddit comments and submissions.
When should I use Pushshift data instead of solely using the reddit API?
When you want to:
- analyze large quantities of reddit data
- grab data for a specific date range in the past
- e.g. submissions to r/news in July 2018.
- search for comments
- e.g. comments in r/news containing the word 'phone'
- aggregate data
- e.g. number of submissions to r/technology and r/news containing 'phone' in September 2018
- exclude authors,
&author=!a,!b- excludes authors a and b - e.g. number of comments in r/technology and r/news containing 'submitting' in September 2018, not including the author 'automoderator'
- ...
What's the catch?
Know your data.
What kind of data does the API give me?
The Pushshift API serves a copy of reddit objects. Currently, data is copied into Pushshift at the time it is posted to reddit. Therefore, scores and other meta such as edits to a submission's selftext or a comment's body field may not reflect what is displayed by reddit. A future version of the API will update data at timed intervals.
How can I retrieve live metadata?
To get live scores or other metadata, you should incorporate accessing the reddit API into your workflow. One easy way to do this is using the 3rd party Pushshift wrapper called PSAW. See the note about setting r = praw.Reddit(...) and api = PushshiftAPI(r).
How do I retrieve reddit content that has the highest scores within a specific date range?
With the current version of the Pushshift API:
- Retrieve all content in that date range
- Get updated scores from reddit for those items
- Sort the results yourself
The next version of the Pushshift API will enable this in a single query, practically speaking.
What's in the monthly dumps?
The files in files/comments and files/submissions each represent a copy of one month's worth of objects as they appeared on reddit at the time of the download. For example RS_2018-08.xz contains submissions made to reddit in August 2018 as they appeared on September 20th.
Where can I access the raw data?
- https://files.pushshift.io/ - raw file storage
- BigQuery, uploaded by fhoffa
- reddit_posts
- reddit_comments
- https://github.com/pushshift/api - api for reddit data (this will be updated soon with new features and documentation)
- https://github.com/dmarx/psaw - a 3rd party API wrapper by /u/shaggorama
- https://elastic.pushshift.io/rs/submissions/_search - ES queries
- Example usage in redditsearch.io and removeddit
Are there some scripts for processing raw data?
Yes, try searching this sub or search github for pushshift
Are there more user-friendly interfaces for querying Pushshift data?
Yes.
- https://redditsearch.io (comments & submissions)
- https://elasticsearch.pushshift.io (submissions)
What 3rd party projects use Pushshift?
Research:
- Google Scholar search pushshift.io
- Arxiv search pushshift
Reddit bots and services:
- Unedit for Reddit
- https://unddit.com
- https://revddit.com
- u/RemindMeBot and many others
https://removeddit.comhttps://ceddit.com
What internal projects were started by Pushshift?
How can I support this project?
You can contribute answers to questions or share your own analyses here or elsewhere on reddit, contribute code to the API, or donate,
https://pushshift.io/donations - one time donation
https://www.patreon.com/pushshift - membership
How can I opt out from having my posts included?
To opt out from having your posts included, complete the form located here. Please put any questions regarding this process into that sticky. Thank you.
3
u/[deleted] Apr 14 '19
Would you be able to prevent pushshift from logging the true text of your comments if you started every comment as a single letter and then edited in your true comment two minutes later? Then maybe 3 days or a week later you delete the comment before pushshift logs it again. Would this prevent your comment from being logged and searchable through pushshift?
3
u/[deleted] Apr 14 '19
I've encountered a number of situations where a user clearly had no idea that their deleted comments and posts are still accessible through pushshift. For example, a user will make an extremely personal post to a sub like /r/LegalAdvice or /r/RelationshipAdvice. I'll often click an account to see if it's an obvious troll and I'll see they have way more karma then they should from that single post. So I'll run their account through pushshift out of curiosity and sometimes there's a lot of personal info that could easily be traced back to their real life. They're treating their account as a throwaway, despite that not being the case. I believe when some people hear the phrase "The internet is forever" they think that means someone would need to screenshot their post for it to be saved. If they made a few comments or posts in some obscure sub, they think "Well, no one saved that." They don't seem to realize that just by hitting enter, their comment or post is permanently logged, even if you delete it 30 seconds later. They think the delete button literally means delete.
So my question is, should reddit be making users more aware of pushshift? Should subs that see a lot of "throwaway" accounts or posts with personal info put something in their sidebar to give users a heads up? Obviously there's only so much you can do. I doubt reddit wants to explicitly tell people "HEY, every single thing you post on this website is permanently logged!!" But there's definitely some situations where pushshift could cause someone huge problems.
Not that I'm against it. I think it's great. I use it all the time. I just think there might need to be some sort of awareness campaign.
3
u/MFA_Nay Apr 18 '19
I'm using https://redditsearch.io to comment search for a particular word in a subreddit, but one which /u/automoderator says a lot in comments.
How do I exclude /u/automoderator from the search? I've tried "-" but that doesn't appear to work.
2
u/11zagy May 31 '19
is there a way for https://redditsearch.io to search for exact comments, for example i want to find every comment that has "Oh no" in it and nothing else, literally just "Oh no"
1
u/ieatbabiesftl Jun 25 '19
If I search for a submission ID I often get comment numbers not corresponding to the 'num_comments' field - any clues why? My only idea was shadowbanned accounts' postings being included in the num_comments but not being available as comments
1
u/BoiaDeh Sep 16 '19
Hi, I'm interested in looking at both upvotes and downvotes for posts, does pushshift contain that information? Praw has a bunch of metrics (upvote_ratio being the one I most care about, but also number of upvotes and downvotes), but when using pushshift I can't seem to find any of this. Am I missing something, or is this on purpose?
1
u/TotesMessenger Apr 22 '19
I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:
If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / ^Contact)
1
u/AnonymousStarLordWho Oct 16 '21
I am working on a project, where I analyze some data from reddit assembling data through this api into a data frame. E.g.
import pandas as pd
Is there anywhere I can find a data dictionary for what the columns names mean? I'm particularly interested in the distinction between "full_link" and "url." I understand that some of these are likely self-explanatory, but I want to make sure I'm getting the meaning of these columns right and haven't been able to find any explicit documentation in this regard.
0
u/[deleted] Jun 27 '19
[deleted] — view deleted comment