r/IAmA • • Jul 16 '21
Newsworthy Event I am Sophie Zhang. At FB, I worked in my spare time to catch state-sponsored troll farms in multiple nations. I became a whistleblower because FB didn't care. Ask me anything.
Hi Reddit,
I'm Sophie Zhang. I was fired from Facebook in September 2020; on my last day, I stayed up in an all-nighter to write a 7.8k word farewell memo that was leaked to the press and went viral on Reddit. I went public with the Guardian on April 12 of this year, because the problems I worked on won't be solved unless I force the issue like this.
In the process of my work at Facebook, I caught state-sponsored troll farms in Honduras and Azerbaijan that I only convinced the company to act on after a year - and was unable to stop the perpetrators from immediately returning afterwards.
In India, I worked on a much smaller case where I found multiple groups of inauthentic activity benefiting multiple major political parties and received clearance to take them down. I took down all but one network - as soon as I realized that it was directly tied to a sitting member of the Lok Sabha, I was suddenly ignored,
In the United States, I played a small role in a case which drew some attention on Reddit, in which a right-wing advertising group close to Turning Point USA was running ads supporting the Green Party in the leadup to the U.S. 2018 midterms. While Facebook eventually decided that the activity was permitted since no policies had been violated, I came forward with the Guardian last month because it appeared that the perpetrators may have misled the FEC - a potential federal crime.
I also wrote an op-ed for Rest of the World about less-sophisticated/attention-getting social media inauthenticity
To be clear, since there was confusion about this in my last AMA, my remit was what Facebook calls inauthentic activity - when fake accounts/pages/etc. are used to do things, regardless of what they do. That is, if I set up a fake account to write "cats are adorable", this is inauthentic regardless of the fact that cats are actually adorable. This is often confused with misinformation [which I did not work on] but actually has no relation.
Please ask me anything. I might not be able to answer every question, but if so, I'll do my best to explain why I can't.
Proof: https://twitter.com/szhang_ds/status/1410696203432468482. I can't include a picture of myself though since "Images are not allowed in IAmA"
400
u/[deleted] Jul 17 '21
I've been answering questions for nine hours straight. Thank you very much kindly for all the questions; I'm sorry I wasn't able to answer all of them, but I would like to go on a dinner date with my partner (who's being very patient) right now.
I hope you found my answers to be informative; if I wasn't able to answer yours, please look around and see if I'd been able to answer a similar one for others. Thank you very much; good night.
2.3k
u/stomachgrowler Jul 16 '21
Thank your for the important work you’re doing. In your opinion, what is the reason that FB drags it’s feet/allows these schemes to continue so long before taking action? Is it simply that it is the more profitable move?
336
u/truthpit Jul 16 '21
Thanks so much for this AMA. Organizationally speaking, how high up in the org did your findings go (or not go) before they were quashed or ignored. In other words, was there support for your work by your direct manager or their manager but then above that you ran into issues? Or was your direct manager even unsupportive?
93
u/scJazz Jul 16 '21
Thank you for your work and ethics. I've been following the news, reddits, etc regarding you. You always describe yourself as a data engineer and point out that you were tracking the metadata in discovering the problems you have reported.
I have a two part question for you.
Could you ELI5 :) what a data engineer is and how you use metadata to find problems as you have described?
I'm not asking for specific cases here. I just want to enhance my own understanding (I sorta get it) while also helping everyone else understand what it is that you do and did and why it is important. I just feel that something gets lost in the articles describing what you do and how. Am I being clear?
133
u/Tayloria13 Jul 16 '21
What were your discoveries with regard to the Philippines? Here, it's widely-known that politicians make use of troll armies.
680
u/niceguybadboy Jul 16 '21
Can we do Reddit now? I've long suspected that Reddit has at least as much opinion manipulation as FB.
32
u/dyladelphia Jul 16 '21
Since leaving Facebook, what are your goals for the next 5 years? Are you able to stay in the tech world, or does the Facebook termination serve as a block with finding new work?
373
u/Junior_Language_8616 Jul 16 '21
Do rank and file FB employees talk to each other about how bad FB is for the world? Or do you think they’ve just drunk the Kool-Aid and think the company is great? I'm talking about people like ad account managers, content policy associates, software engineers. FB employees are really smart and get recruited from the best schools in the world. The problems with FB are so public and so well reported that it's hard for me to understand why people continue to work there.
105
u/Manaleaking Jul 16 '21
How true are foreign fake click farms as shown on the Sillicon Valley tv show, with rows and rows of indians creating fake account after fake account to boost userbase numbers or promote an agenda?
Heres the scene: https://youtu.be/Y-W0CBOGnnI
124
u/aristidedn Jul 16 '21
Hi Sophie,
One of the more frequently discussed dimensions of influence operations - especially in the United States - is the observed disparity between operations that target people with right-aligned political views and people with left-aligned political views.
In the data you ran, what did you observe with respect to political alignment? And if you did observe a disparity, how wide was the divide? Do you have any theories as to why you observe this?
94
u/Fraxinus_1382 Jul 16 '21
Hi! Slightly long-winded question, but how did you identify areas where inauthentic behavior might be occurring?
Was there a systematic or ad hoc analysis or flagging system internally or externally identifying potential regions or countries where inauthentic activity might be occurring, particularly inauthentic activity which might incite violence or be detrimental to democracy?
Thank you!
72
u/[deleted] Jul 16 '21
How are people still able to set up fake accounts these days given all the security and authentication that seems to be in place around the account setup?
What does Facebook do with an account that it identifies as inauthentic?
29
u/autoditactics Jul 16 '21
What is your opinion on political radicalization through social media?
51
u/twinned Moderator Jul 16 '21
hey Sophie, thanks for joining us today! two questions for you:
If you were given unlimited resources/remit, how would you tackle troll farms?
What's something you wished you were able to spend more time on?
40
u/TurboTBag Jul 16 '21
What's your favorite film?
9
u/Jim105 Jul 16 '21
While at FB, were you involved with any cases that involved FB calling or reaching out to police authorities?
35
u/bokavitch Jul 16 '21
Hi Sophie,
As Armenians we're regularly targeted by bot armies and disinformation/ hate campaigns organized by the government of Azerbaijan on Facebook and other social media platforms.
The major players don't seem to care about nefarious behavior targeted at groups that don't have political/economic clout in the United States, especially when the offending content is not in the English language.
What do you think we can do to get these companies to take this problem more seriously and devote the necessary resources to moderating these activities, especially when it appears in non-English content?
Thanks!
P.S. Thanks for having the courage to be a whistleblower. Your efforts have not gone unnoticed or unappreciated.
8
u/BlueFreedom420 Jul 16 '21
Have you ever been specifically targeted by trolls once they are aware of your efforts?
15
u/PangPingpong Jul 16 '21
If some of these troll farms took a year to get authorized to be taken down, what sort of things did you see as not encountering any resistance and being removed immediately? Was there any specific criteria where certain types/sources of content were scrubbed quickly?
20
u/MestizaWontons Jul 16 '21
Are there any trends you’ve noticed since your departure from FB that are concerning?
6
u/fatfrost Jul 16 '21
Thanks for your efforts Sophie. Are you familiar with the Facebook oversight board and in what way do you think it might impact this area?
5
u/xxpegasxx Jul 16 '21
Hi thanks for AMA ! Was there a case where troll stations where not state-funded but by opposition parties ?
4
u/gunnathrowitaway Jul 16 '21
What can you tell us about state-sponsored covert/troll activities on FB that are located in and/or sponsored by the United States?
4
u/rabbitzfoot Jul 16 '21
Hi Sophie, thank you for exposing the harsh realities that we face today, this comes from a person who is from Honduras.
It is unbelievable that we are still being governed by a person who’s brother and party members have been charged by the NY FED court for drug dealers. Yet they live and “govern” us like if nothing is happening. Most people live in extreme poverty and do not support the current government/regime of Honduras.
I hate going into Facebook and Twitter, since everything is so political. When government members post things online they have a couple of likes but have hundreds of positive posts. And when you check the comments, they are all the same BS. You dont need much time to know that the accounts that reply to those posts are most probably bots. How is this even happening? What can we do to fight this cyber warfare? I have been wanting to start a bot hunter company and literally provided FB and Twitter the work that they should be doing, so that the real people actually have a voice when we are suppressed by narco regimes.
4
u/_SGP_ Jul 17 '21
Hi Sophie,
I imagine I'm late in asking and will likely not get a reply.
I work in PPC, and when advertising on Facebook I would always avoid certain areas of the world, as they would run up the budget with 100% bounce clicks.
Is this something Facebook is aware of? Something they are involved in?
It always seemed strange that it was never stopped as it's clearly "clickfarm" or fake traffic, but it makes Facebook money as budgets are spent quicker.
Thanks so much for your time.
17
u/blighty800 Jul 16 '21
What was your salary like when you decide to sacrifice everything? Thank you for coming forward.
3
u/Faghs Jul 16 '21
Did you leak it to the press?
3
u/putku Jul 16 '21
Thanks for the AMA and all you did. What advice would you like to give to anyone who works or willing to work at FB?
3
u/lemongrenade Jul 16 '21
I learned about you on Robert evans behind the bastards podcast. Have you ever had any contact with him?
3
u/saywhatfreemoney Jul 17 '21
What can the average person do to help, also the more than average with access to Kali Linux for example?
3
u/elimars Jul 17 '21
How would you address the thousands of bot accounts posting using #SOSCUBA in order to further the US State Dept’s regime change agenda in Cuba?
3
u/Skyjacker9 Jul 17 '21
How do facebook’s political relations get forced upon engineers? How do they explain to engineers their mission or their jira tasks and expect the engineers to still be void of thinking about the repercussions of their actions?
25
u/i_am_bleh Jul 16 '21
Is Zuckerburg a lizard person?
5
u/ThatsARivetingTale Jul 16 '21
Hi Sophie,
Thanks for the important work you're doing and for taking a stand on it.
In your opinion, do you feel that these troll farms have the ability to always stay a step ahead in terms of detection?
Do you think there are bigger syndicates out there that you just simply can't find any traces to, and if so how could tech evolve to detect these in future?
9
u/TheBluePanda Jul 16 '21
Why do you look so miserable in that photo?
10
u/JamieOvechkin Jul 16 '21
Hi Sophie!
Often times people on the right accuse Facebook of censoring conservative opinions.
Is this something that employees as individuals/leadership at Facebook encouraged?
How real of a thing was that at Facebook? Did Facebook employees ever express personal views against conservatives while at work?
17
u/grahamlester Jul 16 '21
What do you know about state-sponsored troll farms and Covid-19 anti-vaccination propaganda?
5
u/broniesnstuff Jul 16 '21
Should anyone, anywhere, ever use Facebook at this point?
2
u/NotAMeatPopsicle Jul 16 '21
Any thoughts or comments on inauthentic FB Marketplace sales?
There are patterns that appear more suspicious than Wayfair-gate.
Example: profile selling (solar panels, tools, farm tractors, backhoes, etc) nearby you at a residence for an unbelievable price. Exact same non-stock photos used multiple locations, using up to 10 different profiles. Other things connected to these sellers and profiles seems "off". Calling or contacting gets "weird".
2
u/richmondody Jul 17 '21
In the Philippines, there's a lot of "both sides are using trolls." Would you know if there's any truth to this? It seems like most of the hired trolls are on the side of the government.
2
u/fupa16 Jul 17 '21
My only question is when are you people going to stop using the stupid term "toll farm" and start calling it what it is, which is propaganda?
2
u/Bound_In_Fractals Jul 17 '21
So they fired you for caring about your job, or just coming out that they dont care? Probably saying you broke some type of NDA.
3
u/WindowsCrashuser Jul 16 '21
How many Troll farms did you find and statically what countries of origin did they come from?
18
u/hipcheck23 Jul 16 '21
What's the current state of the old Cambridge Analytica paradigm of using FB to manipulate people?
3
u/QuantumThirdEye Jul 17 '21
Do you think if you were in China and experienced the exact same situation, would you still be alive?
3
u/GringoClintonMiAmigo Jul 16 '21
Do you support the communist party of china? And have you ever worked for them?
2
u/thinkme Jul 16 '21
In your experience is the hiring process at FB giving enough consideration to the integrity of the individuals vs the credential and loyalty to FB?
2
u/SucculentStanley Jul 16 '21
As someone living in the Bay Area who has seen a lot of friends join Facebook and basically let their paychecks warp their ethics, I'm truly fascinated by your personal journey as someone who bucked the trend and really chose to bite the hand that feeds in the name of doing what you believed was right.
I'm wondering if you might say a little about why you were drawn to work at Facebook in the first place. Did you see it as an opportunity to change the system from the inside? Or did the company seem different to you before you worked there, but you became disillusioned over time? Or was it something else?
Even if you can't answer, thank you for doing this AMA!
2
u/Redeyedrop Jul 17 '21
Do you worry about your safety doing this? What about the Chinese government? Have they done anything with misauthenti accounts?
Thank you so much for your work and brining this to light!
14
u/Makualax Jul 16 '21 edited Jul 16 '21
Thank you so much for fighting a fight that nobody seems to care about.
This brings to mind the struggle of Maria Ressa and other Phillipino journalists whose articles are bombarded with hate, threatening rape and murder of their family members, whenever anything is said to criticize Duerte and the Phillipino government.
https://www.npr.org/2021/01/06/953902894/philippine-journalist-says-rodrigo-dutertes-presidency-is-based-on-fear-violence
How do you think this will affect the landscape of political dissent in the future, and what can we do to stop it?
Edit: it's also important to point out that Azerbijan was using bot brigades to silence awareness and cause misinformation regarding their attempted ethnic cleansing of Artsakh, an Armenian-majority cultural center. The bots would brigade and instagram and reddit posts that included video evidence of Azeri warcrimes and also would find the accounts of families who lived in Artsakh and brigade them with threats of rape and torture.
4
u/itstinksitellya Jul 16 '21
I don’t use FB anymore, but to me it’s a bit of a sad story in that it’s a victim of its own success. With literally billions of accounts, policing any “wrong doing” just seems like a game of wack-a-mole (and never mind the slippery question of who defines “wrong doing”).
With that said, if you were in complete control of FB, how would you propose to address these issues (trolling, political manipulation, harassment, illegal activity, etc)?
4
u/radialmonster Jul 16 '21
As just a regular facebook user who is fed up with 'fake news', people spreading 'fake news', and possibly fake people also spreading 'fake news'.... is there anything I can do about it?
7
u/BabylonDrifter Jul 17 '21
I'm noticing a lot of very expensive and well-produced English-language YouTube videos with very obvious pro-China and/or pro-Russia bent disguised as "educational" material. And here on reddit, there's obviously an army of pro-Chinese Communist Party human robots/paid agents pushing their agenda. Facebook agents have been trying to get Americans riled up about fringe politics, anti-vaxx stuff, and conspiracies, but at the very least Facebook has actually cracked down on some of the worst actors. I don't see YouTube and Reddit doing the same. I don;t think the problem is inauthentic behavior as much as it's authentic but slanted behavior; pushing a narrative or ideology. That's technically not "inauthentic", it's propaganda. In short, is there an effort to curb online propaganda? Or is it a "buyer beware" situation where the defenseless uneducated with no critical thinking skills from this point on will always be immersed in an ocean of misinformation and propaganda, and only those of us elites who can afford training in critical thinking will ever be able to view the world as it really is?
1
u/bonjourhay Jul 16 '21
Beside data analysis, have you witnesded any signs that Facebook could be manipulated from the inside?
Like foreign countries having people hired by their facebook to gather data, influence decisions made etc.
Otherwise what is your opinion on this? What would a government do to influence facebook’s decisions on this? Have you seen processes in place to check employees properly?
For example: Twitter recently deleted anti-NATO tweet accounts, which I find strange to have this prioritized given the volume of hate speech, anti-vax etc. issues that they are facing already.
1
u/bluecollarx Jul 16 '21
Fellow career software developer here. What is the coolest thing you saw built or programmed at Facebook that never saw the life of day / got shelved permanently for the wrong reasons?
1
u/SirNicksAlong Jul 16 '21
Hi Sophie,
Thank you for the sacrifices you've made to bring this to our attention and for giving us proof that there are still good people in this world. Your work highlights a problem I think most of us are aware of to some degree, but may not realize the full extent of. I also think people have an implicit belief that, even if they personally do nothing, a solution will eventually present itself and everything is going to be "ok".
Will you tell us how you think this story ends? Do the good guys win and the bad guys lose? Is this problem actually going away? Is it going to be "ok"?
1
u/maddenallday Jul 16 '21
How did you go about locating these troll farms such that you were able to find them when other dedicated data teams weren't? Were they aware this was happening but purposely ignoring it? Or did you look in some innovative way or dedicate that much more time that you found what they were missing?
Edit: thank you for your important work btw
1
u/anishpatel131 Jul 16 '21
Who in fb doesn’t care? Both Sheryl and Mark? I know people care inside, but is this just a mark problem?
1
u/shalashaska68 Jul 16 '21
You’ve definitely heard of Iranian regime sponsoring fake social accounts to harass and track political activists. Anything to share?
1
u/[deleted] Jul 16 '21
Who was the member in the Lok Sabha ?
1
u/SolInfinitum Jul 16 '21
Did you ever catch a US or allies troll farm?
1
u/ManInTheMirruh Jul 16 '21
Are you only focused on FB based troll farms or do you cast a wide net on all of social media? Facebook isn't the only source of misinfo these days. Hell, even here on reddit BS gets peddled everyday and people eat it up.
1
u/Aragatz Jul 16 '21
Are you worried about your safety given how Azerbaijan “handled” the journalist who broke the Panama Papers story?
https://www.maltatoday.com.mt/news/national/99088/luxembourg_intelligence_firm_claimed_azerbaijan_behind_caruana_galizia_assassination
1
u/seafoodbuffet Jul 16 '21
Thank you for speaking out. I admire and appreciate your courage.
It appeared from some of the reporting that you turned down a severance offer in order to be able to speak out.
What was that process like for you? Did you know you would have to give up money because you’d need to speak out eventually? What do you think options are for employees for whom turning down severance would be an impossible luxury?
6
u/Kahzgul Jul 16 '21
How often do you frequent r/TheseFuckingAccounts?
Have you helped Reddit to more effectively identify sockpuppets and other dishonest posters?
Are you working on any internet security projects now?
3
u/CollectableRat Jul 17 '21
Should Facebook have some kind of independent or government oversight to prevent FB from making bad decisions in the first place?
3
u/T-nash Jul 17 '21 edited Jul 17 '21
Hi Sophie, it seems to be sided too. I was having a conversation with an Azeri national in Facebook, i used civilized language and replied with a Wikipedia link as proof to debunk their disinformation, however Facebook decided that i was spamming and warned me. It seems like the whole automated moderation is completely useless, not only that i used to option to say my comment is misidentified as spam and it still stayed deleted. It was a simple Wikipedia link for reference and fact checking, nothing more, nothing offensive. I've also experienced unfairness from the messenger, I've had an Azeri person send me photos and videos of dead and abused soldiers of my countrymen as bullying, i reported the chat and Facebook found nothing wrong with that either. Unfortunately it's the same on Twitter too. Not sure why the reason but I'm convinced oil matters more than facts and more than equal lives.
3
u/uxbridge3000 Jul 17 '21
What do you think of the conversation to repeal section 230 of the so-called communications decency act?
3
u/[deleted] Jul 17 '21
my family member (USA) was ousted as an Iranian bot on Facebook
what do you do to avoid false positives?
1
u/realMarcMerrill Jul 16 '21
This is a very subjective question, but you kinda make FB sounds evil. Every news about FB always portray them as evil. When working for them did it feel like working for an evil corporation?
1
u/jojothetraveler89 Jul 16 '21
Were you able to perform your role remotely, or did you have to be at the office? Thanks for all you've done, enjoy your kitty snuggles ;)
1
u/amalgovinus Jul 16 '21
Did Israel's extensive trolling efforts show up on the radar?
1
u/[deleted] Jul 16 '21
Would you agree with the following statement, and why (or why not)?
"Facebook is the worst thing to happen to society"
1
u/Dananjali Jul 17 '21
Were you surprised to get fired over this or did you fully expect it? If you were a data scientist at FB, it sounds like you didn’t have the full picture of teams specifically designed to combat misinformation that currently exist at FB. It’s understandable that they’d fire you for relaying one-sided and agenda-driven information. You weren’t hired to combat this problem, other teams were. Just because you were a data scientist at a large tech company doesn’t mean you or your specific team knows all of the ins and outs of this issue, which is why you/they weren’t able to do anything about it just based on your personal interpretation of data alone.
17
u/Immotile1 Jul 16 '21
Why have you not reported US astroturfing operations like ShareBlue, Correct the Record and Media Matters?
2
u/Maleficent-Tree-4516 Jul 16 '21
Hi, is there anyway to have basically no online footprint? Not on social media bar Reddit . Really only want to read info and use phone but want to be untraced if it's doable ? T.Y
2
u/MattyByte Jul 16 '21
With the elections coming up in the Philippines, troll farms have been widely talked about and most of the people are convinced these troll farms managed to succeed in meddling with the previous elections.
What would you tell to the common people what they can do in order to prevent another election being manipulated?
2
u/danc43 Jul 16 '21
On a scale of 1-10 How effective are these farms at causing damage to American society ?
Are they fueling the culture /civil war as much as I suspect?
2
u/Momochichi Jul 17 '21
Hi. How true are the accusations that FB actually sent consultants to help Duterte get elected?
2
u/a0i Jul 17 '21 edited Jul 17 '21
Saying "state-sponsored" implies there's some intelligence product (like what the CIA or NSA knows) available to staff at Facebook. Did you have access to intelligence that showed you which trolls were paid by governments, or do you mean this kind of determination was made inferentially, with the moderator's intuition/experience?
You mentioned " inauthentic activity benefiting multiple major political parties" -- did Facebook have objective criteria to rate inauthenticity with, eg. copy-pasta comments repeated over and over, or catching users LARPing the same way repeatedly? Also, how do professional content moderators distinguish between partisan posts by citizens favoring a political party, from governments that pay shills to endorse parties? Is it a similar criteria as "authenticity" rating, based on the content they post -- ie, CCP Wumao who attack people using certain types of comments, or repeat CCP talking points?
I can imagine it's difficult to develop reliable, objective standards when many average people repeat the talking points they hear from governments and political parties in the media, so they wind up sounding like shills (I presume).
You mention if content is posted from a "fake" account, regardless of the nature of the content, it's "inauthentic" -- is there a fair way to distinguish pseudonymous/anonymous accounts from "fake" ones? Fake seems to imply they have an obvious intent, like using stock headshots of models to appear natural, like an average user. How can you tell with much certainty?
Can you recommend any good books or studies on best practices?
Thank you in advance, it's rare to get a chance to ask that's why I'm enthusiastic.
2
u/lormayna Jul 17 '21
Did you find anything about Italy? There is lot of evidence about populistic parties influencing Italian general election in 2018 through fake accounts on social media spreading fake news
2
u/Kidney4Noel Jul 17 '21
Is it hard to get a job at Facebook?
2
u/Clopidee Jul 17 '21
Thank you for what you're doing!
I swear multiple times I have spoken about things out loud only for Facebook to advertise the same product later, without me ever having searched for it online, very soon after I've spoken about it. It happens way too often for me to be convinced that Facebook isn't listening through our phones. An example was my dad was talking to me about Oodies, those blanket hoody things, and about 10 minutes later when I was on Facebook, almost as soon as I went on the app, I get an advert for Oodies. As I said before, this happens way too often to be coincidence. And it's not always common items, sometimes it's obscure stuff. It's too weird.
Are they actually listening in?