Trust and Safety Engineering
Last modified on April 08, 2022

New research
Old App Vulnerabilities
Simple Config Errors
Patching
People use the same passwords.

1. Scale
   Need a large enough system to study
   
2. Non-diverse studies and solutions
   West overrepresented, English language overrepresented
   
3. Measurememnt and definition challenges
   “How to define / measure abuse?” turns out to be a tough question
   

In almost every platform, the biggest form of abuse is spam. Is it the worst thing? No– just the largest thing.

How much am I catching? how much am I missing?

Badness leads to poor experience, lowers user trust, and in some cases leads to user harm.

Internal Measurement: user reports
External: Crawling and scraping, crowdsourcing, metrics, UX studies

“Two basic principles of management, and regulation, and life, are:
You get what you measure.
The thing that you measure will get gamed.” - Matt Levine, Bloomberg

Metric perversity: optimizing a metric ends up being “perverse,” i.e. optimizing for time spent works really well– but that doesn’t mean it’s good for the customer.

Lots of difficult tradeoffs: Free vs. reliable. Neutral moderation vs. trustworthy information. surveillance-resistant vs. No real-world harm. etc.

Lagos-based researcher who identified this scam. Had her personal/business Instagram hacked by scam Binomo account.

Has 200k+ followers, big social media following. Used her face, her brand to get people to invest in scams. Even used her husbands’ picture, children’s picture…yeesh. Internet/scam literacy may be a problem in Nigeria- people easily trust whatever they see. Lots of trust in Nelly = good, scammed = bad.

• direct line of contact when you get hacked
  
• strong passwords
  

There are markets for:

• stolen data
  
• easy-to-use malware
  
• phishing kits
  
• hackers for hire
  
• botnets
  

Spam is the primary vehicle for cybercrime - “the original trust and safety problem” - a lot of the structures to deal with other kinds of abuse were initially created for spam.

Almost always financially motivated.

“Spanish prisoner fraud” - send me money to bail me out in jail, and I’ll give you some of my fortune

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

In other countries, however, law enforcement have a lot, lot more rights to search and seizure (e.g. India.)

Stored Communications Act (SCA): US laws allow a lot more access to metadata than actual content of the messages.

Passed in response to Cold War stuff– FBI wanted to track down Russian spies in the US. BUT we didn’t want those same permissions used against US citizens.

FISA Amendments Act (FAA): Allows govt to target more groups of people, in order to target more dynamic targets, e.g. Al Qaeda.

Domestic abusers are a very difficult category of attack to stop: they often have unlimited access to devices, passwords, emails, victim’s social network, knowledge of all password reset questions, financial/physical leverage.

“Legitimate” spyware - Find My, Kidguard, etc. Can be abused

AirTags – pretty easily abused to track people.

Wrote article refuting Gofundme’s statement that they would ban January 6th-type people. Not a lot of oversight / moderation @ payment systems.

After writing the story, she received far-right Twitter comments, lots of vitriol.

Right-wing media pundits picked it up => death threats, threats to family, etc.

Takeaways:

• speed, anonymity, etc. of social networks enables this type of harassment
  
• small amount of ring-leaders
  

hate speech, domestic violence, etc.

• sealioning - “just asking questions” - antivax people do this
  
• dogpiling - large mob of people
  
• swatting
  

Against Zoe Quinn and other female game designers - coordinated dogpiling attack, doxxing, account hacking, NCII, etc.

Journalists

Taylor Dumpson - target of alt-right harassment, “trollstormed,” 8chan, etc. Got legal action, public apologies, etc.

Dissidents – in authoritarian countries, like Russia.

clear policies for “awful but lawful” content:
“do not threaten, harass, or bully”
“do not repeatedly contact someone in a manner that is […]”

US doesn’t have many laws against hate speech – most is protected under the First Amendment, except when there’s a threat of violence

Reporting flows - the more specificity you put, the more exactly people expect their experience to fit into one of the given categories. Balance that with – we want enough data to use something specific – e.g. pass it through a hate speech classifier.

Part of a back-and-forth between abuse reporters and platform – evolves over time

Public block can egg the person on even more

User karma – how many people have blocked? how often contacting strangers?
Relationship between sender and receiver

Instagram “rethink” feature

Hate vs. hateful
Hateful vs. cyberharassment
Dangerous vs. hate
Undesirable vs. hate
Extremism vs. hateful

US: no legal definition of hate speech. Most hate speech is protected under the First Amendment

Other countries: detailed policies that limit hate speech.

International treaties: shows you things that absolutely should be protected, things that MAY be restricted, and things that MUST be restricted

Cycle between message boards, content hosting sites, funding sites, etc…

The eight stages of genocide:

• classification
  
• symbolization
  
• dehumanization
  
• organization
  
• polarization
  
• prepraration
  
• extermination
  
• denial
  

Example: Myanmar - Rohingya genocide. Early on, Rohingya Muslims were represented. Military coup => dominated by Buddhist majority.

Alliance between extremist religious groups and the military to dehumanize Rohingya.

Growth of mobile phones, Facebook was big in Myanmar - Rohingya hate speech proliferated there.
Some of the hate speech is metaphorical

Advertising, recommendation engines, public impersonal, public personal, private groups, private personal, group message

Amplification <===> Free expression

“OK” symbol was co-oped by 4chan trolls, said it was a white power symbol

algorithms are biased against black people
AI has limited “true” understanding - difficult to distinguish post ABOUT hate speech, rather than hate speech itself

• how can researchers stop the spread of something they cannot see?
  
• what can / should be limitations on research materials?
  
• what is the role of companies in this dynamic?
  

misinformation: unintentionally inaccurate information

disinformation:

1. deliberately false or misleading info
   
2. info with the intent to deceive
   

political propaganda: information used to promote a particular political cause or point of view (could be misleading, but could be true)

“fake news”: false news stories (not used by researchers anymore)

Nation States: to control their citizens, maintain power
Terrorist orgs: control citizens, influence outsiders
Domestic:
Mercenaries: achieve clients’ goals
Spammers: economic

• China
  
• Russia
  
• Iran
  
• US
  
• Saudi Arabia
  

• distrust in authority
  
  • media is f***ed up: INCLUDING mainstream (e.g. NYT)
    
  • media => Iraq War
    
• amateur participation + virality, automation
  
  • zeitgeist no longer decided by ~20 people in control of the media
    
  • upside: Black Lives Matter, MeToo, etc. can exist
    
  • downside: misinformation
    
• cross-platform spread
  

printing press, radio, television: => virality of ideas by organizations
social media: => virality of ideas by individual people.

Steve Bannon: ideologue. Has a huge following, even with no editorial standards.

all of us are open to propaganda

Real is Fake: convince people that a killing video is “fake news”
Fake is Real: convince people that a human body parts eating video is real

KGB operation - took advantage of real passions and hatreds - used anti-Semitism in Germany, to drive hate.

Twitter trending - easy to manipulate

(policy for mis/disinfo is very difficult)

• increase transparency
  
• share research about misinfo
  
• “immune system”: fact-checking, media literacy, funding
  
• inform legislation
  

reduce velocity/virality through “friction”
reduce financial incentives for disinfo
label potential mis/disinfo
treat influencers differently
machine learning classifiers

New class of issues from putting pseudo-strangers together for some kind of interaction.
Creates a number of new risks- can be higher stake because of the physical interactions.
Many safety issues involve breaking laws in the real world- use Section 230
Safety depends on design choices of platforms

There is a real chance to make things better- e.g. the taxi business kinda sucks atm. Uber could be better.

violence between drivers/guests
Brazil: drivers have a bunch of cash (can be robbed)

People can be located using trilateration (Grindr)
People can be monitored, persecuted on Grindr in repressive countries

set rules for interactions above the law (not just illegal stuff)
enforce identity (background checks, validation, etc.)
2-way rating systems (drivers & passengers, hosts & guests, etc.)
investigation teams
work with local authorities / LE
publish measurement, transparency reports
educate users and employees on risks, mitigations

“scaffold” trust & safety as you’re scaling up -

ideation: think about the space as it exists now, before you disrupt it.
pre-seed: include risks/downsides in slide deck
seed: safety in design
series a: hire someone for t&s
hypergrowth: hire t&s team

Alibaba, Tencent, ByteDance etc. all in top-10 revenue of social media companies.

these companies might have to deal w/:
political moderation requirements, less experience w/ safety issues, Western privacy laws, data stored in China, decide not to follow ECPA, issues with transparency, etc.