At a congressional hearing in 2020, Mark Zuckerberg sang the praises of Facebook’s artificial intelligence content moderation algorithm:
“In terms of fighting against hate speech, we’ve built AI systems. […] Right now, we are able to proactively identify 89% of the hate speech that we take down, before I think it’s even seen by other people” (in Knutson, 2021).
Then, in 2021, Facebook whistle-blower Frances Haugen leaked extensive documents from the company revealing how the platform’s newsfeed algorithm amplified anger-inducing and harmful content (Hagey & Knutson, 2021). The documents revealed numerous other issues, including the platform’s damaging effect on teen mental health, Facebook’s substantial problem with hate speech, and that its content moderation algorithm was unable to distinguish between videos of car crashes and cockfights (Linebaugh, 2021).
Included in these papers was a report that stated, “according to the best estimates I have seen, we only take action against approximately 2% of the hate speech on the platform” (Knutson, 2021).
The statistical difference mentioned here highlights the lack of transparency Facebook (now Meta) brings to the table when issues of concern to the public are raised.
This combination of engagement-based algorithmic ranking and a woefully inadequate ability to detect and remove hate speech has resulted in devastating real-life harms. Harms that include the genocide of the Muslim Rohingya population of Myanmar; a tragedy that now sees Facebook’s parent company, Meta, facing a US$150 billion lawsuit for not doing enough to stop the virulent spread of hate speech that spread across the country via the platform (Inskeep, 2021).
This case raises, yet again, the problems with allowing behemoth social media companies to self-regulate themselves. Haugen’s leaked documents reveal substantial public concerns and detail the extent to which Facebook understood these issues, which they declined to fix (Linebaugh, 2021).
Meta, and companies like it, require third-party regulatory intervention and implementation of transparency measures with enforceable consequences to hold them to account and to minimise real-life harm from online hate speech.
The difficulties in moderating hate speech
Multiple factors make moderating online hate speech difficult for social media platforms.
Problematic content is deemed as such in conjunction with the laws of where the service is provided and by the platform’s own community standards (Roberts, 2021; Arun, 2022).
While hate speech violates the community standards of most platforms, identification and removal are challenging when problematic speech may differ depending on region, language, and cultural nuance.
AI algorithms are not yet capable of delineating such nuance, as seen in the leaked Facebook documents. Due to the sheer amount of content uploaded daily, human moderators must employ a ‘catch-as-catch-can’ approach (Roberts, 2021, pg. 34).
Familiarity with a region’s culture and local language is also critical for moderators. As Roberts (2021) states, “moderators must have linguistic competency in the language of the content […], be steeped in the relevant laws governing the site’s country of origin and have expert knowledge of user guidelines and other incredibly detailed platform-level specifics concerning what is and is not allowed” (pg. 35).
This sentiment was reiterated by Mark Zuckerberg in a US Senate hearing when he said, “Hate speech is very language specific. It’s hard to do it without people who speak the local language,” (in Milko & Ortutay, 2022).
Unfortunately, platforms are highly secretive about the number of content moderators they employ.
Haugen’s leaked papers reveal that, in 2019, content moderation was the most expensive problem for the platform owing to its employment of human moderators (Knutson, 2021). The company subsequently shifted to rely more upon its AI and, in doing so, decreased the number of human moderators while making design choices to make it more difficult for users to report harmful content (Knutson, 2021).
A report compiled by the House of Commons Home Affairs Committee identified multiple ways platforms such as Meta, Google, Microsoft and Twitter are ineffective at moderating hate speech. These included poor and inconsistent adherence to community standards, lax response to removing harmful content flagged by users, and reliance upon technological inventions that are nowhere near robust (House of Commons Home Affairs Committee [HoC HAC], 2017).
It was also noted that platforms employ an unspoken ‘hierarchy of service provision’, prioritising resolution of hate and abuse content promptly for high-profile users or in the event of media exposés (HoC HAC, pg. 14).
The combination of poor hate speech detection, insufficient Burmese-speaking human moderators, and the dismissal of user reporting of inappropriate content are culminating factors that led to devastating violence against Myanmar’s Rohingya minority.
Facebook’s role in the Rohingya genocide
For years, Myanmar military personnel used Facebook to masquerade as fan accounts of celebrities, monks, and high-status national heroes, disseminating hate speech and disinformation about the country’s Rohingya Muslim population (Mozur, 2018).
The proliferation of hate speech and disinformation fostered fear and mistrust between the Rakhine and Rohingya people, deteriorating their already poor relationship (United Nations Human Rights Council [UNHRC], 2018). The extent of this online harm was exacerbated by Myanmar’s reliance on Facebook as “arguably the only source of information online for the majority” (Hogan & Safi, 2018). The campaign came to a head in 2017 with the military spreading rumours to both Buddhist and Muslim groups that “an attack from the other side was imminent” (Mozur, 2018).
In August 2017, the Arakan Rohingya Salvation Army (ARSA) launched a coordinated attack across outposts across northern Rakhine State; the counter-response was deemed “immediate, brutal and grossly disproportionate” (UNHRC, 2018, pg. 8). The military launched ‘clearance operations’, targeting the country’s Rohingya population.
Their attack resulted in widespread violence, rape, at least 10,000 deaths, and the fleeing of 725,000 Rohingya to Bangladesh (UNHRC, 2018).
An investigation by the United Nations stated Facebook’s role in the crisis was “significant” and that its platform was used as an “instrument for those seeking to spread hate in a context where, for most users, Facebook is the internet” (UNHRC, 2018, pg. 14).
In the aftermath, Facebook reported removing accounts linked to military personnel and, by late 2018, had pulled over 64,000 pieces of content that violated its hate speech policies (Cheng, 2021). Zuckerberg advised Facebook would add “dozens” of Burmese speakers to their content moderation team (Milko & Ortutay, 2022).
However, despite promising additional resources, an internal audit in 2020 revealed that “its algorithm still could not sift for covid-related posts when they are written in local Myanmar languages, which could weaken the company’s attempts to weed out false information on the platform” (Cheng, 2021).
Facebook further noted that they rely upon its users to report harmful content and that there are low report rates in Myanmar. It has however been noted that multiple groups in Myanmar repeatedly flagged harmful content and were met with prolonged responses regarding content removal of up to 48 hours, if at all (Rajagopalan, Vo & Soe, 2018).
Despite Facebook’s reported interventions, their moderation of hate speech in Myanmar continues to be poor. In 2022, human rights organisation, Global Witness, posted eight fake advertisements in Burmese that violated Facebook’s rules for hate speech in Myanmar, which were all approved (“Facebook approves adverts containing hate speech inciting violence and genocide against the Rohingya”, 2022).
Meta now faces class-action lawsuits from both the US and the UK by Rohingya refugees in an attempt to hold the social media giant to account (Inskeep, 2021).
Regulatory intervention: The road to accountability
This case study raises pertinent questions surrounding how to legally hold Meta accountable in a context where social media platforms are protected as intermediaries rather than publishers of content. It also brings to light the multiple ways in which the company was unable to effectively self-regulate itself to keep its users safe.
There is a longstanding problem with suitable action from platforms when it comes to online content moderation. As Woods and Perrin (2021) note, “the regulatory environment has not only facilitated but encouraged this failure, as some forms of engagement with user interactions […] endangered platforms’ immunity [and] is exacerbated by the consolidation of the market into a small number of super-dominant providers who have little incentive to compete on the basis of safety” (pg. 93).
Nation-states influence speech in two ways: via laws and through informal pressure of companies to self-regulate their own community standards (Arun, 2022). Platforms have always tried to strike a balance with governments to ensure they’re able to continue servicing local markets while staying self-sufficient enough to deter regulatory intervention (Arun, 2022).
Moderating online hate speech concerns are often made difficult due to the right to freedom of speech and involves “competing principles of promoting freedom of expression and minimising censorship on the one hand and having effective legal sanctions against hate speech and online abuse on the other” (Flew, 2021, pg. 94).
Social media companies are typically protected under Section 230 of the US’s Communications Decency Act, which grants them “broad immunity from liability for user-generated content posted on their sites” to “encourage platforms to be ‘Good Samaritans’ and take an active role in removing offensive content, and also to avoid free speech problems of collateral censorship” (Klonick, 2018, pg. 1602).
However, the lawsuit lodged at Meta notes that local law in Myanmar does not have such a law, and secondly, that Section 230 would be irrelevant as Facebook’s algorithmic amplification of hate speech and disinformation went beyond distribution alone to engage in the content creation process (Inskeep, 2021).
Woods and Perrin (2021) advocate for a service design-focused approach to regulating platforms. They note that “the platforms which affect user behaviour reflect choices made by the people who create and manage them, and those who make choices should be responsible for, at least, the reasonably foreseeable risks of those choices” (pg. 95).
The House of Commons’ Hate crime: Abuse, hate and extremism online report strongly suggests increased transparency and accountability of platforms by way of publishing “quarterly reports on their safeguarding efforts, including analysis of the number of reports received on prohibited content, how the companies responded to reports, and what action is being taken to eliminate such content in the future” (HoC HAC, 2017, pg. 23). As platforms lack transparency regarding their content moderator metrics, such reports could include the number of human moderators employed and their language capabilities.
Unfortunately, these suggestions are not binding or enforceable. The report recommends that governments intervene to enforce such reporting if platforms weren’t to take it upon themselves voluntarily.
While Facebook, Microsoft, YouTube, and Twitter voluntarily joined the European Union’s ‘Code of Conduct’, which requires platforms to expedite the review and removal of flagged illegal hate speech within 24 hours, this is, however, a voluntary agreement and ramifications for poor adherence are unclear (HoC HAC, 2017; Arun, 2022).
While laws and internal governance measures such as community standards work as frameworks with which to label hate speech, the Rohingya case draws attention to a complex landscape of inefficiencies in which a self-regulating company was unable to effectively moderate harmful content.
As seen from the leaked Facebook documents, when notified of harmful outcomes from its engagement-based ranking algorithm, Facebook pursued growth and profit over minimised user harm.
Therefore, it is unlikely that an entirely self-regulated and internal complaints process would ever result in outcomes that went against Meta’s interests. Consequently, transparency measures, such as enforced ongoing reporting, in addition to third-party intervention and enforceable measures when platforms are non-compliant must be implemented.
Conclusion
The violence and harm perpetrated against Myanmar’s Rohingya population have been catastrophic, affecting hundreds of thousands. Equally devastating is the knowledge that the aggressive hate speech and disinformation campaign propagated through Facebook could have been avoided.
The leaked documents show that Facebook is ineffective at taking serious action to remove harmful hate speech from its platform and that its algorithms amplify harmful content. Facebook’s inability to act, teamed with the company’s preference for growth over user safety, is evidence that expecting adequate self-regulation of these behemoth companies is a fallacy.
A range of transparency measures are required here, such as ongoing audits and reporting of content moderation quotas, third-party intervention, and enforcable punitive measures to companies who cannot comply with efforts to reduce user harm.
The class-action lawsuit levelled at Meta is significant as it attempts to convince lawmakers that platforms such as Facebook are no longer safe as mere distributors of content due to its engagement-based AI algorithm.
While it will be up to the courts to deliberate upon the case, the onus shouldn’t be upon refugees driven from a violent nation to bring ineffective social media companies to account. Third-party intervention is required to ensure content moderation of harmful hate speech content is appropriately, accurately, and swiftly removed to ensure the safety of users and the public.
References
Arun, C. (2022, March 20). Facebook’s faces. Harvard Law Review. Retrieved from https://harvardlawreview.org/2022/03/facebooks-faces/
Cheng, A. (2021, December 7). Rohingya refugees sue Facebook for $150 billion, alleging it helped perpetuate genocide in Myanmar. The Washington Post. Retrieved from https://www.washingtonpost.com/world/2021/12/07/facebook-rohingya-genocide-refugees-lawsuit/
Facebook approves adverts containing hate speech inciting violence and genocide against the Rohingya. (2022, March 20). Retrieved from https://www.globalwitness.org/en/campaigns/digital-threats/rohingya-facebook-hate-speech/
Flew, T. (2021). Regulating platforms [Kindle version]. Retrieved from https://www.amazon.com.au/Regulating-Platforms-Digital-Media-Society-ebook/dp/B09MYHS5XJ/ref=tmm_kin_swatch_0?_encoding=UTF8&qid=&sr=
Hagey, K., & Knutson, R. (Hosts). (2021, September 18). The Facebook files, part 4: The outrage algorithm. In The Journal. Gimlet. Retrieved from https://www.wsj.com/podcasts/the-journal/the-facebook-files-part-4-the-outrage-algorithm/e619fbb7-43b0-485b-877f-18a98ffa773f
Hogan, L., & Safi, M. (2018, April 3). Revealed: Facebook hate speech exploded in Myanmar during Rohingya crisis. The Guardian. Retrieved from https://www.theguardian.com/world/2018/apr/03/revealed-facebook-hate-speech-exploded-in-myanmar-during-rohingya-crisis
House of Commons Home Affairs Committee. (2017). Hate crime: Abuse, hate and extremism online (No. HC609). Retrieved from https://publications.parliament.uk/pa/cm201617/cmselect/cmhaff/609/609.pdf
Inskeep, S. (Host). (2021, December 14). Facebook’s parent is being sued by Rohingya refugees over Myanmar violence [Radio broadcast episode]. Retrieved from https://www.npr.org/2021/12/14/1064011279/facebook-is-being-sued-by-rohingya-refugees-over-myanmar-violence
Klonick, K. (2018, April 10). The new governors: The people, rules, and processes governing online speech. Harvard Law Review. Retrieved from
https://harvardlawreview.org/2018/04/the-new-governors-the-people-rules-and-processes-governing-online-speech/
Knutson, R. (Host). (2021, October 19). The Facebook files, part 7: The AI challenge. In The Journal. Gimlet. Retrieved from https://www.wsj.com/podcasts/the-journal/the-facebook-files-part-7-the-ai-challenge/318d6439-93e5-4352-9f08-402af9a79d4b
Linebaugh, K. (Host). (2021, October 3). The Facebook files, part 6: The whistle-blower. In The Journal. Gimlet. Retrieved from https://www.wsj.com/podcasts/the-journal/the-facebook-files-part-6-the-whistleblower/b311b3d8-b50a-425f-9eb7-12a9c4278acd
Milko, V., & Ortutay, B. (2022, March 22). ‘Kill more’: Facebook fails to detect hate against Rohingya. ABC News. Retrieved from https://abcnews.go.com/Business/wireStory/kill-facebook-fails- detect-hate-rohingya-83576729
Mozur, P. (2018, October 15). a genocide incited on Facebook, with posts from Myanmar’s military. The New York Times. Retrieved from https://www.nytimes.com/2018/10/15/technology/myanmar-facebook-genocide.html
Rajagopalan, M., Vo, L. T., & Soe, A. N. (2018, August 28). How Facebook failed the Rohingya in Myanmar. Buzzfeed News. Retrieved from https://www.buzzfeednews.com/article/meghara/facebook-myanmar-rohingya-genocide
Roberts, S. T. (2021). Behind the screen: Content moderation in the shadows of social media. [Yale University Press version]. doi:10.12987/9780300245318
United Nations Human Rights Council. (2018). Report of the independent international fact-finding mission on Myanmar (No. A/HRC/39/64). Retrieved from https://www.ohchr.org/sites/default/files/Documents/HRBodies/HRCouncil/FFM-Myanmar/A_HRC_39_64.pdf
Woods, L., & Perrin, W. (2021). Obliging platforms to accept a duty of care. In M. Moore & D. Tambini (Eds.), Regulating big tech: Policy responses to digital dominance (pp. 93–109). doi:10.1093/oso/9780197616093.001.0001