The spreadsheets and the shadow
That content moderation can cause trauma is not a mystery by now. The risk factors are well-documented, and most of them arise from the choices that we make long before anyone logs on to open a file. I know this because I have lived it and did not fully grasp what was happening in real time. It has taken me a long time to find the right language to write this post, and my hope is that it might help someone who needs it.
When people think about the cost of content moderation, they probably picture someone interacting on a daily basis with the absolute worst of what is out there. Much has been written about that already, and my reality was actually much broader than that. As well as managing the usual misuse you get wherever there is user generated content, I also had to take down documents released by public authorities that contained the private information of people whose safety depended on them being removed quickly. This could range from a single name in a single file to detailed descriptions of torture and child abuse. Most of the files were fairly mundane, but others were horrific, and those were much harder to just move on from. When you don’t know which you are getting, over time, you start to stay on constant high alert.
In small doses this was manageable, but at scale it was not. Circumstances meant that at one point I was tasked with single-handedly working through a backlog of over ten thousand spreadsheet releases, most of which I knew in advance contained hidden information that shouldn’t be there. Working to a deadline, I typically had a few minutes or less to assess each one, so I couldn’t afford to sit with what I was seeing. I compressed what I was feeling, zoned out, got seized by anxiety, threw up, and ultimately kept going.
I was working much too fast to process what was happening in real time. After finding a breach that was distressing, I had to move straight to the next one, because there were thousands more files left. On some days, this meant handling several serious incidents in a row. One morning, I’d found 54 breaches before the first meeting of the day.
I motivated myself to push on by saying that if I could get it done, then nobody else would have to do this, and by hoping that I was helping. With time, I have realised that I had not been helping myself.
The compression that made the work endurable also made the impact it was having on me invisible. Anyone looking at my reports would just have seen the kind of sanitised statistics and counts that you get in any bland audit. They did not see what I saw. I knew that behind the data were real people with real lives. I now know that some had moved countries, armed themselves, and changed their entire daily routines out of fear for their lives and their families. I feared that at the time, and working against that backdrop considerably raised the imagined cost of getting any single judgement call wrong. It was front of mind every single time I opened a file. I spent hours of my free time building increasingly elaborate triage systems to try to reduce the risk that I might let one slip through.
When home is where you work, the shadow of the work doesn’t leave at the end of the day. The mishandling of people’s private information was a violation, but it always felt like there was a second violation when I looked at it too. This feeling built, and some of the hardest breaches to deal with were those that affected people whose circumstances mirrored my own. I saw myself in the data.
As the weeks went on, I slowly co-opted the language of scale and triage without noticing I was doing it. I reduced what I was seeing to categories, severities, confidence levels and cell counts. This was a useful skill to have whilst I was inside of the work, but it also obscured the mounting cost of it, even from myself. The reports that I wrote were safe and thorough, and nothing in them would have hinted at what the work felt like to do. In a way, I had been trained by the work itself to leave that part out. When I did try to talk about it, I struggled to find the right words. I was very keen not to burden others. That was a mistake on my part, especially as the one off exercise eventually transitioned into a BAU task.
I wish I had understood sooner that getting the work done wasn’t evidence that I was coping. I had buried the damage so well that I was even gifted a joke mug about spreadsheets. It sits at the back of the cupboard, and looking at it now only reminds me of how completely invisible the reality of that work was.
I have spent much of the past year emerging from this particular shadow. As part of that, I have been given a lot of information about what is known about the risks. What research there is maps closely onto my own experience. Frequent exposure to distressing content, a heavy workload, time pressure and a lack of proactive wellbeing checks are all things that make harm more likely. Remote work, isolation and the existence of concurrent workplace stressors are further risk factors. The things that have been shown to help seem straightforward. Things like being rotated off the work after exposure to the most extreme content and ensuring that no single person feels they are carrying the full weight of these decisions alone.
Things are better now and I found more breaches yesterday free of the usual grip. I have written this because I had not read about anything like this in this particular context before, and I might have acted differently myself had I done so. I now know that this risk exists wherever humans are the last line of defence between harmful content and the wider public. The accumulation of what may seem at first glance to be lesser things, under pressure and in isolation, can do damage.
If you are struggling with this kind of work, please know that you are not alone. Below are some resources that might help.
- Samaritans: Available 24/7. You don’t need to be in a crisis to call. Call 116 123 for free, confidential, and non-judgmental support.
- Mind: The UK’s leading mental health charity. They can provide practical advice. Their website also has excellent resources on workplace mental health and understanding vicarious trauma.
- PTSD UK: Dedicated specifically to supporting anyone affected by Post-Traumatic Stress Disorder and Complex PTSD (C-PTSD), regardless of what caused it.