As communications and information-dissemination systems become smarter and more complex, it seem anything’s possible. Just be careful what you wish for: remember it’s one in, all in.
Fake news comes in two flavours: so bitter it turns us purple with rage, or so saccharine it can lure us into a false belief. The internet has always had a certain percentage of denizens prepared to deploy a clever lie in pursuit of their aims, whether anger or unquestioning acceptance. What would it look like if we weaponised lies in order to protect ourselves against that sort of mendacity – and used a clever lie to get it started?
At the beginning of June an odd and wonderful Tweet came through my feed: a screenshot of a page from the online edition of The New York Times with an intriguing title: “Reddit’s new ‘Heaven Banning’ Policy Proves Controversial”.
Heaven banning? The article itself – at least the part captured in the screenshot, explained the term:
When Albert Tallan lost his job last spring, he took his frustrations out in various subreddits.
At first, his comments were not well received. He was called a troll and mocked for the frequent solecisms in his prose. But slowly things seemed to change. Mocking transitioned to fulsome praise.
From his perspective, it seemed he was being a pillar of various subreddits he visited. In reality, his comments were invisible to all users besides himself.
These sycophantic replies, also visible only to him, were the work of bots (codenamed ‘angels’ by Reddit staff) used to enmesh trolls in a parallel fictional Reddit.
Reddit’s engineering staff, careful to preserve the viability of the site as a space for conversations free from trolling, had developed and deployed a system that could detect trollish behavior from its millions of users. As Reddit has a commenting and voting system that allows its users to rate user posts, bad behaviour is quickly visible – and, under Reddit’s new system, the user exhibiting that behaviour was quietly migrated out of the real Reddit – the one with tens of millions of humans – into a ‘parallel, fictional Reddit’ fully populated by bots (automated programs) dedicated to nothing more than flattering the troll.
Reddit created a mirror for the troll, made them the fairest of all – and made it impossible for them to look away. Call it the Narcissus Trap.
Technically, it wasn’t even that hard to create a heaven banning tool. The current generation of conversational AI systems, such as the GPT-3 large language model (LLM), can easily generate an infinitely varied stream of sycophantic praises, providing just enough variation to prevent the troll from ever becoming aware that they’ve been caged.
Reddit’s ‘heaven banning’ gives everyone exactly what they want
Reddit’s ‘heaven banning’ gives everyone exactly what they want – the troll gets to be the smartest, wisest and most thoughtful person on their own private Reddit, while the rest of the community, breathing a sigh of relief, can carry on with their global-scale conversations. It’s genius.
If only it were real.
When something sounds this good, it always pays to read the fine print. In this case, it reveals that this ‘article’ was published in the 8 August 2024 issue of the New York Times – just over two years into our future. So rather than the description of a new technique to deal with poorly socialised internet users, it’s only another bit of fake news.
Read also: Is Google’s AI chatbot LaMDA sentient? Computer says no
Or – if we can be a bit more generous – it’s actually a provocation, disguised as science fiction. A proposal disguised as a fait accompli. Asara Near, author of the post (and, presumably, the fake Times screenshot) hints at that, writing that heaven banning “…is entirely feasible with the current state of AI/LLMs”.
It’s not real – but it is wholly realisable, using today’s technology. The only thing keeping us from implementing a heaven banning system on a site like Reddit (or Facebook or Twitter etc) is merely that no one has yet put in the hard yards.
Any technology deployed against anyone can be deployed against everyone.
With a growing intolerance – both socially and institutionally – for trolling and online harassment, it likely won’t be very long until heaven banning becomes an accepted and ‘safe’ practice for isolating trolls. By 2024, it could be an broadly accepted practice.
But if that happens, and we live in a world where we deploy systems to imprison trolls within halls of mirrors, how can we ever know that we, ourselves, remain free? Heaven banning carries a sting in its tail, because any technology deployed against anyone can be deployed against everyone. Facebook already selectively ‘curates’ its Newsfeed to reflect the emotional state of its users; while not heaven banning, per se, it’s got all of the necessary pieces – except a component that would churn out endless fake-but-agreeable Facebook posts.
It’s a small step from where Facebook is in 2022 to where they could be, with a few billion happily heaven-banned users, trapped within prisons of their own design, each reading and commenting on only those things that reinforce their own beliefs and sense of self-worth.
Although there’s been a lot of talk lately about whether large language models are sentient – that is, whether they can truly be said to think for themselves – some more measured voices on this topic point to how these systems tend to serve as a mirror for our own beliefs. If you want to believe these systems are sentient, say these researchers, it’s quite likely those systems will produce outputs that will tend to confirm your beliefs. Which means it’s all mirrors, and everywhere we look we see our own reflections.
How might we ‘test the spirits’, to ascertain whether we’re in the real world, connected to real people? Although large language models offer a simulation of a fantasy of the real, they fall apart when confronted by randomness, weirdness and inconsistency. Quickly change the track of a conversation, and they fail to grasp the change of context. The real world, filled with noise and chaotic, unexpected interruptions, lies outside the carefully curated data sets used to train these models.
So if you suspect you might have been trapped within a fantasy world, populated by your own desires – amplified and fed back to you – rattle the bars of your cage. Make up nonsense. Be random and unpredictable. Then watch how the world responds. If it ignores you – or worse, if it babbles incoherently – you have your answer, and need to have a good long think about how to free yourself from the prison of heaven.
Originally published by Cosmos as Under the banning of heaven
Mark Pesce
Mark Pesce invented the technology for 3D on the Web, has written seven books, was for seven years a judge on the ABC's "The New Inventors", founded postgraduate programs at USC and AFTRS, holds an honorary appointment at Sydney University, is a multiple-award-winning columnist for The Register, pens another column for IEEE Spectrum, and is a professional futurist and public speaker. Pesce hosts both the award-winning "The Next Billion Seconds" and "This Week in Startups Australia" podcasts.
