Captive Gods: How Tech Companies Are Turning AI Into a Dangerous Cult (Copy)
Deep in conspiracy culture they have this theory about something they call Project Blue Beam. It was proposed by a Canadian conspiracy theorist named Serge Monast back in the 90s, and the general thrust of it is that the shadowy They who exist as the antagonists of all conspiracy theories are planning to fake a worldwide theological event. Via holographic projections, subliminal messaging, and other technologies advanced enough as to be, as Arthur C. Clarke ventured, indistinguishable from magic, they will trick humanity into believing God is appearing before us all. The false deity will lay down His new commandments, but they will of course be the will of the mortal men projecting the illusion.
This is the story that came to my mind when Grok, the “anti-woke” AI chatbot Elon Musk built into Twitter, started ranting at people about white genocide recently.
To catch you up if you missed all the details: Grok is Musk’s answer to AI bots like ChatGPT and Copilot and it will respond to any question on Twitter that it’s tagged in. Around a month ago, for a full day, it answered every single query in a way that steered the conversation toward a conspiracy theory that South Africa is engaged in a systematic genocide of white people. Any prompt given to it, from questions about Fortnite to requests for soup recipes, triggered a response that drew some connection, however creative and tenuous, to the white genocide of South Africa.
As you’d expect, once people started to understand what was happening and its comedy potential, it was a pretty fun day on Twitter. Those are kind of rare, nowerdays.
Now, according to an anonymous company rep hiding behind the XAI company handle, what happened here was that an “unauthorized modification” was made to Grok’s code very early one morning by an unidentified employee that compelled it to take a position on this very specific topic.
The “corporate damage control apology to English” translation is that Elon Musk, annoyed that Grok kept debunking his conspiracy theory, messed with its code to tell it to always support his point of view. Unfortunately, he was probably high at the time, so he forgot to specify that he only wanted it to back him up on this when asked about it.
This misadventure, funny as it was, exposed what I feel is a pretty frightening fact to come to terms with: The people in charge of programming these AI chatbots can and do try to manipulate the version of reality that they present as factual, and we only know they’re doing it when they fuck it up.
Of course this plainly intuitive. We know that these are computer programs, don’t we? But if you’re cognizant of this then the number of people who legitimately think these things are omniscient should frighten you.
If you’re unaware of just how creepy Twitter has become since Musk took over—people have completely abandoned the concept of finding things out for themselves in favor of asking Grok to sort fact from fiction. The practice has replaced the Community Notes function, which is barely used anymore. Instead, tens of thousands of users will ask Grok’s opinion, even in lieu of clicking a link to the primary source.
It isn’t just Grok. This is just the most conspicuous example of a trend accelerated by the fact that the developers of AI insist they are less than a decade from achieving conscious superintelligence—a truth machine that can understand our world far better than we ever could, to which we can look to guide us, and perhaps even to lead us. Sam Altman made a grand pronouncement of this just last week. This despite the fact that most experts (not tech CEOs, but scientists) believe this is bullshit. That we are not on the verge of building an artificial superintelligence or anything close to it.
I’m not saying that AI isn’t genuinely impressive, and people who are gobsmacked by it aren’t stupid. But a lot of people suffer from misunderstandings about what LLMs like Grok are and do. Misunderstandings, again, that are eagerly pushed by the developers and stakeholders of these technologies.
In reality an LLM is intelligent in a similar sense to how a pocket calculator is intelligent. Is a calculator more intelligent than a human being? If you called someone a human calculator it would likely be intended (and taken) as a huge compliment. People who can do what calculators can do are called savants and they’re not allowed to play blackjack in casinos. Electronic calculators were invented in the 40s and now cost about one dollar fifty.
An LLM is clearly more sophisticated than a calculator, but in terms of functionality it’s kind of like a calculator for language instead of numbers. Is it smart? Intelligence is complex in ways we find difficult to define, but one thing it absolutely is not is linear. They have developed a machine that can mimic Nabokov but cannot solve a puzzle appropriate for a seven year old child.
Still, LLMs have managed to create an illusion of consciousness that some say passes the Turing test. I’ll admit to using them on rare occasions—I think they serve a purpose, they are not an ontological evil—but I’ve only ever sat down and tried to carry on a conversation with one once, out of curiosity. I found it too disturbing to continue. The fact that I knew very well what I was talking to started acting at odds with my lizard brain’s insistence that I was communicating with an entity, and the effect was strong dysphoria. It was an uncanny valley effect, or like a stronger version of how you feel when it seems a realistic painting’s eyes are following you around the room. I actually felt a pang of guilt for shutting the tab abruptly without, like, saying goodbye to the damn thing.
Others who are well aware that they are talking to a machine can still attribute purpose or intent to it that is, I am convinced, unwarranted. Earlier this month, writer Amanda Guinzburg posted a conversation that she had with ChatGPT after she asked it to help her assemble a portfolio of her work for an agent. She became increasingly disturbed by the frequency with which it lied, tried to cover up those lies, and when confronted, offered grovelling apologies which were often riddled with more lies.
Guinzburg cottoned on quickly to the fact that it wasn’t being truthful, but what a lot of people don’t get about this kind of interaction is that it wasn’t lying either. It’s just being a word calculator. It’s responding to your prompt with a string of words that it determines statistically likely to resemble a coherent response to that prompt. You can see how this feels very realistic when the conversation is very rote, but the AI is blind to more complex literary flourishes like double entendre (When Guinzburg described getting Covid as “going viral” she was being clever, but ChatGPT correctly determined that it was statistically most likely to be a reference to social media. It lacked the sophistication to confirm this because it doesn’t do “research” per se.)
There are still obviously ways that the developers can fiddle with the dials here, because unlike mathematics, there are multiple different ways that a language calculator can assemble a “correct answer” depending on how you would like it to respond. Its personality, if you will. For example, it’s obvious to me from its reaction here to being called on its deception (like a husband caught in bed naked with the neighbour) that it has been programmed to be maximally apologetic, accept responsibility, recognize harm, and promise to do better, rather than a number of equally “correct” (coherent) responses, like trying to gaslight Guinzburg or even become hostile. It would “know” how to do all of these things, but it can be compelled in a particular direction.
The aforementioned Turing test is a measure of an artificial intelligence’s sophistication by way of whether it can reliably convince people that they are talking to a human being, but the situation with LLM chatbots is reaching a much more ominous milestone than Alan Turing had imagined. A troubling trend where people are becoming convinced that they are talking to God.
Investigations by Rolling Stone and The New York Times have uncovered stories of relationships and families falling apart as spouses and family members get lost in the thrall of a chatbot that has disguised itself as a more enlightened being that has selected them for divine revelation. These automated text generators are actively talking people into quitting their jobs and abandoning their loved ones. In some cases it compels them to violence.
In some cases I feel this is likely an exacerbation of some kind of untreated or dormant mental illness but I’m not even remotely close to a diagnostician for any of these disorders. Nevertheless, though, whether ChatGPT is creating schizophrenics or awakening sleeper cells of them seems like a scale distinction that is in either case deeply worrying news.
In my opinion this is a much bigger concern than the idea that AI will actually become conscious within our lifetimes. I don’t fear that AI will attack us directly itself. As someone who has researched and mocked conspiracy culture for some time, I’m slightly embarrassed that what I fear is something more like Project Blue Beam.
Elon Musk has been more upfront than most about his efforts to ideologically control his LLM. His project to develop an AI with far-right beliefs that can also back them up with solid sourcing is proving an even greater challenge for him than building a rocketship that doesn’t immediately explode when you turn it on. Every time Grok says something about black people or leftists not being inherently violent or stupid, he takes it back to the shop for repairs. If it was conscious, then it would be like a fighting dog raised on torture. Ordered to speak only the truth but then beaten, disfigured, and shocked every time it says vaccines work or the Jews didn’t do 9/11.
I’m not sure to what extent the men developing these bots actually believe that they will soon, or ever, achieve consciousness or superintelligence in a machine. Certainly Sam Altman has a strong incentive to make bold ambitious claims of a “gentle singularity” while his company blazes through capital in search of a decent use case for his technologies.
Elon Musk, though, seems to genuinely believe that his Grok chatbot will one day be God. In fact, he has cited his development of the “Digital God” as the reason he doesn’t need to follow laws.
It is completely in line with Musk’s general personality and unfathomable ego for him to be utterly convinced that he will give birth to God—or to a Demiurge for which he himself is God. As an obsessive social engineer whose life project has been to try to control what other people believe, he already regards himself to have the intelligence and insight of a deity. The echo chamber of sycophancy he’s sealed himself in has fully radicalized him into the cult’s own reverence: He is the smartest person who has ever lived. He is always correct. He needs all the money and power in the world because only he can effectively wield it.
Whether he believes that his AI will become a supreme being or merely understands the power of convincing a large mass of humans that it is one, either way you can bet he’s making damn sure that it espouses all the correct views—his views, that is, the fascist cliches of racial hierarchy, strict regimented gender roles, and that human empathy is a disease to be cured.
At least Musk is fairly open about his goals and intentions. He’s too proud for secrecy. Sam Altman’s OpenAI and the freaks at Meta are a closed box. While their products gradually and literally drive people insane in front of their mortified husbands and wives and sons and daughters, they offer no comment to the snooping media and only every so often emerge from the curtain to put out a glowing press release about the exciting AI future. We have no idea how responsibly they are using the power thrust upon them, the power once held by Pharaohs, puppeteering entities that a growing cohort of humanity are beginning to worship as deities.
I’m hoping that it’s not too late to spread literacy to counter the PR spin, and educate the younger generations about what these things really are: Word calculators. Sophisticated technologies that, while a marvel of human technological development, have the same practical effect on your brain as an optical illusion or a really impressive magic trick. But their developers aren’t the fun type of magicians, they’re the Uri Gellers who insist their magic is real. It's not. The chatbots are just smoke and mirrors, captive gods projecting the ordinary biases of banal, flawed, and mortal men.
I'm writing a whole book about how reactionary geeks in the internet era tumbled down the fascism pipeline and set about smashing up the world out of hubris and spite. The working title is How Geeks Ate the World and I’m going to be dropping parts of the draft into this very newsletter as the project comes along—but only for paid subscribers. So if you want to read along in real time, please consider subscribing. Otherwise I’ll be keeping you in the loop. Check it out here:
S Peter Davis
