Show Us the Prompts
How AI prose parasitizes the charity that makes reading possible
“Dogma,” Albert Einstein told an audience at the Sorbonne in 1929, “is the enemy of progress.” In English, no less. Just kidding—it’s what ChatGPT whispered in the ear of the rector of my university at the start of this academic year after she decided to outsource her inaugural address to AI. The speech she delivered was a string of grammatically impeccable platitudes garnished with four quotations, three of which turned out to be hallucinations — the kind of false erudition that academics like to flaunt. The fourth quotation was real, but it was a cringeworthy piece of self-flattery lifted from an obscure 1946 speech: “There are few earthly things more splendid than a university.” ChatGPT certainly knows how to appeal to academic vanity.
My former rector Petra De Sutter is not the only person to have been embarrassed by a digital ghostwriter. In March, the former academic turned Reform UK candidate Matt Goodwin published a book littered with fabricated quotations and non-existent references, which earned him the nickname “MattGPT.” In a case of cosmic karma, Steven Rosenbaum’s The Future of Truth: How AI Reshapes Reality turned out to contain half a dozen fake or misattributed quotes. Governments and consultancy firms have written expensive reports citing studies and papers that do not exist. An online database now lists more than 1,800 cases of AI hallucinations in court filings.
The good news for anyone recruiting an AI ghostwriter is that the odds of public mortification are falling. Fabricated quotes and references are just about the only way to catch a lazy author out, since there is virtually no innocent explanation for them. But as Large Language Models have improved, hallucinations like these have grown rarer. And without those dead giveaways, we are left with AI detection tools, which are irreducibly probabilistic. The best of them is Pangram, which boasts a false-positive rate of one in 10,000, given a long enough chunk of text and some reasonable assumptions. Writers accused of AI plagiarism routinely deny it, and there is no way to prove the charge conclusively.
If my former rector were to outsource next year’s address to a capable model, with some adroit prompting, the result would probably be a good deal less anodyne than last year’s. The best LLMs have become such virtuoso performers that a more fundamental question now presents itself: should we even care if a paper or a speech was produced by a digital ghostwriter?
An epidemic of slop
This is the opening section of my latest essay for Quillette. I’m not allowed to post the full piece here, but you can read the rest on their website. In any event, Quillette is one of the best magazines around, so please get a subscription!
The problem of AI slop in writing has been bothering me for a while. As I write in Quillette, many of the earlier cases centered around hallucinated quotes, which are pretty much the only way to catch an author with their pants down. But as LLMs have improved and trained through human feedback, their tendency to hallucinate sources and quotes has diminished (though not completely disappeared). So now we’re arriving at the stage when there are no longer any smoking guns – just probabilistic estimates from Pangram.
If AI writing becomes so good that it is no longer distinguishable from human writing (except by being far superior to it), should we care? Will we all be reading AI slop soon, with human writing becoming a niche artisanal craft for anti-AI snobs? That’s the question my essay is asking. My main inspiration was this recent piece by Tyler Cowen in The Free Press, which predicts that before long hardly anyone will care whether or not a text was written by a human. Likewise, Richard Hanania wrote that it doesn’t really matter if the author of a piece is human or not. With the possible exception of memoirs, which are inherently personal, he does not care one way or the other. I beg to differ.
Meanwhile, I keep coming across more examples of AI writing, even hit pieces. In Quillette, I discuss this viral piece on the Islamization of France that initially impressed me with its vivid and fluent prose, but then made me increasingly suspicious. Pangram duly confirmed that the piece was 99 percent AI-written (in the comments section, nobody seemed to have noticed).
Every time I read a piece like that, I’m curious about the prompts that were used. Was it something like “write a catastrophist obituary for France and how it died from creeping Islamization”? Or was it more elaborate: a long bunch of bullet points with elements and instructions? Or a collection of notes to be arranged into a full story? In any event, by the time you reach the end of the piece, it’s clear you’re reading AI slop:
You cannot save a country that has decided to destroy itself — slowly, voluntarily, and with a clear conscience. You did not fall, France. You chose to kneel. Goodbye, France.
I also asked Claude to do a fact-check of the piece, and it seems to be littered with slight distortions, errors of chronology, and misrepresentations. I didn’t verify each of them, but it doesn’t exactly increase my confidence.
Or take this viral piece by Israeli writer Nachum Kaplan (originally here). It is also from “my” political side, lampooning the idiocy of pro-Palestinian discourse, so if anything it panders to my own views and tastes.
Multiple people sent it to me as a brilliant takedown of pro-Palestinian lunacy. But although I find the piece highly entertaining and incisive, I’ve always thought it’s a bit lame to recount an unverifiable conversation you supposedly had with other people, in which you portray your interlocutors as maximally moronic and yourself as maximally savvy and eloquent, delivering one witty riposte after another. In Yiddish, as it happens, they call this trepverter: the witty comeback you only think of when you’re already halfway down the stairs.
I didn’t share the piece online, as I found it a bit too smug, but I didn’t expect that it was written by AI. But you’re not fooling Pangram, which diagnoses it as 100% AI-generated. (But surely here there must have been extensive prompting describing the general outline of the dinner conversation, and some jokes and comebacks? Surely AI is not becoming that good?)
When you’re unwittingly collaborating with a machine
The promiscuous use of AI also raises ethical questions for co-authorship. Since this is a novel technology, different people have different intuitions about how much AI outsourcing is acceptable. I’m hearing more stories from friends and colleagues who have co-written articles, only to belatedly discover that they were mostly collaborating with a machine.
It’s hard to share details, because much of this is embarrassing for all parties involved. But one fellow progress writer, who shall remain anonymous, complains about a “human” collaborator he worked with, saying that “almost everything he has written is very slightly wrong.”
AI slop, as I write in Quillette, is far more subtle and sophisticated than human slop, and exploits the principle of charity essential to human communication. You have to slow down and read each sentence with care to realize that there’s “nobody home”.
Show us the prompts
At the very least, I think we need far more transparency when it comes to AI-assisted writing. In my experience, very few people who have been accused of AI plagiarism by Pangram have come clean about it. At best, they claim to have used AI only to lightly polish their writing (which I do all the time, and, to my knowledge, it has never resulted in a false positive). If you don’t think you’ve done anything wrong, and you’re perfectly happy to be ventriloquized by a machine, why not simply show us your prompts?
On occasion, I have published AI-written pieces myself, but always including a disclaimer. When Thomas Piketty, Jason Hickel and Joseph Stiglitz wrote their atrocious op-ed in The Guardian promoting degrowth, which Pangram promptly diagnosed as 100% AI-generated, it took me less than a minute to generate a response of roughly equal length, drawing on mainstream economics, ecomodernist arguments about decoupling and the Environmental Kuznets Curve (EKC), as well as my own earlier writings on the subject. I then posted Claude’s effort without even bothering to read it, but with a disclaimer at the bottom. If you can waste my time, I can waste yours. Or, to borrow a variation on Hitchens' Razor: “What can be asserted without thought can be dismissed without thought.”
In one other instance, in which I was responding to an AI-generated “critique” of my own work, I asked Claude to write a full rebuttal on my behalf, in my style, and to stage a dramatic “reveal” in the final paragraph. Here’s how Claude executed the joke (translated from Dutch):
The entire piece exudes the typical rhythm of a large language model asked to “write a nuanced opinion piece against Maarten Boudry.” I am curious what ChatGPT or Claude would make of it if you presented it to him. In fact, I am so curious that I deliberately had this reply written by an AI. Every single word. This sentence, too. The previous one, too. The entire piece above, including the cheerful opening paragraph, the reference to Koopmans, and the joke about the Wi-Fi. I haven’t changed a single letter. The only thing I did was serve Claude an extensive prompt, filled with my writing style, my standard arguments regarding Islam and migration, my previous pieces, and all my recent posts on X regarding the VRT issue. The machine did the rest itself. And honestly: well done, isn’t it, for a language model? Seems only fair to me: if the prosecutor has his accusations drafted by an algorithm, then the defense is entitled to draw upon the same arsenal. The only difference is that I admit it.
It took two or three rounds of prompting to fine-tune the piece, but the final text is 100% AI; I didn’t change one word. Most people, I’ve been told, had no idea they were reading AI until they reached the final paragraph.
I've already replaced my proofreader, my accountant and my legal advisor with AI. Am I next in line?
Completely missed the MattGPT episode and I'm really annoyed he fell for that (and tried to make us fall for it). As if going from publisher to politician wasn't bad enough. Here's an academic with energy and a good writing hand, who's picking up the glove of hard topics surrounding migration, and he makes a Chat-clown out of himself - AND the subject, as if it doesn't have enough stigma surrounding it already.
I tend to agree with David Lynch in him saying that "Ideas are the best thing going"....
And when it comes to writing, much like the song recently put out by Boy George, what matters the most to me is the originality and meaningfulness of the idea.
Now obviously the individual writing talent of a person is also built by the skill to produce the idea, and using AI may reveal some lack of writing talent, but it seems to me that this discussion is about the quality of the output, not the judgment of the talent of the originator.
Boy George stated essentially that he doesn't play an instrument anyway, so there's a little difference between him prompting AI to produce music and him prompting a group of live musicians, only that the AI won't reject it's instructions.
He composed the lyrics and the melody just the same as he would have composed on any other song without playing an instrument.
With regards to AI slop in text or speech writing, it's kind of the same thing. Most people who are using AI to ghostwrite, most likely would have previously been using a person to do that, who could also easily make mistakes and embarrass the person, so the larger concern is the economic impact of hiring one less person and the societal psychological impact of trusting machines over your fellow professional human.