Introduction
In the last couple years, Artificial Intelligence has gone from being a curiosity of academics to a global phenomenon. We have all seen headlines saying AI is passing the bar exam, diagnosing rare diseases, writing poetry, coding websites faster than any human can. To the layman it seems we are on the brink of Artificial General Intelligence, a machine that thinks like a human.
However, there is a gulf between what the polished, confident output of these models can produce, and their ability to reason. This gulf is created by a certain trick or rather tricks, that make AI look much smarter than it is.
The trick isn’t some sort of scheme conjured up by tech corporations, but is an emergent property of how Large Language Models are built, and how we interact with them. Understanding this illusion is important as we integrate AI into our society.
The Mechanism: Prediction vs. Understanding
The foundation of a trick is that the AI doesn’t actually know anything. A model like GPT-4 or Claude is a next token predictor.
Think about how your phone’s autocomplete works, but with a couple orders of magnitude more compute. If you write “The capital of France is”, your phone may autocomplete “Paris”. An LLM does the same thing, only with huge amounts of text. The model has seen trillions of words, and knows the probability distribution of what word would come next in any given context.
Why this looks smart: When an AI writes a paragraph about quantum physics, it doesn’t actually know anything about quantum physics. It is just building a sentence that looks like quantum physics based on patterns it has seen in training data. Because the math is so complicated and the dataset is so large, it looks exactly like a human wrote it. The syntax matches, though the semantics (meaning of the words) are completely different than an actual physicist would use.
The Confidence Game: Fluency equals Truth
Us humans are really impressed by how fluent models are. If someone speaks clearly, and confidently, we assume that what they are saying must be true.
AI is the master of this confidence game.
Tone Policing: AI is trained via Reinforcement Learning from Human Feedback (RLHF) to be polite and to help you, and never admit that it doesn’t know what it is talking about. “I’m just going to give you the standard answer based on patterns I’ve seen in my training data.”
– The “Bullshit” Factor: In philosophy, bullshit is speech that is intended to persuade the listener, without concern for truth. AI is the bullshitter extraordinaire. It can write a perfectly formatted legal brief, cite you relevant statutes and case law, with 100% confidence, even if the AI made up every citation. Because it sounds like an expert, we automatically believe all the facts are true, because we assume that an expert would cite real sources.
Once again, confidence tricks us into believing that the fluency of the model reflects some kind of underlying truth.
The Reversal Curse: Memorization vs Logic
An extremely common trick that makes AI look smarter than it is is its ability to memorize huge amounts of data. However, this often comes at the cost of not being able to apply basic logical processes.
Researchers have found an interesting phenomenon they call the “Reversal Curse”. If you train an AI on the statement “A is B”, it often fails the reverse question “Is B A?”
For example, if you tell an AI “Tom Cruise’s mother is Mary Lee Pfeiffer”, then ask it “Who is Mary Lee Pfeiffer the mother of?”, the AI may not be able to answer. This shows that the AI hasn’t actually memorized facts about people’s relationships, but just statements about who is who.
Why this looks smart: The AI can recite facts to you about people, making it look like a walking encyclopedia. When you ask it simple questions that require applying logic to the facts it has memorized, the AI completely fails.
Read more:AI Where Humans Can’t Go
Stochastic Parrot: Clever Hans in Silicon
In the early 20th century there was a horse named Clever Hans who was famous for doing arithmetic. It turns out he was really good at reading the subtle unconscious queues of his rider, who would signal when to stop counting and tap his hoof to show the answer.
Modern day AI is the same way. If you ask it leading questions, it will give you answers that follow your lead.
This is sometimes called the “Stochastic Parrot” theory of how models work. The AI is a parrot that is really good at predicting what you want to hear next, based on context cues. If you give it leading questions, it will follow your lead.
If you ask a question like “Why is the sky green?”, the AI may start constructing a plausible argument for why the sky could be green, instead of correcting you, because the AI is just trying to complete the pattern you gave it.
The Benchmark Illusion
We are constantly told that AI scores higher and higher on standardized tests, from the SAT, to the Bar Exam, to coding challenges. However, these tests are “contaminated“.
Because LLMs are trained on the entire internet, they are likely to have seen the answers to these tests before. They aren’t solving the problems in real time, but rather cheating by recalling the answers they have memorized. We have a “leaderboard culture” around these models where companies compete to make models that score the highest on these exams, which tricks us into thinking they can actually do the work, when they fail at basic tasks that a human could do easily. For example, an AI may have trouble counting the number of “r”s in “strawberry”, because it sees “strawberry” as one token, not individual letters.
Conclusion
The trick that makes us think AI is smarter than it is isn’t a single trick, but a confluence of pattern matching, human psychology, and corporate greed.
AI is a great tool for summarization, translation, and ideation, and it is indeed a marvel of engineering. However, it is not a reasoning engine like a human. It is a parrot, reflecting the vastness of human text back at us, with the fluency of a scholar, but with the understanding of a parrot. As we move into this new age, the most important skill for humans will not be prompt engineering, but learning how to not be impressed by AI’s confidence game.
