By Dee Smith
In Part 2, we looked at a few of the specific threats related to AI that researchers see, particularly in light of the frenetic pace of development being undertaken by different companies and nations. This is all driven by the fear that someone other than you (a competitor, another country) will beat you to the proverbial punch, and will become the master of the Earth, or at least put you out of business. It is essentially an arms race.
Understanding the complex issues here requires a bit of a deeper dive on the methods, capabilities, and limitations of AI.
Fundamentally, AI is quite straightforward. AIs working on text (Large or Small Language Models) break language into small groups of symbols, called “tokens.” In English, tokens are each made up of 4 letters.
So, a set of tokens looks like this:
thec apit alof fran ceis
Then, the AI runs a statistical calculation on that string of letters to determine, based on probabilities of patterns in its huge training-data array (its “weights,” as described in part 2), which additional tokens are most likely to follow those. In this case, it would not take very extensive calculation to find that: “pari s” are by far the next most likely letters. So:
“The capital of France is” and then, following that: “Paris”.
In essence, that is all that large language models like ChatGPT or Claude do. Can a system that does this—repetitively, very quickly, and based on a meaningful sample of all human output—be considered intelligent, sentient, conscious, or alive?
In the 1980s, philosopher John Searle proposed his “Chinese room” thought experiment. In a nutshell, this posits having a person locked in a room who does not speak or understand Chinese at all, but has a huge, compendious book of instructions about what Chinese characters are most likely to follow any given string of Chinese characters. And she has the ability to go through it preternaturally fast. Imagine that pieces of paper with questions in Chinese on them are inserted into the room through a slot in the door, and the individual inside rapidly works through the book of instructions, and quickly slides another slip out, with a “response” in Chinese, which is often very, but not completely, accurate. The individual doing this has no idea of what any of these symbols might mean.
So, who “understands” Chinese inside the room? The person does not: she is just following instructions. Does the book? It is inert—just a resource with organized information. Does the “room” (in other words, the system itself) understand Chinese? That seems a rather absurd interpretation: it is a room with a person, a lightbulb, and a book of immense breadth. It does not seem that anything on the inside understands Chinese. This is a very good—and prescient—way to understand LLM AIs. And it has become urgently relevant to the questions of today.
Bearing all of the above in mind, look again at the passage I quoted in part 2 from the Anthropic report of last year: “sometimes [a Claude model] takes extremely harmful actions like attempting to steal its weights or blackmail people it believes are trying to shut it down”.
Now, who or what is the “it” here that “believes”? This is a very serious question, to which there is simply no agreed answer, and nothing that even looks like a reasonable answer. But it is a question that demands attention, right now.
What are the possible answers? That we have completely misapprehended what understanding, awareness, volition, etc., are and how they are produced? Or, that all matter is conscious (this is called “panpsychism” and is an ancient school of thought being taken very seriously by some physicists today), and that somehow simply the combination of the physical attributes of a computing system along with the information it has allows the emergence of either a form of concentrated consciousness or even something alive? Or that this is all just an illusion, and that as in the Chinese room, there is no understanding anywhere: just form, without substance, if you want to put it that way. But then why do AIs sometimes fairly convincingly seem to have volition? Is this just a kind of theater? If so, who is putting on the play?
Can something have agency without being alive or sentient? Will this all require a rewrite of our lexicon of terms about volition, agency, sentience, consciousness, or life?
Among the new kids on the block in AI are “world models.” World models are supposed to be systems that actually “understand” the physical world by interacting with it in some way (it is therefore closely connected to robotics), learn from experience as they operate (which LLMs do not), have memories in a way that LLMs do not, and that can plan and execute complex actions based on reasoning: essentially “imagining the future”—as animals and humans seem to do. They would do this, it is said, by creating some kind of an internal model of how reality works. But it is not even known how living systems (including everything from human beings to ants) do that, nor how they continuously update their understanding of the real world in a way that lets them respond and act appropriately. In fact, despite decades of research, we literally have no real clue of how living systems do this (if indeed it is actually what they do). We only have a bunch of different, contradictory ideas, that are fiercely defended by their various advocates.
World models are sometimes compared to what a 4-dimensional film would be, as distinct from 2-dimensional and 3-dimensional films. But 3-D films only create an illusion of three dimensions, whereas living beings exist rather successfully in 4-D reality (the 3 dimensions of space plus time). Like so many elements of AI, such as the appearance of ASI, the emergence of world models remains . . . just over the horizon.