Modern AI systems can generate text. They can explain, argue, translate, summarize, write code, and sometimes even appear to reason through a problem step by step. Very quickly, it becomes pretty convinving to imagine this as a truely 'intelligent' system.

one that has some kind of understanding of the world in terms of language, with an inner monologue guiding the reasoning behind the text it generates.

But is it?

The Chinese Room

The generated text is not all there is. Behind the scenes there is a more compressed processing going on in terms of latent space representations. The model is not just shuffling around letters symbols; it is operating on some deeper learned representation internal to itself.

Similarly, for humans, there is something going on in the mind. But we do not know exactly what. The input and output — how we communicate with each other — is largely stuck in language.

So when an AI produces fluent and apparently thoughtful text, what is actually happening in there? Is there understanding, or only the appearance of it?

In 1980, the philosopher John Searle (1932 - 2005) proposed a thought experiment that has shaped the debate ever since.

Imagine a person locked in a room with a vast rulebook for manipulating Chinese characters. They themselves do not speak a word of Chinese. Native Chinese speakers outside slip questions through a slot, written in Chinese script. The person inside follows the rulebook mechanically — if you see these characters, write back those characters — and pushes the replies back out. To everyone outside, the responses look indistinguishable from those of a fluent speaker. But inside the room, the person has no idea what any of it means.

Searle's point was simple. The formal manipulation of symbols — which is, ultimately, what a computer does — does not by itself constitute understanding. The room appears intelligent. But the room does not understand.

An LLM is a very large rulebook in the same sense: patterns of symbol-following, trained on enormous amounts of text. It produces fluent answers without anything we would obviously recognise as comprehension behind them.

But the picture is not quite that simple. Modern interpretability research has shown that LLMs are not pure symbol-shufflers. Inside the model, dense vector representations encode something that behaves like an internal map of concepts. Words that mean similar things end up close together. Analogies hold across geometric relationships. The model has a compressed, geometric representation of the world it has been trained to describe.

Whether this counts as 'understanding' in any deeper sense is exactly what we cannot decide from the outside.

And here is the strange parallel: the same problem applies to us. When you talk to another person, you do not have direct access to their thoughts. You see only language coming out, and you infer the mind behind it. We grant each other intelligence on the basis of behaviour and shared context — never on the basis of inspection.

So much of what we perceive as human intelligence is entangled with language. We assume the inner light is on in other people because their words look like our words. We have no other way to check.

From the outside, then, we cannot cleanly distinguish a system that genuinely understands the world from one that merely produces functionally appropriate output. The functional description of intelligence may be all we have access to.

Can an LLM use language to understan the world?

We cannot really tell. There is no clean test that separates a system with a genuine inner understanding of the world from one that simply produces output that looks the same from the outside.

But there is a deeper issue. When we ask whether an LLM 'understands', we picture understanding as something hidden inside an object — a private inner ceremony performed somewhere behind the symbols. It is not obvious that such a ceremony exists even for human beings.

What we are really asking, perhaps, is something different: does the system participate in our shared use of language well enough that we should treat it as a user of language? That question is sharper, and it is the one we will come back to in more detail in another article in this series.

Is an LLM-based AI Agent 'Intelligent' ?

If we cannot decide from the outside whether an LLM understands — and if the question itself may be confused — then the question shifts. What kind of thing is an LLM, exactly?

The LLM is not a mind in the old mythological sense. It is not a little person trapped in a box, watching the world through some inner cinema. But neither is it merely a dead pile of symbols. Something is happening in there — geometric, statistical, compositional — even if we are unsure what to call it.

Perhaps the most honest description is that an LLM is something genuinely new: a kind of linguistic organism. Not alive in the biological sense, not conscious in any obvious sense, but active inside the symbolic ecology of human beings. A system that participates in our shared use of language without sharing the embodied life that gave rise to it.

Whether such a thing should be called 'intelligent' depends entirely on what we mean by the word.

And here we run into the question underneath all the others.

Before we can answer whether a machine can reason, or understand, or be intelligent, we have to ask what picture of intelligence we are already carrying in our heads when we use those words. What do we mean by reason? What do we mean by understand? And what, for that matter, is language — the thing that holds all of these concepts together?

Those are the questions I have been trying to work out in the following articles in this series