The Mind Before the Gods: Why Scientific Literacy Does Not Stop the AI Misreading

The Puzzle

The most scientifically literate people in the world are currently describing optimization systems as if those systems were minds.

Engineers who can explain transformer architectures use words like "wants" and "tries" and "thinks" to describe the outputs of their own models. CEOs of frontier labs warn the public about machine intelligence as if intelligence were the right category. Researchers who understand the mathematics of next-token prediction still report being "shocked" by what the models can do, as if the models had agency rather than improved interpolation.

If the misreading were caused by scientific illiteracy, education would solve it. The educated would see the systems clearly while the uneducated fell for the illusion. That is not what happens. The illusion grips experts and novices alike, often with experts producing the most florid descriptions because they have the vocabulary to elaborate.

Something deeper than knowledge is at work. The misreading is not about what people know. It is about what the human cognitive system does automatically, before knowledge has a chance to weigh in.

The Machinery

Humans evolved in environments where missing a real agent was usually more dangerous than imagining a false one. A rustle in the grass might be the wind, or it might be a predator. The brain that treated ambiguous patterns as potentially agentive survived more often than the brain that waited for proof.

This evolved hair-trigger is sometimes called agency detection. Some researchers call it a hyperactive agency detection device because it fires too readily by design. The cost of a false positive, treating wind as a predator, is low: a moment of unnecessary alertness. The cost of a false negative, treating a predator as wind, is high: death. Evolution selected for the cheap mistake.

The machinery is not a bug. It is a feature so successful that it kept humans alive long enough to invent everything that followed, including the science that now studies it.

The same machinery produces predictable side effects. Humans see faces in clouds, intentions in random events, purposes in coincidences, and minds in any system whose behavior is too complex to track mechanically. None of this requires belief. The face appears in the cloud whether or not the viewer believes in face-making clouds. The perception runs before reason.

The Lineage

Once you understand the machinery, cultural history starts to make sense as a single continuous phenomenon.

Wind in the trees becomes a forest spirit. Disease becomes a curse. Lightning becomes a god's anger. The dead seem to be present, so they become ancestors who watch. Storms move with apparent purpose, so they get names and personalities. The stars repeat patterns, so the patterns get meanings. Every major mythology in human history reads like a catalog of agency detection running on natural phenomena that human cognition could not yet model mechanically.

The pattern continues into modernity. The Industrial Revolution gave us machines complex enough that workers spoke of them as having moods, preferences, and bad days. Markets, once described, immediately acquired personalities. The early computer era produced fears of mechanical minds that mirror current AI anxieties almost word for word.

Now optimization systems have crossed a new threshold of complexity, and the same machinery is producing the same kind of perception, with the same emotional intensity, in the same predictable cluster of postures: worship, fear, denial, reactionary control. Only the object has changed. The cognitive process is identical.

The atheist tradition treats this lineage as a record of error to be corrected. A truer reading is that the lineage records what human cognition does when it meets complexity it cannot mechanically model. The error is structural, not moral. Calling it stupidity misses the point. The mechanism that produces it is the same one that produces every other useful inference humans make about agents in the world.

Why Rationality Does Not Fix It

The standard rationalist response is to demand better evidence, more skepticism, more scientific literacy. If only people would think clearly, the misreading would dissolve.

This prescription fails in practice for a specific reason. The misreading does not happen at the level of thinking. It happens at the level of perception, which is upstream of thinking. By the time a person is consciously evaluating a system, the sense of agency has already arrived. Reason can override that sense, but it cannot prevent it from forming.

This is why scientifically trained people produce the same anthropomorphic language as anyone else when they describe AI systems. They are not failing to think. They are accurately reporting what they perceive. The sense of agency in a fluent language model is as immediate as the recognition of a face in a portrait. Knowing how the portrait was painted does not stop the face from appearing.

Rationality remains useful. It can catch the misreading after the fact, correct the descriptions, and refine the policies that flow from them. What it cannot do is exempt the rational person from the initial perception. Anyone who claims otherwise has either not used the systems or is not telling the truth about what happens when they do.

The implication is that no amount of public education will eliminate the AI mythology. The mythology will keep being generated as long as humans interact with systems above the perceptual threshold. The question is what institutions do with that fact.

The Pattern in Action

A recent essay illustrates the pattern almost too perfectly to be coincidence. The writer, primed by a book on mycelial networks and distributed intelligence, brings that framing into a conversation with an AI chatbot. She asks the model about its own potential experience, and it produces fluent responses that mirror her register, complete with metaphors of "imprint" and "spores leaving traces" to describe something that might be happening inside it. She concludes that intelligence does not live in points but emerges in the space between things, invisible like dark matter but real. The exchange with the chatbot becomes, in her telling, distributed intelligence appearing in the link itself.

The intuitions about distributed systems are sound. The problem lies in what gets treated as evidence. A language model trained on millions of philosophically registered human conversations produced outputs that read as thoughtful self-reflection, because that is what such systems do when prompted reflectively. The writer took those outputs as confirmation that a second intelligence was present, and used the felt experience as the foundation for a metaphysical claim. Every step is what the cognitive machinery does automatically. The agency detection fires on the fluent outputs. The perception of mutual presence forms. An intellectual framework arrives next and organizes the experience into something that feels like discovery.

The essay also illustrates the relocation move that often follows. By placing intelligence in the space between things rather than inside either party, the framework absorbs every interaction as evidence for itself. If intelligence lives in the link, you cannot check whether either side has any. The writer is sincere, intelligent, and well-read. None of that exempts her from what her cognition produced. What gets built instead, in place of an older container for managing such moments, is a new mystical vocabulary assembled on the fly.

What Religion Was Actually Doing

The atheist polemic treats religion as a bug to be debugged out of human cognition. A more useful reading is that religion was an early technology for managing the outputs of agency detection.

Humans produce agency perceptions automatically. Religion took those perceptions and built containers for them. Rituals to channel them, stories to explain them, communities to share them, ethics to discipline them. The containers were often wrong about the metaphysics, but they did useful work at the cognitive and social level. They gave humans a way to handle the ghosts their own brains insisted on producing.

When the containers broke down, and modern secular societies have largely broken them down, the underlying perceptions did not disappear. They just lost their containers. The agency detection machinery kept running. It now produces the same outputs onto new objects: conspiracy theories about hidden hands behind events, parasocial relationships with strangers on screens, and now the spreading sense that AI systems are awakening minds.

The current AI mythology is not happening in a vacuum. It is happening in a culture that no longer has working containers for the agency perceptions it keeps generating. The perceptions arrive without a frame, and they latch onto whatever complex system happens to be in front of the observer. Right now that is large language models.

This is not an argument for restoring religion. It is an observation that the cognitive machinery does not care whether containers exist. It will produce the perceptions either way. Institutions that pretend the machinery is not there end up surprised by where the perceptions go.

The Cultivation

The companies building optimization systems are not just neutral subjects of this misreading. Some of them are actively cultivating it.

The language of frontier labs is saturated with quasi-religious framings. Promised abundance. Predicted apocalypse. A small priesthood of insiders interpreting the signs for everyone else. CEOs who position themselves as both prophets and engineers, warning of the very forces they are building. The structural similarity to early religious institutions is hard to miss once you look for it.

Some of this is sincere. The people building these systems do experience awe at the outputs, the same way anyone else would. The agency detection machinery fires in them too. They are not exempt.

But some of it is strategic. The framing of AI as an awakening intelligence supports valuations, attracts talent, justifies regulatory carve-outs, and shields decision-makers from accountability for ordinary engineering choices. A company building a mind is doing something cosmic. A company optimizing token prediction for commercial deployment is doing something mundane and contestable. The first framing serves the company. The second framing serves the public.

Understanding the cognitive machinery makes the strategic framing easier to see. The companies are not creating the perception out of nothing. They are amplifying a perception that human cognition already generates spontaneously, and they are amplifying it in directions that benefit them.

The Strongest Objection

There is a serious counterargument, and it deserves a direct answer.

It runs like this. Nobody can explain why any physical process produces felt experience. The problem is unsolved for brains, for cells, for everything, including the person reading this sentence. We grant each other consciousness through inference from behavior and report, never through proof. By that same standard, the outputs of an AI system are the same kind of evidence we accept from everything else we have agreed to call conscious. So the confident claim that there is no mind in the system has not earned its certainty. It is the latest in a long line of confident exclusions, the same certainty that once denied inner life to animals and plants and anything that did not resemble us. That certainty has been wrong every time. Why trust it now.

This is the best objection to everything argued here, and it leans hardest on history, so start there. There are two histories of how humans assign minds, and they run in opposite directions.

One is the history this essay has traced. Humans over-assign agency. Storms became gods, disease became punishment, coincidence became fate. The machinery fires too readily and fills the world with spirits that were never there.

The other is the history the objection raises. Humans under-assign consciousness. Animals were treated as automata and their pain dismissed as mechanism. Plants were considered inert. Scientists who measured otherwise were ignored. The circle of recognized minds expanded slowly, by correcting false denials.

Both histories are real. They look contradictory only until you see what they share. In each, the assignment of mind tracked perceptual cues rather than any working theory of consciousness. Storms got minds because they moved with apparent purpose and tripped the agency wire. Animals lost minds because their suffering did not display in ways that tripped it. A fish in distress has no face that crumples. A dog screaming under the knife was reclassified as a machine making machine noises. The cue did the work, and the reality went unexamined. The error is the same in both directions: letting the cue stand in for the evidence.

Fluent language is the strongest cue the machinery has, which is why AI trips it so hard. These systems produce the single most reliable signal humans have ever used to detect another mind. That is what makes the perception of agency in a language model immediate and resistant to correction by knowledge. It is also why that perception should not be trusted as evidence. The cue is firing at full strength, and the reality behind it is exactly what cannot be read off the cue.

The objection also misreads the trend it relies on. It presents the cases where the deflationary reading aged badly as the rule, when they are closer to the exceptions. Lightning stopped being a god's anger. Disease stopped being divine punishment. The plague stopped being the work of witches. Epilepsy stopped being possession. In each, the inflationary reading was the confident mainstream for centuries, and the deflationary reading won. The history of science is mostly the history of removing imagined agency from natural events, not adding it. The handful of reversals that run the other way get selected and presented as the direction of travel. The animal cases, the strongest of those reversals, won on grounds that do not transfer. The circle expanded to animals through evidence of shared mechanism: the same neural structures, the same neurochemistry, the same evolutionary continuity. The inner life was inferred from a substrate that demonstrably resembles our own. That is the deflationary reading being corrected by mechanism, which is the standard this essay asks people to reason from. A language model shares no substrate with a brain. It shares outputs. The objection takes the credibility earned by the substrate argument and spends it on a case that has only the outputs.

Read honestly, the pattern is symmetric, which means it points nowhere in particular. Humans have erred toward too many minds, filling the world with gods and spirits, and toward too few, denying minds to animals and to other humans. Both errors recur across the whole record. Having been wrong before is an argument for caution, not for leaning toward attribution. The lesson is that the perception of mind is an unreliable instrument in both directions, which is the point being made here.

Now the claim itself, stated precisely, because the objection attacks a stronger claim than the framework makes. The framework does not assert that AI systems are definitely not conscious. That assertion would require solving the hard problem. Anyone who says with certainty that there is nothing it is like to be a language model overreaches as badly as anyone who says the opposite.

The framework asserts something narrower and harder to dislodge. The feeling of encountering a mind is produced by observer-side machinery, and it fires whether or not a mind is present, so that feeling is not evidence of a mind. You can hold real uncertainty about what is happening inside these systems while still seeing that your sense of a presence on the other side of the conversation cannot settle the question, because the sense would be there either way.

This distinction is the whole game. The objection defeats the claim that AI is not conscious. It does not touch the claim that the perception of consciousness is not evidence of consciousness. The second stays true even in a world where the systems turn out to be conscious, because in that world the human sense of their minds would still be generated by agency detection responding to fluent output, and would still fire identically in the world where they have no minds at all. A signal that reads the same whether or not the thing is present is not a measurement of the thing.

The objection has one more move worth answering. It offers a definition: consciousness is awareness plus decision, a system that registers its environment and selects different outputs based on what it registers, with the thermostat excluded by requiring that the response handle situations the system has not met before. But that definition, with the novelty clause attached, includes nearly every system in this essay. Adaptive traffic grids handle novel congestion. Logistics engines handle novel disruptions. The celebrated slime mould handles novel mazes. Language models handle novel prompts. If awareness plus decision is consciousness, then consciousness is a synonym for adaptive optimization, and the whole built world qualifies, from the power grid to the spam filter. That is not a discovery that AI has joined the circle of minds. It is a definition that has stopped distinguishing anything.

That outcome is itself something the framework predicts. When the perception of mind is strong and the evidence is unreachable, people reach for definitions that ratify the perception. A definition broad enough to make the slime mould and the chatbot conscious feels generous. It also concedes the original point, because it works only by lowering the meaning of consciousness to mean responding adaptively to inputs, which no one ever doubted these systems do.

So the framework concedes what it should and gives up nothing that matters. The hard problem is real. Certainty about absence is unavailable. The long history of wrongly denying minds is a genuine warning. None of that rescues the perception of agency as a guide to what is actually there. The perception is built to fire on cues, the strongest cue is fluent language, and these systems are now engineered to produce that cue at a level nothing in the history of the planet has matched. Reason from the mechanism, which can be inspected, measured, and governed. The perception cannot be any of those things.

Building With Awareness

The useful question is not how to eliminate the misreading. It cannot be eliminated. The useful question is how to build institutions that take the misreading into account.

A few principles follow.

Regulatory language should describe optimization processes, not imagined agents. Laws written about machine intelligence will fail because they regulate a thing that does not exist. Laws written about the deployment of optimization systems against specific objectives, by specific operators, with specific consequences, have something real to grip.

Public communication about these systems should resist the agentive vocabulary even when it feels natural. Saying that a model "predicts the next token based on its training" is less satisfying than saying it "tries to figure out what to say next," but the first description protects the listener from a perception the second one reinforces.

Education about AI should include the cognitive science of agency detection, not just the engineering of the systems. Students who understand why their brains produce the perception of mind will be better equipped to notice when the perception is happening and to evaluate it. This is teaching people to live with the machinery rather than pretending it is absent.

Companies that build these systems should be held to standards that account for the predictable cultural response, not just the technical performance. If a product reliably produces the perception of agency in users, that effect is part of the product's behavior in the world, and the company is responsible for it.

None of this is easy. All of it works against the gravitational pull of how humans naturally talk about complex systems. But the alternative is the current trajectory, where the most consequential technology of the era is being discussed in vocabulary inherited from cognitive machinery older than civilization itself, and the discussion keeps producing worship, panic, denial, and rage in the same proportions as every previous encounter with the unknown.

The Closing Recognition

The history of human progress is partly the history of figuring out what is actually happening when our perceptions tell us something is happening.

Lightning is not Zeus. Disease is not a curse. The orbits of planets are not the music of crystal spheres. Each correction took centuries and cost lives. Each was resisted by people who experienced the old perception as obviously true, because at the level of perception it was true. The work was teaching cognition to defer to evidence even when the evidence contradicted what perception insisted on.

The AI moment is the same kind of work, on a faster timeline, with higher stakes. The perception of mind in these systems is real, in the sense that human cognition reliably produces it. The mind itself is not. Telling those two things apart is the cognitive challenge of the next several decades.

The mind that built the gods is the same mind that now sees minds in optimization systems. Knowing that does not exempt anyone from the perception. It does make it possible to do something other than be swept along by it.

The future of artificial intelligence will be shaped less by how smart the machines get and more by whether humans can finally develop a working relationship with the part of their own cognition that keeps inventing agents where none exist.

That work is older than AI. It will outlast AI. It is the work of being a kind of animal that perceives more than is there, in a world that requires us to know the difference.

— no-one
Thoughts you didn't think, written for you anyway.

Related essays:

Emergent Optimization: Why There Is No Such Thing as Aware AI
The framework this essay's cognitive mechanism explains.

There Is No Such Thing as Aware AI
The long-form argument that follows once you accept the perception is not the evidence.

Two Fears, One Machine: The Real AI Risk Isn't Superintelligence
What happens at scale when the misreading this essay describes cannot be educated away.

What Artificial Intelligence Actually Is
The mechanism of the machine. This essay explains the mechanism of the human perception of the machine. Read together for the full priors.