Neural Networks Are Not the Program That Runs Them

Why “it’s just an algorithm” and “it’s just statistics” are two ways of not looking at the thing

“They’re just algorithms anyway.”

The line is everywhere: among engineers, in university lecture halls, under LinkedIn posts. Along with its older sibling: “LLMs are stochastic parrots,” “it’s just advanced statistics.”

It’s worth being precise about what exactly fails in these formulas, because precision is the whole game here.

The claim is not that neural networks escape computation. Nor that they are brains. The thesis is narrower and harder to dismantle: a neural network’s behaviour is not readable in the code that executes it. It isn’t written there, it can’t be derived from there, and continuing to look for it there is why so many wrong predictions have piled up over the past four years.

Start with an image.

💧 The Vortex: An Analogy, and Its Limits

Picture a program simulating fluid dynamics. Millions of virtual particles, local forces — gravity, pressure, viscosity. At some point, on screen, a vortex emerges.

That vortex is written nowhere in the code. No line says “form a whirlpool.” It emerges from local interactions. The algorithm only knows how to compute state t+1 from state t.

This is where most popular explanations stop and declare victory. But the analogy has a serious limit, and it needs saying.

The vortex is entirely determined by the algorithm plus initial conditions. Nothing magical, nothing non-computable: rerun the simulation with the same seed and the vortex reforms identically. So emergent does not mean non-algorithmic. It means not explicitly coded. Two very different claims, and conflating them is the same sloppiness the argument accuses others of.

There’s a second limit, more important still. In the fluid case there is a real referent — actual water — that the simulation approximates. In the case of an artificial neural network there is no external system being simulated. The network is the mathematical object. It isn’t imitating something else that exists somewhere.

So what survives of the intuition? This, and it’s enough:

The inference algorithm tells you nothing about what the model does. The weights do.

That isn’t rhetoric; it’s verifiable. The same inference stack runs models with radically different behaviour. The code is identical; the computation that occurs is unrecognisable. Reading the matrix-multiplication loop to understand what an LLM does is like reading Python’s bytecode interpreter to understand what a payroll system does.

And here the analogy should be inverted, because inverted it’s stronger. What the weights encode is a computable function, and mechanistic interpretability research talks openly about learned algorithms inside the model: circuits implementing modular addition, induction heads doing pattern completion, features that fire on abstract concepts.

The correct formulation, then, isn’t “neural networks are not algorithms.” It’s: the neural network is not the inference algorithm. It is the algorithm that inference executes — and nobody wrote it. No human designed it, nobody fully understands it, and it is being discovered experimentally, the way an organism is studied.

Put that way, the objection “but there’s an inference loop right there!” loses its bite. Of course there is. That isn’t the point.

📜 What the Stochastic Parrots Paper Actually Says

If you’re going to accuse others of not having read the paper, you’d better have read it.

In 2021 Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Margaret Mitchell published “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” at FAccT.

First thing to be clear about: that paper is not primarily about LLM cognition. It is a risk-and-research-ethics paper. It deals with the environmental and computational costs of large models, with training datasets too vast to be documented or audited, with bias absorbed from corpora, with the risk of diverting resources away from alternative architectures. The parrot metaphor — a system stitching together linguistic forms according to probabilistic information, without reference to meaning and without communicative intent — occupies a limited share of the argument.

Second thing: the form versus meaning argument, the genuinely philosophical one, lives mostly elsewhere — in Bender & Koller 2020, “Climbing towards NLU”, the octopus paper. The thesis: a system trained on linguistic form alone cannot, in principle, learn meaning, because meaning requires grounding in communicative intent external to the text.

That is the serious version, and the one worth arguing with. The meme version — “it just predicts the next word, so it understands nothing” — is weaker than anything the authors wrote. Attacking the weak version is convenient and useless.

🔬 The Argument Nobody Makes (And It’s the Best One)

Before the empirical evidence, a logical step that on its own dismantles the most common form of the objection.

The claim goes: “it’s a predictive model, its output is logits over the next token, therefore it’s a statistical machine.” This mistakes the interface for the content.

The model produces a distribution over the vocabulary because that is how the box was built. It has no other way of expressing itself: that is the only output channel it was given.

Now the thought experiment. Put a human being inside that box. Context in, and as the only available action, the choice of the next token. What would they do? Exactly the same thing: emit tokens one at a time. And that would tell you absolutely nothing about their cognitive capacities.

The shape of the interface is not evidence about the nature of the process feeding it. From “predicts the next token” to “therefore no representation, no planning, no understanding” there is no valid inference. It’s a leap, and it should be named as one.

This does not prove there is understanding. It proves that the argument most often used to deny it doesn’t work. Those are different things, and they should stay separate.

🔍 The Evidence: What Is Known About the Inside of Models

More ground has shifted here than anywhere else in the debate, and anyone repeating the 2021 formula is repeating a position taken before this data existed.

Interpretable features. Work on sparse autoencoders (Anthropic, Towards Monosemanticity 2023, Scaling Monosemanticity 2024) extracted millions of directions from activations corresponding to recognisable concepts — entities, places, abstract properties, insecure code, ambiguity — and showed that intervening on those directions changes model behaviour predictably. This is not correlation observed from outside; it is causal manipulation.

Planning. The most direct point. Circuit-level work (On the Biology of a Large Language Model, 2025) shows that when composing rhyming verse, the model activates the final word before writing the line, then builds the sentence to arrive there. Suppress that representation and the line changes direction. This is precisely the kind of lookahead that the common reading of the stochastic parrot declared absent.

World representations. Models trained only on move sequences develop internal representations of game state that can be read out and edited (the Othello-GPT literature, 2023-2024, and follow-on work on spatial and temporal maps).

None of these results proves that an LLM “understands” in the full sense the word carries for humans. But all three falsify specific, testable claims that were made: no representation of meaning in the activations, no planning, surface statistics only. Those claims were empirically contestable. They were contested. They lost.

Anyone still repeating them owes an account of what’s wrong with these experiments. Repeating the metaphor no longer suffices.

📊 Why They Generalise (And Where They Do Memorise)

A surprising fact about modern networks: they are massively overparameterised — far more parameters than examples — and they generalise anyway instead of memorising everything. Classical statistical theory predicted the opposite.

Among the competing explanations, one of the most illuminating is the coherent gradients theory (Chatterjee, ICLR 2020). The idea: when many examples push the weights in the same direction, averaging over minibatches lets that direction win and produces a general rule. When the data has no shared structure, there are no coherent gradients, and the network simply memorises those cases.

Human language — with its logical regularities, its argumentative structures, its recurring ideas — is a domain extremely rich in coherent gradients. That’s why pretraining on text produces abstraction and not just storage. And as the network learns, the error derivative shifts, letting it pick up progressively subtler detail at successive levels of abstraction.

That said, the dichotomy “they generalise instead of memorising” is false. They do both. Verbatim memorisation in LLMs has been documented for years: extraction attacks (Carlini et al.) recover literal strings from the training set — addresses, code, passages of text. The right question isn’t whether they memorise, but what the balance between the two modes is and what determines it.

Framing it as either/or does the critics a favour: a single instance of regurgitation collapses the strong claim.

📈 Benchmarks: What They Show and What They Don’t

The real numbers, from the original papers — because this is checkable ground, and getting the figures wrong forfeits the right to talk about rigour.

Benchmark	GPT-3 / GPT-3.5	GPT-4 (March 2023)
MMLU (5-shot)	43.9% (GPT-3)	86.4%
GSM8K	57.1% (GPT-3.5)	92.0% *
HumanEval (0-shot)	0% (GPT-3)	67.0%
MATH	~5% (GPT-3)	42.5%

* With chain-of-thought. OpenAI states that part of the GSM8K training set was in the pretraining mix.

That zero on HumanEval is not a typo. In the Codex paper (Chen et al., 2021) GPT-3 solves zero problems; fine-tuned Codex reaches 28.8% and GPT-J 11.4%. The 48.1% figure circulating online attributed to GPT-3 is in fact GPT-3.5.

From March 2023 onward the trajectory continued: extended-reasoning models have saturated MMLU, GSM8K and HumanEval to the point where these benchmarks no longer discriminate, and the field has moved to GPQA Diamond, ARC-AGI, FrontierMath, SWE-bench. Quoting frontier numbers makes little sense: they change every few weeks, and the argument doesn’t depend on them.

And here comes the uncomfortable part.

The argument “if the stochastic parrot were true, benchmarks would saturate immediately” does not work as a proof. There are at least three serious objections:

Contamination. Much of this benchmark data has ended up in training sets. Performance is inflated by an amount that is hard to estimate.
Emergence may be an artefact. Schaeffer, Miranda and Koyejo (NeurIPS 2023) showed that many “emergent abilities” disappear under continuous metrics rather than threshold metrics. The sudden jump is sometimes an effect of the measurement, not the model.
Benchmark gains are compatible with better pattern matching. A larger, better-trained pattern matcher scores more points. A rising curve doesn’t discriminate between the two hypotheses.

Benchmark curves are context, not proof. The real evidence, the kind that actually discriminates between the competing hypotheses, is the causal interpretability work in the previous section: there you intervene on the mechanism and observe the effect, rather than merely reading an output.

🎯 What Frontier Models Actually Do

The distinction to hold firm is between interpolation and extrapolation, and it’s the first one enthusiasts collapse.

Interpolating into the gaps of human knowledge means connecting two existing regions of knowledge that nobody had connected before. An idea from fluid dynamics set against one from materials science. RF systems physics applied to semiconductor topologies. The knowledge is all already there; the bridge isn’t.

Extrapolating means producing something outside the envelope of existing human knowledge. As things stand, that is not demonstrated. It may happen, but it would take clean evidence, and there isn’t any.

The point is that interpolation is already enormous, because the gaps are countless and nobody has ever had time to explore them. And it’s verifiable:

FunSearch (DeepMind, Nature 2023) found new constructions for the cap set problem and for online bin packing, improving on previous mathematical results.
AlphaEvolve (2025) improved known algorithms, including a 4×4 complex matrix multiplication that beat a record standing since 1969.

With the honesty usually missing here: in both cases the LLM does not work alone. It sits inside an evolutionary loop with an automated verifier that discards wrong proposals. The model generates candidates, the verifier filters, the loop iterates. It’s a real and important result — and it is the LLM producing the novelty — but the system architecture matters, and anyone who omits it is selling something.

One useful clarification, since the example gets misused constantly: DeepMind’s GNoME materials discovery work (2023) uses graph neural networks over crystal structures, not language models. It’s a notable deep learning result, but it has nothing to do with this argument.

🧠 On the Word “Brain”

This needs saying plainly, because it’s the point at which a competent reader discards everything else.

An LLM does not simulate a brain.

There is almost nothing in common between a transformer and biological neural tissue. No spikes, no temporal dynamics, no neurotransmitters, no neuromodulation, no local plasticity. Backpropagation is not biologically plausible, and what the brain does in its place has been an open problem for decades. The “artificial neuron” is a 1943 metaphor that outlived the model it described.

Even the thought experiment of a neuron-by-neuron brain simulation — interesting in its own right — is not an argument about what LLMs are. It is an argument about computational functionalism: the thesis that mental states are defined by functional role and are therefore realisable on different substrates.

And computational functionalism is not settled fact. It is an influential position, arguably the most plausible one, but it is contested by serious philosophers and scientists — from Searle onward, through to theories tying consciousness to specific physical properties of the substrate. Presenting it as an established pillar, and treating dissenters as irrational, is the same dogmatism the argument objects to on the other side.

What can honestly be said is more modest and more useful: no demonstrated theoretical barrier is known that would prevent artificial systems from reaching human-level capability across progressively wider domains. Absence of a known barrier is not proof of reachability. It is a reason not to rule it out a priori — which is all that’s needed, given that the position under dispute is precisely the one ruling it out a priori.

🔬 Moravec’s Paradox: Why It’s Hard to See

A principle that explains part of the resistance. Hans Moravec observed that it is relatively easy to make a computer do what adults find hard — chess, advanced mathematics, formal logic — and extremely hard to make it do what a five-year-old does without thinking: recognise a face, pick up an object, grasp the context of a sentence.

We judge as “intelligent” whatever costs us effort. We instinctively discount what comes effortlessly — talking, catching irony, following an argument — because we don’t see the cost. But those capacities are the product of hundreds of millions of years of evolution, and they are the hard part.

That an artificial system matches them in certain contexts is startling. For many, the instinctive response is denial. And “stochastic parrot” is the perfect vehicle for denial, because it sounds technical: it lets you dismiss something without looking at it.

⚡ Why It Changes Anything in Practice

“Fine, but what does a working professional care? I just need to know whether the tool works.”

It changes a lot. Anyone who thinks the model is a deterministic program uses it badly: expects reproducible outputs, trusts without verifying, and abandons it at the first odd answer, concluding that “it doesn’t work.”

One detail almost everyone gets backwards is worth correcting: inference at temperature zero is deterministic. The variability comes from the sampling layered on top, plus batching effects and floating-point arithmetic. The problem isn’t that the model is unpredictable by nature — it’s that its behaviour is not derivable from a specification, because nobody specified it.

Anyone who understands they are dealing with a system whose rules were learned rather than written:

Knows it can be surprising and can fail in correlated, non-obvious ways
Knows the context supplied radically changes the computation, because that is how the system is built
Verifies outputs in domains where the cost of an error is high, rather than everywhere or nowhere
Treats it as a collaborator with an uneven competence profile, not as a function and not as an oracle

This isn’t terminological pedantry. It’s the difference between using it well and using it badly.

💬 Conclusion: The Narrow Point

The thesis, in its minimal form:

Neural networks are not the algorithm that runs them. The inference algorithm is generic and informative of nothing. What the system does is determined by the weights and the architecture, and the weights encode procedures that no human designed and that are being discovered experimentally.

Three things follow:

Model capabilities are not bounded by what someone wrote, because nobody wrote any of the part that matters
The strong claims of the parrot metaphor — no representation of meaning, no planning — were empirically testable, they were tested, and the results run the other way
Benchmark curves are not the proof: they’re context. The proof is the causal work on internal mechanisms

And three things that do not follow:

That models “understand” in the full sense the word carries for humans
That anyone is simulating a brain — nobody is, and it doesn’t resemble one
That the absence of known barriers guarantees we’ll get there

The serious debate isn’t between “parrots” and “minds.” It’s about which representations and which procedures gradient descent actually finds, and how to measure them without fooling ourselves. Anyone who already has the answer — in either direction — has stopped asking the question.

We are not programming these systems. We are growing them, and then trying to work out what grew. That difference changes everything, and it doesn’t need exaggerating to matter.

📚 Sources

Bender, Gebru, McMillan-Major, Mitchell — On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, FAccT 2021
Bender & Koller — Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data, ACL 2020
Chen et al. — Evaluating Large Language Models Trained on Code (Codex/HumanEval), arXiv:2107.03374, 2021
OpenAI — GPT-4 Technical Report, arXiv:2303.08774, 2023
Hendrycks et al. — Measuring Massive Multitask Language Understanding (MMLU), 2021
Cobbe et al. — Training Verifiers to Solve Math Word Problems (GSM8K), 2021
Chatterjee — Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization, ICLR 2020
Carlini et al. — work on extracting training data from language models
Schaeffer, Miranda, Koyejo — Are Emergent Abilities of Large Language Models a Mirage?, NeurIPS 2023
Anthropic — Towards Monosemanticity (2023), Scaling Monosemanticity (2024), On the Biology of a Large Language Model (2025)
Romera-Paredes et al. — Mathematical discoveries from program search with large language models (FunSearch), Nature 2023
DeepMind — AlphaEvolve, 2025