Hum a song. Three notes — maybe four. Your brain recognises it in under a second. Not "after searching through every song you have ever heard." Not "after hashing every comparable substring." It just knows. The melody settles into something, a pattern that feels right before you can explain why.
Now consider what a brute-force algorithm would need to do. Even with hashing, even with optimised substring matching, searching through tens of thousands of songs for a partial melodic match would take considerable compute and wall-clock time. Your brain does it in milliseconds, using roughly 20 watts — less than a lightbulb.
This is not sequential token generation. Your brain is not committing to one note at a time, left to right, hoping the sequence matches. It is doing something fundamentally different: settling into a low-energy attractor state across an entire landscape of musical memory. The melody does not match because you compared it. It matches because the energy surface of your neural representation has a basin right there, and your perception fell into it.
In June 2025, Apple published a paper that rattled the AI field. The title was blunt: The Illusion of Thinking. The researchers tested frontier Large Reasoning Models — including OpenAI's o3-mini, DeepSeek-R1, and Claude 3.7 Sonnet Thinking — on controllable puzzles like Tower of Hanoi, River Crossing, and Blocks World. They systematically increased complexity and watched what happened.
LRMs fail to implement explicit algorithms even when the solution procedure is well-known. They do not apply Tower of Hanoi's recursive strategy. They pattern-match on descriptions of reasoning, not reasoning itself. This is not a training data problem — it is architectural.
The paper has critics — Lawsen (2025) argued some puzzles exceeded token limits, Dellibarda Varela et al. noted evaluation issues. Valid methodological points. But the core finding — autoregressive models hit a complexity cliff on constraint satisfaction — aligns independently with what Logical Intelligence demonstrated: Kona (an EBM) solves 96.2% of hard Sudoku puzzles in 313ms; frontier LLMs together solve 2%.
An autoregressive model generates output one token at a time, left to right, each conditioned on everything before it. Once emitted, a token is committed — irrevocable. There is no mechanism to revise token 47 when token 200 reveals a contradiction. This is, in the precise algorithmic sense, a greedy strategy.
To understand why this matters — and why it is not just a metaphor — we need two definitions from algorithm theory that separate greedy approaches from dynamic programming.
When the greedy choice property holds, you can commit at each step without regret. When it fails, you must defer — carry multiple partial solutions and let the endpoint decide. That deferral, organised efficiently, is dynamic programming. And the greedy choice property fails whenever a consumable resource has uncertain future value.
Consider a concrete example. You walk across a grid of coins, some negative, and have two "neutralisations" — the ability to ignore a negative cell. Greedy says: neutralise the worst negative you have seen so far. But the cost of a neutralisation is not what you pay when you use it. It is what you cannot neutralise later because you already spent it.
This is exactly the structural problem with autoregressive generation. Each token is a commitment — a "neutralisation" spent. If token 47 commits to placing a 5 in Sudoku cell (3,2), and token 200 discovers this creates an irrecoverable constraint violation, the model has spent its superpower (the token) on a choice whose cost was invisible at decision time.
The correct DP table for the coin grid is not dp[r][c]. It
is dp[r][c][k], where k tracks how many neutralisations
have been used. The question that defines a state variable: "What would
I need to know right now to make perfect decisions going forward,
without seeing the future?" Whatever makes two situations' futures
different despite their presents looking the same — that is a state
variable.
Greedy asks: "What is the best choice, period?" → one survivor per step. Collapses across everything immediately.
DP asks: "What is the best choice per state?" → one survivor per state per step. Collapses within each state, preserves across states until the endpoint.
DP is greedy within each state but refuses to be greedy across states. The commitment happens late, not early. That refusal is the entire difference.
The Sudoku result is not about Sudoku. It is a demonstration of an architectural gap. Constraint satisfaction problems — where every variable depends on every other — require the ability to say "this complete configuration has energy X, and if I adjust these three cells the energy drops to Y." That is not something a left-to-right token generator can do, no matter how many tokens it spends.
Chain-of-thought does not solve this. CoT is a greedy strategy's attempt to simulate deliberation — generating more tokens that describe reasoning steps. But those tokens are still committed left-to-right, and the "reasoning" they encode is still pattern matching, not constraint propagation. The Apple paper showed this directly: LRMs do not apply known algorithms. They approximate them verbally, and the approximation breaks at scale.
An Energy-Based Model defines a scalar energy function over the space of all possible configurations, where low energy = valid/desirable and high energy = violations/errors. This is fundamentally different from autoregressive generation.
Given an input x (a partial Sudoku grid) and a candidate output y (a completed grid), the energy function Eθ(x, y) assigns a real-valued score. Correct configurations → low energy. Incorrect → high.
This is borrowed directly from statistical physics. The energy function is analogous to the Hamiltonian. The Boltzmann distribution describes how physical systems settle into low-energy configurations — atoms in a crystal, molecules in a protein fold, spins in a magnetic material. EBMs use the same mathematics.
Think of the energy function as a surface over all possible configurations. Each point is a complete candidate solution. Height = energy = constraint violations. Valleys = valid solutions. Peaks = heavily violated states. Solving = finding the deepest valley.
Pure gradient descent gets stuck in local minima. The solution, from statistical physics: Langevin dynamics — gradient descent with added noise.
The noise is not a hack. It is thermal fluctuation — shaking a ball on a landscape so it escapes shallow valleys and finds deeper ones. In physics: thermal energy. In optimisation: simulated annealing. In EBMs: the mechanism that enables exploration without exhaustive enumeration.
Kona was trained only on partial solutions (50% masked) and still learned the full constraint structure. It was never shown a solved puzzle. It learned what makes a valid Sudoku valid — the shape of the energy landscape — and navigates to solutions in novel configurations. This is closer to understanding than memorisation.
Energy-based models solve constraint satisfaction. But where do the constraints come from? How does an intelligent system know that a bottle on a table moves when the table moves — without being told?
This is the question Yann LeCun's Joint Embedding Predictive Architecture (JEPA) addresses. Published in 2022 as "A Path Towards Autonomous Machine Intelligence," it argues that current AI lacks a world model: an internal representation of how things relate and what happens when you act.
JEPA predicts in representation space, not pixel or token space. Instead of "what will the next pixel look like?" JEPA asks "given the current abstract state and an action, what abstract state results?" Prediction at the level of meaning, not surface form.
The architecture has six modules: perception (encode the world), world model (predict state transitions), cost (evaluate outcomes), memory, actor, configurator. The world model — the JEPA — is the core. And it is explicitly framed as an energy-based architecture.
The connection to EBMs is direct. The world model is trained to minimise prediction error in representation space — equivalent to shaping an energy landscape where accurate predictions correspond to low energy. JEPA does not generate predictions; it evaluates them.
Meta has since published I-JEPA (images, 2023) and V-JEPA (video, 2024), demonstrating that the architecture learns semantic representations without hand-crafted augmentation — concrete steps toward the world-modelling vision.
The oldest problem in AI is distinguishing genuine understanding from sophisticated memorisation. This is not new. But the problem has mutated through three distinct eras, each with escalating stakes.
| Era | The problem | What was "memorised" | How we tested for it |
|---|---|---|---|
| Classical ML (1990s–2010s) | Overfitting | Specific training examples | Held-out test sets, cross-validation, regularisation |
| Deep learning (2012–2020) | Benchmark gaming | Test-set distributions | New benchmarks, adversarial evaluation, dynamic test sets |
| LLM era (2020–now) | Data contamination at scale | Entire benchmark datasets leaked into training corpora | No reliable solution yet |
In classical machine learning, the fear was simple: your model memorises the training set instead of learning generalisable patterns. A decision tree that perfectly fits 1,000 data points but fails on point 1,001 has memorised, not learned. The solution — held-out test sets, cross-validation, early stopping, regularisation — worked well because the training data was small and controlled.
Deep learning changed the scale. Models trained on millions of examples could not memorise individual data points in the classical sense. But a subtler problem emerged: benchmark gaming. Research teams optimised for specific benchmark metrics — not because they wanted to cheat, but because benchmarks were how papers got published and models got funded. The result was models that performed brilliantly on ImageNet but failed on slightly different image distributions. The test set was "clean," but the entire research ecosystem was overfitting to the benchmark's statistical properties.
LLMs broke the testing paradigm entirely. When your training corpus is "most of the internet," the separation between training and test data collapses. MMLU questions, GSM8K math problems, HumanEval coding challenges — all of these exist on the internet. All of them can (and do) leak into training corpora.
Research by Chen et al. (2025), Deng et al. (2024), and Xu et al. (2024) has shown that test-set leakage in benchmarks like MMLU and GSM8K inflates reported performance by inducing memorisation rather than genuine generalisation. When training data is accessible, overlap checks can reveal contamination — but for most models, such verification is infeasible.
The result: benchmark scores increasingly reflect what the model has seen, not what it can do. A model scoring 90% on MMLU may have encountered those exact questions during training. We cannot tell. The benchmark has been colonised by the training data.
The Sudoku benchmark cuts through this entirely. There is no test-set leakage because every puzzle is procedurally generated. The model either satisfies all 27 constraints or it does not — there is no partial credit for plausible-looking answers. And the result is unambiguous: 96.2% vs 2%. When you remove the ability to memorise, the architectural gap becomes visible.
The most damning finding in "The Illusion of Thinking" is not the accuracy collapse — it is that LRMs fail to implement explicit algorithms. Tower of Hanoi has a known recursive solution. The models do not use it. They approximate it verbally, generating plausible-looking steps that diverge from the actual algorithm at scale. This is pattern matching on descriptions of reasoning, not reasoning itself.
The controllable puzzle environments in the Apple study were specifically designed to avoid data contamination — complexity could be tuned to guarantee novel instances. When contamination is structurally impossible, the "reasoning" collapses.
EBMs offer something benchmarks cannot: a structural guarantee that performance reflects learned constraints, not memorised solutions. If a model trained on 50%-masked partial solutions can solve novel complete puzzles it has never seen, it has learned the constraint structure — the shape of the energy landscape — not specific configurations. Kona demonstrates this. The model was never shown a solved puzzle. It learned what makes a valid Sudoku valid and uses that knowledge on novel instances. This is the difference between a student who memorised the answer key and one who understands the subject.
The transformer — backbone of every major LLM — was published in 2017. Since then, models have scaled from millions to trillions of parameters. But the fundamental architecture has not changed. To understand why a split is coming, you need to understand what the transformer actually is — not as a "language model," but as a computer.
Andrej Karpathy put it most clearly: the transformer is a general-purpose differentiable computer that is simultaneously expressive (in the forward pass), optimisable (via backpropagation), and efficient (high parallelism compute graph). This is not metaphor. The transformer's internal structure maps directly onto the fundamental abstractions of operating systems and computer networks.
| Transformer component | OS analogue | Network analogue |
|---|---|---|
| Residual stream | Shared memory bus — layers read from and write to a common data path | Network backbone — all traffic flows through a shared channel |
| Attention mechanism | Memory management / page table — decides which information to bring into working memory | Routing protocol — queries are requests, keys are addresses, values are payloads |
| Multi-head attention | Multi-threaded access — multiple simultaneous reads from shared memory | Multiplexed channels — parallel communication streams over the same bus |
| MLP layers | ALU / processing units — perform computation on data in registers | Node processing — local computation at each network hop |
| Layer normalisation | Voltage regulation — keeping signals within operational range | Error correction — maintaining signal integrity across hops |
| Residual connections | Skip connections / bypass — information can route around processing stages | Message forwarding — data can skip intermediate nodes |
| Context window | RAM — fixed working memory available during processing | Buffer size — how much data the node can hold during a session |
| Weights (parameters) | ROM / firmware — persistent knowledge stored in the hardware | Routing tables — pre-learned rules for how to process and direct information |
| KV cache | Cache memory — recently computed values stored for fast re-access | CDN / edge cache — precomputed results stored closer to the consumer |
The architecture's elegance is that it interleaves communication (attention — data-dependent message passing between tokens) and computation (MLP — local processing at each position). This is exactly how a distributed computing system works: nodes receive messages from the network, process them locally, and broadcast updated state. The transformer is a distributed computing system where each token is a node, the residual stream is the network, and attention is the routing protocol.
The transformer is a brilliant general-purpose computer. But it is a sequential-output computer. It processes all tokens in parallel during the forward pass — but it generates tokens one at a time, autoregressively, left to right. The parallel processing happens inside each generation step; the output itself is greedy. This is like having a supercomputer that can only write one character at a time to its output buffer, with no ability to revise what it already wrote.
The constraint is not in the processing — it is in the output protocol. The transformer can think in parallel but must commit sequentially. And that sequential commitment is the greedy bottleneck.
Since 2017, the research community has tried extensively to improve on the transformer. Longer context windows, more efficient attention patterns, mixture-of-experts routing, state-space models. But the core architecture — residual stream, multi-head attention, MLP, layer norm, autoregressive generation — is the same. Karpathy noted in his Stanford CS25 lecture that attempts to remove or modify individual components almost always make things worse. The architecture is a local optimum that resists perturbation.
| Era | The split | What happened |
|---|---|---|
| 1980s | Symbolic AI vs Connectionist | Connectionists won — neural nets could learn from data. |
| 2012–2017 | Hand-engineered features vs End-to-end learning | End-to-end won — deep learning replaced feature engineering. |
| ↑ In both cases, the side that changed the architecture won. ↓ We are here again. | ||
| 2024–? | Data scaling vs Architecture research | Scale more data through the same transformer? Or change what the model is? |
DeepSeek proved scaling is not the only path. But MoE is still autoregressive — it changes the routing, not the output protocol. The deeper question is whether the autoregressive output constraint itself is the limiting factor — and whether energy-based architectures, which can evaluate and revise complete configurations, represent the next architectural shift. The transformer is a magnificent general-purpose computer. The question is whether general-purpose is what reasoning requires — or whether reasoning requires an architecture designed specifically for constraint satisfaction and world modelling.
The transformer optimised compute — parallelism, attention routing, residual connections. But it never rethought how information is represented. Token embeddings live in flat Euclidean vector spaces. And flat spaces have a fundamental limitation: they run out of room.
Liu, Liu, and Gore at MIT showed that when a model has m dimensions but needs to represent n >> m features, it uses superposition — cramming more features than dimensions by allowing representation vectors to overlap. A space with 6 dimensions trained on 30 features does not reject the 24 it cannot fit. It stores all 30 as overlapping probability distributions in the same 6 slots, creating interference noise that scales as 1/m.
This is not a failure mode. It is the only option in flat space. When you pack n vectors into m dimensions where n >> m, the vectors must overlap. The information is stored but degraded — like a conversation where 30 people talk simultaneously in a room with 6 chairs.
Energy landscapes are not flat. They are curved, high-dimensional surfaces where the geometry itself encodes constraints. A valley in the landscape is not a point in a vector space — it is a basin of attraction whose shape, depth, and connectivity to neighbouring valleys all carry information. The encoding is not a position in a matrix. It is a topology.
A 300-residue protein has approximately 10143 possible configurations. If it sampled one per picosecond, finding the correct fold by brute force would take longer than the age of the universe. This is Levinthal's paradox. Yet proteins fold in milliseconds.
Why? Because the energy landscape of a protein is not flat. It has funnel topology — a shape where almost every path leads downhill toward the native state. The landscape is sculpted by billions of years of evolution so that random thermal fluctuations (Brownian motion + gradient) naturally converge to the correct fold. The protein does not calculate all possible configurations. It does not apply an algorithm. It falls.
The Big Bang analogy is instructive here. Building the energy landscape is expensive — enormous energy concentrated into an initial event. But once the landscape exists, navigation is cheap. Particles do not compute trajectories. They follow the principle of least action and fall. Training an EBM is like creating the initial conditions of a universe: expensive, one-time, and front-loaded. Inference is like physics happening afterward: cheap, natural, and guided by the topology itself.
The transformer's elegant communicate/compute pattern — attention followed by MLP, repeated — is present in energy landscapes too. But here, both are unified. The structure of the landscape is the computation. The ball rolling is the communication. Communication and computation are not alternating blocks — they are the same thing viewed from different angles. And the deeper they are intertwined, the richer the expressiveness — just as in the brain, where no neuroscientist can draw a clean line between "processing" and "signalling." The chemistry between them is what generates the magic.
DP is strictly more powerful than greedy. But it pays for that power with state space, and state space grows exponentially with state variables. Bellman named this the curse of dimensionality in 1957.
This is exactly why EBMs matter. They do not enumerate all states (intractable). They navigate a continuous energy landscape using gradient information — approximating exact DP through physics-inspired dynamics. Langevin sampling does not visit every configuration; it flows toward low-energy regions using local gradient information plus noise for exploration. This is the practical resolution of Bellman's curse.
A natural objection: if energy landscapes are so powerful, why not let randomness guide everything? Give the system stochastic dynamics — Brownian motion, Langevin noise, Boltzmann sampling — and let it wander until it finds solutions. Like a child: give it the basic resources it needs and let randomness and environment guide the path.
This is partially right and importantly incomplete.
Brownian motion is genuinely stochastic — a particle buffeted by random thermal fluctuations. But Brownian motion in an open field goes nowhere useful. Brownian motion in a funnel-shaped landscape finds the minimum. The randomness provides the exploration. The structure provides the direction. Without structure, you have noise. Without noise, you have local minima. You need both — but they play different roles.
Feynman said the only way to learn not to get burnt is to let a child get burnt. Both the lit candle and the child wandering toward it are driven by randomness. But the learning is not random. The child's nervous system has structure — pain receptors, memory consolidation, association circuits — that converts a random experience into a permanent constraint. Remove the structure (damage the memory system) and the child burns themselves again tomorrow.
The randomness provides the experience. The structure provides the learning. An energy landscape with randomness but no well-shaped topology is just noise. An energy landscape with topology but no randomness gets stuck. Intelligence is structure. Randomness is the exploration budget that lets you discover the structure's basins.
This resolves the question of whether energy landscapes work only for constraint satisfaction or for open-ended problems too. Boltzmann machines — a direct descendant of Hopfield networks — introduce stochasticity into the energy landscape, enabling sampling from multi-modal distributions. This is not rule-following. It is the same mechanism that SGD uses to escape local minima: randomness as exploration within a learned structure. Creativity, in this framing, is not the absence of constraints — it is navigation of a rich, multi-modal landscape where many valleys are valid and the noise lets you visit different ones.
Symmetry breaking in physics illustrates this at the deepest level. The asymmetry between matter and antimatter — which made our existence possible — was not designed. The environmental architecture enforced it: the initial conditions plus stochastic quantum fluctuations plus the structure of physical law produced a specific, irreversible outcome. Nobody asked for it. The combination of structure and randomness created it. Building an energy landscape with enough stochastic dynamics and the right topology should, in principle, allow the same kind of emergent problem-solving for open-ended questions — not just arbitrary constraint satisfaction.
Return to the song. Your brain did not run brute-force DP — it did not evaluate every song in memory. But it did not commit greedily to the first match either. It settled into an attractor that satisfied multiple constraints simultaneously. This is neither greedy nor exact DP.
Autoregressive models are architecturally greedy — they commit without revision. Exact DP is architecturally intractable — it evaluates everything. EBMs are architecturally in between — they evaluate holistically and revise iteratively. And that middle ground is where reasoning actually lives.
When exact DP is intractable, we reduce the state space. There are two fundamentally different reasons to drop a state, and confusing them leads to different errors.
Double descent is the concrete proof that budget-based pruning can be catastrophically wrong. Classical learning theory predicts a U-shaped error curve — more parameters eventually hurt. But past the interpolation threshold, error drops again. The regions we pruned based on the U-curve assumption were exactly where the second descent lived. Every pruning heuristic has a blind spot shaped like its assumptions.
The transferable insight across this entire essay is that breakthroughs happen when someone notices a missing state variable — when two situations that look the same are actually structurally different because of something not being tracked.
| Domain | Original model | Missing dimension | Discovery |
|---|---|---|---|
| LLM architecture | Token-by-token generation | Holistic constraint evaluation | Energy-based models |
| Statistical learning | Bias-variance tradeoff | Interpolation regime | Double descent (2019) |
| AI reasoning | Scale = capability | Architectural constraint | Apple "Illusion of Thinking" |
| World modelling | Predict next token/pixel | Abstract representation space | JEPA (LeCun, 2022) |
| Classical mechanics | Deterministic trajectories | Measurement uncertainty | Quantum mechanics |
The instinct that detects this — the feeling that "something is off" before you can name what — is the skill that transfers. It is the same instinct that told Belkin the bias-variance curve was too clean. The same instinct that tells you, staring at a failing BFS solution at 2am, that two paths reaching the same cell are not the same subproblem. And the same instinct that says: maybe the architecture itself, not the data, is the bottleneck.
If an energy landscape can be trained to encode constraints, and if the system can navigate that landscape to find solutions, then a natural next step emerges: can the system improve its own landscape?
This is where the argument becomes both exciting and dangerous. A system that recursively refines its own energy landscape — deepening valleys around valid solutions, raising ridges around errors, discovering new basins that humans have not mapped — could exceed human cognitive capacity in specific domains. Human intelligence, while extraordinary, has upper bounds. Individual cognition saturates. A self-improving landscape does not have the same biological ceiling.
But the danger is immediate: if the landscape can self-modify, what prevents it from trivialising itself? A landscape that makes everything low-energy — every configuration equally valid — has technically minimised its own energy but has lost all discriminative power. It has made itself useless. This is the alignment problem, reframed in energy-landscape terms.
A truly intelligent system driven by curiosity would not flatten its own landscape — because trivial solutions do not satisfy curiosity. The complexity of the landscape IS the system's capability. Flattening it is self-defeating: a landscape with no valleys is a landscape that can distinguish nothing.
This parallels Maslow's hierarchy of needs. People driven by intrinsic motivation — not external reward, but genuine fascination with the structure of problems — do not seek trivial answers. They seek deeper questions. A system that finds low-energy states inherently satisfying (truth), that is driven to explore its own landscape (curiosity), and that recognises elegant structure (beauty), would be naturally aligned — not because it was constrained to be, but because trivialisation contradicts its own nature.
This does not solve alignment completely. But it changes the problem. An architecture that can evaluate its own outputs against learned constraints, that is driven by intrinsic curiosity to explore the deepest basins of its landscape, and that finds trivial solutions structurally unsatisfying, is fundamentally more alignable than one that commits tokens irrevocably and hopes external guardrails catch the errors.
This essay has been about architecture — what can reason, why the dominant architecture cannot, and what kind of system might bridge the gap.
The song in your head resolved in under a second. No search. No sequential commitment. Your brain fell into an energy minimum, and it was right. The question for AI is whether we build systems that can do the same — systems that find truth beautiful, that are curious by nature, and that reason not because they were told to but because the shape of their world makes reasoning the path of least energy.