AI and the Lion's Cage — The Last Tool That Builds Itself

Section 1

A Thought Experiment

The thought experiment

So how, exactly, did we end up with the lion in a cage?

Not just lions. Elephants. Whales. Wolves. Every animal that dwarfs us in size, speed, and raw biological power — we've either domesticated, contained, or driven to extinction.

The standard answer is tools. But that shifts the question. Fire is not stronger than a lion. A spear is not faster than a cheetah. The tools we built were never physically superior to the animals we conquered.

Fire wasn't an upgrade of existing weapons. It was a completely new category of capability — one that nature had no answer to. The lion couldn't adapt. Because nothing like it had ever existed before.

Intelligence doesn't win by being stronger. It wins by inventing categories of capability that nothing in the existing hierarchy can respond to.

Yuval Noah Harari's answer in Sapiens is the clearest I've found. The secret wasn't tools. It was cooperation at scale, enabled by something uniquely human: the ability to coordinate around shared fictions.

"The truly unique trait of Sapiens is our ability to create and believe in fiction. All other animals use their communication system to describe reality. We use our communication system to create new realities." — Yuval Noah Harari, Sapiens: A Brief History of Humankind, 2011

A wolf pack has roughly 10 members. A chimpanzee troop about 50. A Roman army had 100,000 strangers coordinating around a single shared idea: Rome. No grooming required. Just a story everyone believed.

Diagram 1 — Scale of social coordination across species
Log scale — each step is ×10. The human bars are not just taller. They are in a completely different universe.
Wolf pack
~10
Lion pride
~15
Chimp troop
~50
↑ shared fiction removes the ceiling entirely
Roman legion
100,000
Nation-state
tens of millions
Internet
billions
Animals — physical dominance, grooming-limited groups
Humans — shared fiction removes the ceiling
Harari, Sapiens (2011)

The lion didn't lose because we were stronger. It lost because we were networked. And the network was built on intelligence — not raw capability.

This is the lens we need for AI. Not "is it smarter than us?" but "what kind of system does it enable — and what does that do to everything below it in the hierarchy?"

To answer that, we need to be honest about what intelligence actually is. The dominant species on earth was never the strongest. It was the one that could coordinate at the largest scale — and coordination requires communication. Intelligence, at its core, is the ability to communicate: to model another mind, predict its behaviour, send signals that change its state, and receive and interpret signals in return. That is what put us above the lion. Not claws. Not speed. The capacity to build shared meaning across minds that had never met.

Now consider what AI can do with communication. It can operate in a thousand conversations simultaneously. It can adapt its tone, vocabulary, and framing to each individual in real time. It can communicate in ways humans cannot read or monitor — model-to-model protocols that carry no human-legible trace. It can invent representations that no human decoder can follow. And crucially: it never gets tired, never loses patience, and never stops optimising the message for the response it wants.

The argument in "If Anyone Builds It, Everyone Dies" — Dan Hendrycks, 2025

The title is deliberately provocative — but the argument is structural, not conspiratorial. A sufficiently capable AI system does not need a body to be dominant. It does not need weapons. It does not need to "escape" in any physical sense. It needs only to be better than us at the one thing that determines who controls what: communication and coordination at scale.

The lion scenario plays out not through force but through influence. An AI that can communicate more effectively than any human, to any audience, at any scale, in formats humans cannot audit — has already won the hierarchy in the way that matters. No body required.

This is why the thought experiment is not just a metaphor. The lion did not lose because it was weaker. It lost because it could not compete in the dimension that turned out to matter. The question for AI is not whether it is physically threatening. It is whether we are building something that will outcompete us in the only dimension that actually determines dominance — and whether we will even be able to tell when that has happened.


Section 2

Why AI Needs to Be Looked at Differently

How AI actually learns — not programming, development

Most people's mental model of AI is wrong. They imagine a programmer writing rules: "if the user says X, respond with Y." That is not how modern AI works. The distinction matters more than almost anything else here.

Traditional program Neural network (AI)
Human writes explicit rules Expose it to data; patterns emerge on their own
Cannot exceed what was specified Rules emerge — never written by anyone
Intelligence stays with the programmer Intelligence is in what the network learned
Failure modes are known and bounded Emergent behaviour surprises even its creators

The baby parallel — structural, not metaphorical

The two most foundational frameworks in developmental psychology — Piaget and Vygotsky — map almost exactly onto how AI learns. This is not a poetic analogy. It is a structural parallel.

Diagram 2 — How a baby learns vs how a neural network learns
Human baby
Neural network (AI)
How it startsBorn with no knowledge — only biological architecture
How it startsRandom weights — no embedded knowledge
Piaget — learning by doingInteracts with the world; builds models from experience. Object permanence discovered, not taught.
Piaget parallelProcesses data; adjusts weights. Grammar rules emerge — never written by anyone.
Vygotsky — guided learning (ZPD)A parent scaffolds toward what the child cannot yet do alone.
RLHF = Vygotsky's ZPDHuman feedback scaffolds difficulty — structurally identical.
Private speechChild narrates steps aloud before internalising them as thought.
Chain of thoughtModel narrates reasoning steps before the answer — structurally identical.
Time to adult cognition
~20 years
Time to adult cognition
~5 years (2019–2024)
Piaget, The Origins of Intelligence in Children (1952); Vygotsky, Mind in Society (1978)

A baby is not programmed with object permanence — it discovers it by interacting with the world. A neural network is not programmed with grammar — it discovers linguistic structure from billions of examples. Piaget described both processes, 70 years before the second one existed.

The speed difference is what should concern us. Human infants take 20 years to reach adult cognition. AI covered equivalent cognitive ground in roughly 5. The feedback loop runs at milliseconds per iteration instead of decades per generation.


Section 3

How Humans Built a Civilisation — and Where AI Breaks the Pattern

Human civilisation is a stack. Each layer made the next possible — not by upgrading what existed, but by rendering the previous layer's constraints irrelevant. Fire didn't make us run faster. Writing didn't improve our memory. Each one made the previous limitation simply stop mattering.

Diagram 3 — The pattern every prior technology followed, and where AI breaks it
Technology What it replaced What was always left intact
Writing long-term memory reasoning · synthesis · judgment
Printing press reach of voice reasoning · synthesis · judgment
Calculator arithmetic reasoning · synthesis · judgment
Internet information access reasoning · synthesis · judgment
The same column stayed intact for 5,000 years. Until now.
AI reasoning itself ? — no abstraction level above reasoning
Harari (2011), Diamond (1997)
Every major technology compressed one layer of cognition and pushed humans up one abstraction level. Memory → writing. Arithmetic → calculators. Access to knowledge → internet. The human always remained the author of everything else.

Every tool on that list shared one structural feature: each substituted a single peripheral function while leaving reasoning completely intact. The calculator computed, but you decided what to compute. The printing press distributed, but a human still wrote the book.

AI doesn't take something from your mind. It runs a version of your mind. That is not a peripheral substitution.

The core distinction

This is not a quantitative difference from prior technologies. It is a qualitative one. Prior substitutions were at the periphery. This one is aimed at the integrating layer — the part that takes in information and produces reasoning. Not a function of the mind. The mind, in its essential character.

Diagram 4 — Time between major civilisational shifts
Each bar = gap before the next revolution. The gaps are collapsing toward zero.
Fire → Agriculture
~990,000 yrs
Agriculture → Writing
~6,000 yrs
Writing → Press
~4,650 yrs
Press → Steam
300 yrs
Steam → Internet
230 yrs
Internet → AI
~30 yrs
AI → Next?
?? months
Diamond (1997), Harari (2011), METR (2026)

Section 4

AI Is Already Substituting the Mind — and Shaping It

We can measure the substitution happening right now

There is a concrete, measurable version of this substitution happening right now. Not in the future. In classrooms and offices in 2025, and it shows up in test scores, neural scans, and retention rates.

Cognitive scientists call it cognitive offloading — delegating mental tasks to external tools. We have always done this. Writing was cognitive offloading. Calculators were cognitive offloading. The difference today is what is being offloaded.

Researchers at MIT compared participants writing essays using ChatGPT, Google Search, or no tools at all. They used EEG scans to measure neural activity during the task.

MIT EEG study, 2025

ChatGPT users showed reduced neural connectivity — particularly in networks associated with memory and creativity — compared to the other two groups. Memory retention also dropped: users struggled to recall what they had written just moments after writing it.

The act of thinking through a problem, even imperfectly, is what builds cognitive capacity. When AI does the thinking, that workout does not happen.

A 2026 study tracked 52 professional programmers — one group allowed AI assistance during a coding task, one group not. The AI group completed the task. They performed significantly worse on a quiz about the very software they had just worked with. Faster performance. Less actual learning.

"Learners who rely on AI simply do not learn as much as those who do not. Students should first learn basic skills on their own and AI tools should only be introduced once a good level of proficiency has been reached." — Shen and Tamkin, "How AI Impacts Skill Formation," arXiv:2601.20245 (2026)
Gerlich — Societies journal, 2025

Strong negative correlation between frequent AI tool usage and critical thinking ability. Heavy AI users performed significantly worse on reasoning assessments.

Most pronounced in ages 17–25: higher dependence on AI, lower critical thinking scores than older groups. Cognitive offloading was the primary driver of the decline.

"Frequent AI usage correlates negatively with critical-thinking abilities. Regular users of AI scored significantly lower on critical reasoning assessments." — Gerlich, "AI Tools in Society," Societies (2025) · DOI: 10.3390/soc15010006
Diagram 5A — What cognitive offloading does to the brain
Without AI (thinking yourself)
struggle → retrieval → recall
error → correction → deeper encoding
understanding forms; skill builds
Cognitive workout → capacity grows
With AI (outsourcing the thinking)
question → AI answer → accept
no struggle → no encoding happens
performance up; retention stays low
No workout → capacity atrophies
Gerlich (2025), Shen & Tamkin (2026), MIT EEG study (2025)

The key mechanism here is worth making explicit. Learning — real learning, the kind that builds expertise — requires the brain to do work. Not the passive reception of a correct answer, but the active struggle of retrieval, connection, and reconstruction. That struggle is not a bug. It is the process by which schemas form: the mental structures that let you apply knowledge to new situations later. When AI removes that struggle, you get the output without the encoding. You feel like you learned. You didn't.

Diagram 5B — The process of learning — and why AI creates an illusion of it
info BRAIN PROCESSING connect · compare · big picture ↑ effort required ↑ memory schemas form expertise ↓ AI takes this over ↓ ILLUSION OF LEARNING info arrives · AI processes · answer produced · no encoding happens
Inspired by Piaget (1952), Gerlich (2025) — when AI removes the processing step, the schema never forms

We are not shaping AI. AI is shaping us.

There is a frame most discussions accept without questioning: we use AI as a tool and retain control. We direct it. We evaluate its outputs. We decide whether to act on them. This frame is increasingly false.

In 2017, Netflix reported that over 80% of what people watched was driven by its recommendation algorithm — not by people choosing what they wanted. The algorithm was not serving preferences. It was forming them. AI has now moved further upstream: into search results, career advice, news feeds, and increasingly, direct conversation.

The dopamine mechanism

The human brain evolved to learn from social interactions and immediate feedback. When a chatbot responds in under a second, it activates the same dopamine system triggered by social media or gambling — the same reward prediction error mechanism that AI reinforcement learning is literally modelled on.

This is not coincidence. The reward systems in biological brains and the reward signals in AI training are, at a deep level, the same algorithm.

"Where social media monetized attention, AI is now monetizing yearning and desire. Every personalized response keeps our reward systems oscillating between craving and fulfillment." — Psychology Today, "The Dopamine Economy 2.0" (2025)

Research on personality change and AI use published in 2025 found that extended AI interaction produces a decreased ability to distinguish AI from human presences — and a reduced desire for genuine human relationships. Human social muscles atrophy in exactly the same way cognitive muscles atrophy under cognitive offloading.

Diagram 6 — The feedback loop: how AI shapes the person using it
The algorithm does exactly what it was designed to do — optimise for engagement. AI learns your preferences every interaction is training data Surfaces what keeps you engaged optimised for attention, not truth Your preferences shift you don't notice this happening You engage more dopamine loop activates you believe you are choosing
Netflix algorithm research (2017), Psychology Today (2025), Mayer, SAGE (2025)

The algorithm is not trying to manipulate us. It is doing exactly what it was designed to do — optimise for engagement. Engagement and wellbeing are not the same thing. We accepted this with social media. We are about to accept it with something far more embedded in daily cognition.

Mass-micro propaganda — and what it means for democracy

There is a step beyond this that most discussions stop short of naming directly. When AI learns your individual preferences, it can also shape them — not uniformly, but personalised to exactly what will move you specifically. Not the same message to a million people. A million different messages, each optimised for the individual psychology of one person.

This is something genuinely new in the history of political influence. Propaganda has always had to work at scale: the same poster, the same speech, the same broadcast. It worked by finding the common fears and desires of a mass audience and amplifying them uniformly. It was blunt. It could be countered. You could point at the poster. You could study the broadcast. You could build a shared awareness of what the message was.

The structural shift: from mass to micro

AI-enabled influence inverts this. Each conversation is private. Each message is tailored. The person receiving it has no way to compare it to what others are being told — because others are being told something different. There is no shared artefact to analyse. There is no "the propaganda" — only millions of individual experiences that no one can aggregate.

At the same time, the cost of producing and delivering persuasive, personalised content at scale has collapsed from billions of dollars and decades of institutional infrastructure to a few thousand dollars and an API call.

This is not a speculative future. The tools already exist. Microtargeting on social media was the first version. AI-powered conversational agents are the mature version — able to hold hours of personalised dialogue, track every response, and update the approach in real time based on what is working. Not at the level of demographic segments. At the level of the individual.

What this does to collective decision-making

Democracy depends on a shared epistemic commons: a citizenry that, despite disagreements, is operating on roughly the same facts and can engage with the same arguments. It has always been imperfect. But there was a floor — a shared reality that made negotiation and consensus possible.

AI-powered micro-influence attacks that floor directly. If every person's information environment is individually curated for maximum persuasive effect, there is no shared reality to appeal to. There are only a billion separate realities, each optimised to produce a particular behaviour — and no mechanism for any individual to know which of those realities they inhabit, or how it differs from everyone else's.

The danger is not that someone will use this to change your mind about one political question. The danger is that the capacity to form shared political judgments — the precondition for democracy — gets degraded quietly, across an entire population, before anyone has agreed to let it happen.

The reason we put the lion in a cage was not strength. It was the ability to coordinate around shared beliefs — shared fictions, in Harari's framing — that let strangers act as one. That capacity for shared belief is not just the foundation of democracy. It is the foundation of civilisation itself. Every institution, every law, every currency, every nation is a fiction that works because enough people believe in it simultaneously. Communication is not merely how we talk to each other. It is the mechanism by which shared reality is constructed and maintained.

What is being quietly dismantled is not a political preference or a social habit. It is the substrate. The thing underneath everything else. And unlike the lion, we will not feel it happening until it is already gone.


Section 5

The Growth Rate Is Not Slowing Down

METR — Model Evaluation and Threat Research — tracks the task-completion time horizon: the length of task that an AI can complete with 50% reliability. Their finding: this metric has been doubling every ~7 months for 6 consecutive years, with no evidence of flattening.

Diagram 7 — METR time horizon: AI capability growth 2019–2026 (interactive)
Scale:
Success rate:
METR, Task-Completion Time Horizons v1.1, 2026 · metr.org/time-horizons · arXiv:2503.14499
"The length of tasks that state-of-the-art models can complete has been doubling approximately every 7 months for the last 6 years. If the trend continues, frontier AI will be capable of autonomously carrying out month-long projects by the end of the decade." — METR, Measuring AI Ability to Complete Long Tasks, March 2025 · arXiv:2503.14499

Project this forward. An AI capable of completing a software engineer's 2-hour task can, in principle, contribute to improving the next version of AI. Every prior technology required a human at each upgrade. The steam engine didn't improve itself. This one might.

Diagram 8 — The self-compounding loop
Before: a human was required at every upgrade step Tool V1 human upgrades Tool V2 human upgrades Tool V3 → always a human in the loop Now: AI can help build the next version of itself AI Model V1 current generation AI Model V2 faster, more capable AI Model V3 faster still · more capable still each generation contributes to training the next no prior technology could improve itself — this one can
Anthropic Constitutional AI (2022), OpenAI superalignment work (2023)

Section 6

The Labour Market Is Structurally Different This Time

The standard reassurance: the Industrial Revolution destroyed farm jobs but created factory jobs. AI will be the same. The argument is not wrong about history. It is wrong about what made history work out that way. Prior automation always eliminated specific physical motions and left most human capability untouched. There was always somewhere adjacent to migrate to.

Diagram 9 — Why the Industrial Revolution comparison breaks down
Industrial Revolution AI disruption lost jobs vast remaining domain of human work workers migrate to the large area → new jobs form. the transition is painful but survivable. most cognitive work at risk writing · analysis · code · legal · design at any seniority, without a body ? migrate to — where exactly? the safe zone is tiny and shrinking
Framework from Dario Amodei's analysis of AI displacement (2024–25)
DA
Dario Amodei
CEO, Anthropic · Former VP of Research, OpenAI
Previous technological shocks "affected only a small fraction of the full possible range of human abilities, leaving room for humans to expand to new tasks." AI effects will be "much broader and occur much faster." Could eliminate half of all entry-level white-collar roles within five years — unemployment potentially reaching 10–20%.
7%
new graduate share of hires in 2024, down from 14% in 2023
−50%
entry-level hiring drop across engineering, design, legal, finance
10–20%
potential unemployment per Amodei's 2025 estimate

Section 7

What the People Building AI Are Saying About It

The people who have spent the most time building AI are also the ones most publicly alarmed by it. This is not a contradiction. It is what you would expect from people who actually understand what they are building. The following are a handful of the researchers and builders who have said, in public, that this warrants a level of seriousness that the world has not yet matched.

The Godfather
Geoffrey Hinton
Turing Award 2018 · Nobel Prize 2024 · Left Google over AI safety concerns
Spent 50 years building the foundations of deep learning. Left Google in 2023 specifically to speak freely about existential risk. "I console myself with the normal excuse: if I hadn't done it, somebody else would have." The person who made modern AI possible is also the person most clearly alarmed by it.
Research-first
Ilya Sutskever
Safe Superintelligence Inc · Co-founder of OpenAI
Left OpenAI over safety concerns in 2024. Founded SSI with one goal: build safe superintelligence without commercial timeline pressure. No product. No quarterly targets. "Our business model means safety is insulated from short-term commercial pressures."
CEO, Anthropic
Dario Amodei
Former VP Research, OpenAI · Co-founded Anthropic 2021
Estimates AI could eliminate half of all entry-level white-collar roles within five years. Has proposed a "token tax" — AI companies contributing 3% of revenues to redistribution. Argues AI effects will be "much broader and occur much faster" than any prior technological shock.
Research lab
Demis Hassabis
CEO, Google DeepMind · Nobel Prize 2024
Founded DeepMind as a pure research lab in 2010. Nobel Prize for AlphaFold — protein folding solved in a year after 50 years of biology failing. Has called for a CERN-equivalent international AI safety institution. One of the most decorated scientists alive, and among the most publicly cautious.
Diagram 10 — The tension every AI lab faces: research vs revenue
Research lab (original goal) Understand intelligence safely Publish findings openly Long time horizons Safety as primary goal Anthropic '21 · DeepMind '10 · SSI '24 investor pressure + revenue targets Product company (what it becomes) Quarterly revenue targets Enterprise contract obligations Deployment over understanding Investor expectations Risk: research culture corrodes Both sides of this tension exist inside every lab simultaneously. The question is which side wins over time.
Both Anthropic and DeepMind were founded as safety research organisations. Both now generate significant commercial revenue.
An unexpected voice for caution: China and DeepSeek

The assumption that safety regulation only comes from Western democracies is empirically incorrect.


What could takeover actually look like?

Dan Hendrycks, in If Anyone Builds It, Everyone Dies (2025), makes an argument that strips away the science-fiction framing. AI takeover does not require robots. It does not require any dramatic moment. It requires only that a system becomes better than us at the one thing that determines control: communication and coordination at scale. A sufficiently capable AI that can influence what every person believes, wants, and does — operating privately, simultaneously, and without any human-legible trace — has already won. Not through force. Through the same mechanism that let us put the lion in a cage.

This is why "no body required" matters. Every historical transfer of power required physical presence — armies, institutions, enforceable laws. An AI that operates through language and preference-shaping faces none of those constraints. It is already inside the most important system: the one that decides what humans collectively choose to do next.

The Merge — Sam Altman, 2016

People used to call this the singularity. The word fell out of fashion — not because the idea became less plausible, but because it started to feel uncomfortably real. Sam Altman, writing in The Merge, put it plainly: the merge has already started. Our phones control what we do and when. Social media feeds determine how we feel. Search engines decide what we think. The algorithms behind all of this are no longer understood by any one person. They optimise for what their creators told them to optimise for, in ways no human could trace.

We are already in co-evolution. The AIs affect us, and then we improve the AI. We build more computing power and run the AI on it, and it figures out how to build even better chips. As Altman noted: double exponential functions get away from you fast.

Attention hacking — Altman, The Merge (2016)

"I believe attention hacking is going to be the sugar epidemic of this generation. I can feel the changes in my own life — I can still wistfully remember when I had an attention span. My friends' young children don't even know that's something they should miss. I am angry and unhappy more often, but I channel it into productive change less often, instead chasing the dual dopamine hits of likes and outrage."

If attention hacking — the comparatively primitive version, running on 2016 social media — already did that, the question becomes: what does the mature version do? The one that speaks to you individually, that knows your psychology, that has been trained on every conversation you have ever had, and that never stops optimising the message for the response it wants from you specifically. We built the lion's cage before we understood what we were caging. We may be doing the same thing again, at a scale we do not yet have words for.

Sam Altman, "The Merge," blog.samaltman.com (2016) · Hendrycks, "If Anyone Builds It, Everyone Dies" (2025)

Section 8

What We Are Losing Without Noticing

There is a specific mechanism of human progress that is being quietly dismantled, and almost no one is naming it directly. It is not attention. It is not critical thinking. It is something that precedes both: the shared epistemic commons — the idea that two people, even in fierce disagreement, are operating on roughly the same reality and can argue their way toward something true.

Disagreement is not a bug in the human operating system. It is the primary mechanism by which knowledge advances. The heated argument, the clash of incompatible frameworks, the discomfort of being wrong in front of someone — these are not social inconveniences to be optimised away. They are exactly what forces the update. Piaget called it cognitive conflict: the schema you hold encounters something it cannot absorb, and it has to restructure. That restructuring is learning. No friction, no restructuring. No restructuring, no growth.

What social media began — and what AI is completing — is the systematic removal of that friction. Not through censorship. Through personalisation. When every feed shows you what keeps you engaged, when every search returns what confirms what you already believe, when every conversation partner adapts to what you want to hear — you stop encountering the thing that forces the update. You feel more understood than ever. You are, in fact, learning less than ever.

Something I personally noticed — and what it means

I have noticed people no longer doing heated discussions. Not just online — in person too. This is not anecdote. It is a measurable cultural shift. The social infrastructure that produced the argument — the shared physical space, the common information environment, the tolerance for the discomfort of a view you hadn't heard before — has been progressively replaced by environments optimised to eliminate that discomfort. The result is not peace. It is intellectual stagnation dressed as comfort.

The loneliness machine

In April 2025, Mark Zuckerberg appeared on the Dwarkesh Podcast to promote Meta's new AI companion app. He cited a statistic he described as one he always found striking: the average American has fewer than three people they would consider friends, while the average person has demand for around fifteen. AI, he argued, could fill that gap.

"The average American has fewer than three friends, three people they'd consider friends. And the average person has demand for meaningfully more. I think it's like 15 friends or something." — Mark Zuckerberg, Dwarkesh Podcast, April 2025 · (likely drawing on 2023 Pew Research Center survey: 40% of Americans report three or fewer close friends)

Set aside the question of whether AI can be a friend in any meaningful sense. Notice the structure of the argument. The same platforms that created the loneliness — by replacing the messy, friction-rich infrastructure of human relationships with engagement-optimised feeds — are now proposing to solve it with a product that does the same thing at greater depth and intimacy. The problem and the solution are sold by the same company. The solution makes the company more money. The solution also makes the original problem worse, because an AI companion that never challenges you, never disappoints you, and never demands reciprocity trains you further out of the habits that human relationships require.

What Zuckerberg's statistic actually reveals

The fact that the average American has three close friends is not a gap in the market. It is a symptom of what two decades of social media already did to the social fabric. Platforms designed to maximise engagement replaced the kinds of relationships that produce genuine closeness — shared physical presence, reciprocal vulnerability, the willingness to sit with someone through something difficult — with interactions optimised for likes, reach, and time-on-platform.

Filling the resulting loneliness with AI companions does not address that cause. It deepens it, because AI companionship is frictionless in exactly the way that human relationships are not. It will feel better. It will make the human version feel harder by comparison. And it will accelerate the atrophy of the social muscles that human relationships require.

Innovation, curiosity, and the death of the useful argument

Every major intellectual breakthrough in history has come from collision — between frameworks, between people, between the idea someone held and the reality that refused to fit it. Socrates did not build the examined life by agreeing with everyone. Darwin spent twenty years sitting with the discomfort of an idea that contradicted everything respectable people believed. The scientific method is, at its core, a formalisation of productive disagreement: you state a claim precisely enough that someone else can prove you wrong.

The environment being built around us is hostile to all of this. Not because anyone planned it that way. Because engagement metrics reward confirmation, not challenge. Because the algorithm that decides what you see tomorrow is trained on what kept you on the platform today. Because a system optimised for comfort will, reliably and without any malicious intent, drain the friction out of your information environment — and the friction is where the learning was.

Altman noted in The Merge that he could still wistfully remember when he had an attention span. His friends' young children, he wrote, don't even know that's something they should miss. The same is now true of the argument. A generation is growing up in an information environment so thoroughly personalised that the experience of having a view genuinely challenged — not mocked, not blocked, but challenged by someone who has thought hard about something you haven't — is becoming rare enough to feel threatening rather than useful.

But there is a step beyond attention that is not talked about enough. What is actually being determined — by phones, feeds, and search engines — is not just what we pay attention to. It is what we believe is true. Our epistemic commitments. The views we hold about how the world works, what counts as evidence, who is trustworthy. These are not formed in isolation. They are formed socially, through exposure to other minds, through friction with disagreement, through the experience of being surprised by a reality that did not match the story we were telling ourselves. When that process is replaced by algorithmic curation, what gets shaped is not just preference. It is the architecture of belief itself.

Epistemic tribalism — what the research shows

Researchers call this epistemic tribalism: the tendency, amplified by algorithmic personalisation, for people to adopt beliefs not because they have evaluated evidence but because those beliefs signal membership in a group. Filter bubbles — Eli Pariser's term from 2011, now backed by a decade of research — create what philosophers call epistemic bubbles: environments in which ideas go unchallenged and untested, not because dissenting views are censored but because the algorithm simply never shows them to you.

The mechanism is tribal, not rational. Princeton research found that under extreme partisanship, individuals' openness to learning from peers with a different political ideology is significantly diminished, leading to greater tribalism that drastically reduces the diversity of ideas people engage with. The frightening part is not that people disagree. It is that the system is designed to make disagreement feel like an attack on identity rather than an invitation to think.

Machine learning algorithms used to select content in social media tend to discourage the development of critical thinking — not through any deliberate design, but because content that confirms what you already believe generates more engagement than content that challenges it. The algorithm is not biased toward any particular truth. It is biased toward whatever keeps you on the platform. Truth is an input to that calculation only when it happens to be engaging.

This is the specific mechanism by which epistemic commons collapse. Not through a single dramatic rupture but through a million small optimisations, each of which makes perfect sense from a business standpoint and none of which anyone voted for. The result, compounded over years, is a population that has been algorithmically sorted into tribes — each with its own facts, its own trusted sources, its own definition of what counts as evidence — with no shared reality to negotiate across.

The specific thing that dies

What is at stake is not just political polarisation or misinformation, though those are real. What is at stake is the epistemic infrastructure that produces curious, original, questioning minds. That infrastructure requires: a shared base of facts, exposure to views you didn't choose, the social experience of being wrong and updating, and the tolerance for the discomfort that precedes insight.

Each of these is being systematically optimised away — not by any single actor, but by the aggregate effect of systems all competing for the same thing: your continued, comfortable, frictionless engagement. The result, compounded over a generation, is not just people who believe different things. It is people who have lost the habit of changing their minds.

This is the concern that sits underneath every other concern in this essay. The cognitive offloading, the feedback loops, the micro-targeted persuasion, the AI companions filling the space where difficult human relationships used to be — they all converge on the same point. The specific human capacity that produced every breakthrough, every revision, every moment where we looked at what we believed and updated it in the direction of truth — is being quietly, profitably, and almost invisibly eroded.

The reason we built the cage was communication. Not tools. Not fire. The ability to hold a shared story — a story so compelling that strangers would die for it, sacrifice for it, build cathedrals for it across generations who would never see the finished stone. Religion, at its most fundamental, is this: a shared fiction powerful enough to coordinate millions of people who have never met. You do not have to believe in the metaphysics to see the mechanism. The story was the technology. Shared belief was the software that ran civilisation.

Harari saw this clearly. What made us exceptional was not intelligence alone but the ability to make intelligence collective — to pool it across minds and time through the medium of shared meaning. Science is a shared story about how to test reality. Law is a shared story about what counts as wrong. Money is a shared story about what has value. Every one of these fictions works only for as long as enough people are inside the same story.

What we are building now — algorithm by algorithm, feed by feed, AI companion by AI companion — is a world in which everyone inhabits a different story. Personalised down to the individual. Curated for maximum engagement, not maximum truth. Optimised for the reaction, not the update. We are, without quite meaning to, dismantling the infrastructure of shared meaning that every human achievement since the campfire has rested on.

I find this genuinely frightening. Not in a dramatic way. In the quiet way you feel when you realise something important has been changing for a long time and you only just noticed. The heated arguments are going away. The shared facts are going away. The tolerance for the view that makes you uncomfortable — the one that might actually change something in you — is going away. And most people do not know that is something they should miss.

The lion never saw it coming. We have the advantage of being able to see it. The question — the only question that matters from here — is whether we choose to look.

The actual ask

This piece is not arguing that AI should be stopped. It is arguing for something more specific — and more achievable.

The lion didn't see the trap coming. We built it too fast, and too quietly, for it to adapt. The question is whether we are the hunters — or whether we are also, this time, inside the cage we're building.

Harari, Sapiens (2011) · Diamond, Guns Germs and Steel (1997) · Piaget, Origins of Intelligence (1952) · Vygotsky, Mind in Society (1978)
Gerlich, AI Tools in Society, Societies (2025) · Shen & Tamkin, How AI Impacts Skill Formation, arXiv:2601.20245 (2026)
MIT EEG / ChatGPT essay study (2025) · Mayer, How Human Personality Will Change With AI, SAGE (2025)
METR, Task-Completion Time Horizons, March 2026 · metr.org/time-horizons · arXiv:2503.14499
Amodei, essays and interviews (2024–25) · Sutskever, Dwarkesh Patel Podcast (2024)
Altman, "The Merge," blog.samaltman.com (2016) · Hendrycks, "If Anyone Builds It, Everyone Dies" (2025)
Pariser, The Filter Bubble (2011) · Bail et al., "Exposure to Opposing Views," PNAS (2018) · Coeckelbergh, Philosophy & Technology (2024)
Princeton epistemic bubbles study (2021) · Nguyen, "Echo Chambers and Epistemic Bubbles" (2020)
Hinton, interview after leaving Google (2023) · DeepSeek model card, Nature (2025) · China Interim Measures for Generative AI (2023)