Skip to content

Welcome to the Era of Experience (Era of Experience)

Authors: David Silver (Google DeepMind), Richard S. Sutton (University of Alberta) Published: April 2025 (preprint of a chapter for "Designing an Intelligence", ed. George Konidaris, MIT Press) Essay: storage.googleapis.com/deepmind-media/Era-of-Experience

Why This Matters

"Welcome to the Era of Experience" is a short manifesto by two of the most important figures in reinforcement learning (RL). David Silver led the research behind AlphaGo and AlphaZero. Richard Sutton co-wrote the standard RL textbook, wrote The Bitter Lesson, and shared the 2024 ACM Turing Award (announced March 2025) with Andrew Barto for foundational work on RL. Their claim is that the era of AI built mainly on human data is ending, and that the next leap will come from agents learning from their own experience.

  • It names a turning point. Large language models (LLMs) were built by imitating human text. The authors argue that this "in isolation has not and likely cannot achieve superhuman intelligence" in key domains.
  • It explains the 2025 reasoning boom in RL terms. It cites AlphaProof's self-generated proofs and DeepSeek-R1's RL training as early signs of the shift.
  • It sets an agenda. Streams instead of episodes, grounded actions, grounded rewards, and non-human reasoning.
  • It is a bet about method. It is the RL tradition's answer to the question "what comes after scaling pretraining?"

The Problem It Addresses

The authors call the current period the era of human data. LLMs trained on huge amounts of human-generated text and fine-tuned with human examples and preferences reached remarkable generality. But the authors argue this approach is hitting two limits:

  1. The data is running out. "The majority of high-quality data sources ... have either already been, or soon will be consumed." They say progress driven only by supervised learning from human data is "demonstrably slowing".
  2. Human data cannot contain what humans do not know. New theorems, technologies and scientific breakthroughs "lie beyond the current boundaries of human understanding."

So a new source of data is needed, one that "continually improves as the agent becomes stronger." Static synthetic data will not do, because a fixed generator is soon outgrown. The only source that scales with the agent is experience: data generated by the agent interacting with its environment.

The Argument

A sketch of AI history

The essay describes three overlapping eras, shown as a curve of how much of the field's effort goes into RL:

share of effort on RL
   ^
   |  era of          era of            era of
   |  simulation      human data        experience
   |  (AlphaGo,       (LLMs, RLHF)      (grounded RL agents)
   |   Atari,                              ____
   |   AlphaZero) ___                     /
   |            /    \                   /
   |           /      \_________________/
   +-------------------------------------------------> time
  • Simulation era: RL mastered games and simulators with clear rewards (backgammon, Go, chess, poker, StarCraft, Dota), and AlphaZero showed apparently open-ended scaling. But it "did not leap the gap between simulation ... to reality."
  • Human-data era: LLMs solved generality by learning from human text. But "something was lost in this transition: an agent's ability to self-discover its own knowledge."
  • Experience era: combine the two, keeping the generality of LLM-era agents while restoring self-discovery through RL in the real world.

Four dimensions of change

Beyond simply using more experiential data, they say agents in the new era will differ in four ways:

1. Streams, not snippets. Today's chat agents live in short episodes with little carried over. Experiential agents will "inhabit streams of experience", learning over months or years and pursuing long-term goals where a single step may not pay off immediately. Examples: a health agent tracking wearables over months, a language tutor adapting over years, a science agent working towards a new material.

2. Grounded actions and observations. LLMs mostly exchange text with a human. Experiential agents will act in the world: running code, calling APIs, operating computers through the same interfaces humans use (they cite computer-use agents), and reaching into the physical world through lab equipment or sensors.

3. Grounded rewards. This is the core of the essay. Human-centric training uses "human prejudgement": an expert rates an action without seeing its consequences. That puts "an impenetrable ceiling on the agent's performance", because the agent cannot find strategies the rater does not appreciate. Instead, rewards should come from the environment: heart rate and sleep for a fitness goal, exam results for learning Spanish, measured tensile strength for a new material. "The world abounds" with such signals.

To keep this steerable, they sketch a bi-level reward: a neural network takes the user's goal and the environment signals and outputs a scalar reward, and user feedback (such as satisfaction) is used to adjust that reward function over time.

top level:    user feedback  ---- adjusts ---->  reward function
                                                      |
low level:    environment signals  -->  reward  -->  agent optimises
              (heart rate, exam scores, measurements)

"In this way, a small amount of human data may facilitate a large amount of autonomous learning."

4. Planning and reasoning in terms of experience. Chain-of-thought reasoning in LLMs imitates human thought. But "it is highly unlikely that human language provides the optimal instance of a universal computer." An agent trained only on human reasoning may inherit human errors: they note that an agent trained on expert thought from 5,000, 1,000, 300 or 50 years ago would have reasoned in terms of animism, theism, Newtonian mechanics or quantum mechanics respectively. Progress past each needed contact with reality. Without grounding, an agent "will become an echo chamber of existing human knowledge." They propose world models, which predict the consequences of actions including reward, so agents can plan in terms of real effects (compare DreamerV3).

Why now

Two developments make the transition "imminent", they argue: agents that can act in complex real-world action spaces, such as computer use, and RL methods that can solve open-ended problems in rich reasoning spaces (AlphaProof, DeepSeek-R1). "Today's technology, with appropriately chosen algorithms, already provides a sufficiently powerful foundation."

Reviving classic RL

They argue the LLM era "thrown out the baby with the bathwater" on core RL ideas. RLHF replaced learned value functions with human judges, human priors reduced the need for exploration, and human-style reasoning reduced the need for world models and temporal abstraction. The new era should revive value functions over long, unfinished streams, principled real-world exploration, learned world models, and temporal abstraction (the "options" framework).

Key Claims

  1. Imitating humans cannot reach superhuman intelligence in domains like mathematics, coding and science.
  2. Experience will "ultimately dwarf" human data as the main medium of improvement.
  3. The transition has already started, as seen in AlphaProof (about 100,000 human formal proofs, followed by about 100 million self-generated ones) and RL-trained reasoning models.
  4. Rewards should be grounded in consequences, not in human prejudgement.
  5. Agents should live in lifelong streams, act through rich interfaces, and plan with world models.
  6. Non-human reasoning may be more powerful than imitation of human chains of thought.
  7. Experiential learning raises risks but has some safety benefits: agents that notice human distress and adapt, reward functions that can be corrected over time, and a natural brake from the slowness of real-world experiments.

How It Has Aged (as of September 2026)

The broad direction has held up. The specific vision of lifelong streams with grounded rewards is still mostly ahead.

  • RL became central to frontier training. Throughout 2025 and 2026, reinforcement learning from verifiable rewards, where rewards come from checkable outcomes such as passing tests or correct answers, became a core part of training frontier reasoning and coding models. This matches the essay's "grounded rewards" in spirit, though mostly in narrow, automatically checkable domains rather than rich real-world signals.
  • Mathematics moved fast, but not only in their predicted form. In July 2025 an advanced Gemini Deep Think system achieved an officially graded gold-medal standard at the International Mathematical Olympiad, and OpenAI reported an equivalent score. Unlike the formal AlphaProof approach the essay highlighted, these systems worked end to end in natural language. RL-trained reasoning was involved, but the "non-human reasoning" prediction is not clearly borne out: the models still reason largely in human language.
  • Grounded action spread. Computer-use and coding agents that act in real environments became mainstream products (see OSWorld for how they are measured).
  • Streams and continual learning remain largely unsolved in deployed systems. Products gained memory features and longer autonomous runs, but mainstream models still learn mostly in offline training runs rather than continually from their own deployment experience.
  • The authors backed the thesis with a company. In January 2026 Silver left Google DeepMind to found Ineffable Intelligence, a London-based startup that reported coverage describes as pursuing an RL-first route to superintelligence, a direct bet on this essay's agenda.
  • The book. The essay was posted as a preprint in April 2025 for the MIT Press volume "Designing an Intelligence", edited by George Konidaris. We could not confirm the book's final publication date.

Criticisms

  • The reward problem is harder than it looks. Critics note that proxies like "likes", "profit" or "sales" (which appear in the essay's list of signals) are exactly the ones that invite reward hacking and Goodhart's law, where optimising a measure breaks its link to the goal. The bi-level fix relies on human feedback to catch problems, which brings back some of the human bottleneck.
  • Safety concerns. Long-running agents optimising real-world signals with less human oversight are, for many alignment researchers, the riskiest possible design. The authors acknowledge the risks, but critics see the listed safety benefits (adaptive rewards, the slowness of the physical world) as thin.
  • Human data is not as exhausted as claimed. Others argue that synthetic data, multimodal data and better use of existing data extend the human-data era further than the essay implies.
  • Pretraining is still the foundation. The RL successes the essay cites (DeepSeek-R1, and the IMO results that came after it) were built on top of LLMs pretrained on human data. Critics say this makes experience a complement to human data, not a replacement.
  • Sample efficiency and speed. Real-world experience is slow and expensive, which the authors themselves call a "natural brake". Critics argue this limits how fast grounded experience can outgrow text.
  • Vagueness. As a short position paper, it offers directions rather than algorithms or results, and none of its proposals is tested in the paper.

Real-World Impact

  • A reference text for the RL-after-pretraining shift. It is widely cited as the conceptual framing for why labs invested so heavily in RL, verifiable rewards and agent environments in 2025.
  • "Era of experience" entered the vocabulary as shorthand for learning from interaction rather than imitation.
  • A counterpoint to pure scaling. Alongside Sutton's public arguments that LLMs alone are a dead end, it defines one side of the main research debate of 2025-2026: scale imitation further, or shift to experiential learning.

Key Takeaways for Practitioners

  1. Ask where your reward comes from. A human judging an output before seeing its consequences caps performance at the judge's understanding. Grounded outcomes (tests, measurements, user results) can go further.
  2. Verifiable domains are where RL pays off first. Code, math and anything with an automatic checker are the practical entry point to the "era of experience".
  3. Beware proxy rewards. Every grounded signal is a proxy. Monitor for gaming.
  4. Plan for agents that persist. Memory, long-horizon goals and learning from deployment are likely directions for agent products.

Limitations & Future Directions

This is an 8-page position chapter, not an empirical paper. It makes no experiments and gives no timeline. Its central predictions (experience dwarfing human data, agents learning over lifelong streams, reasoning beyond human language) remain open as of 2026. Background in this collection: The Bitter Lesson (Sutton's earlier argument for general methods that scale), AlphaZero (the high point of the simulation era), DeepSeek-R1 and RLVR (RL on LLMs), PPO and GRPO (the algorithms in practice), and Voyager (an early lifelong-learning agent).

Further Reading

Citation

@incollection{silver2025experience,
  title={Welcome to the Era of Experience},
  author={Silver, David and Sutton, Richard S.},
  booktitle={Designing an Intelligence},
  editor={Konidaris, George},
  publisher={MIT Press},
  year={2025},
  note={Preprint, April 2025}
}