● Waymos World Model Creates Infinite Scenarios, Supercharging AI Training
Waymo’s Strategy for the “End of Autonomous Driving”: Using World Models to Generate Hundreds of Millions of Scenarios and Training a Driving Agent to the Finish
Three Things You Must Catch in Today’s Post (From the Start to the Key Takeaway)
1) Waymo’s key takeaway is that it doesn’t stop at road real-world data; it expands training by generating “imaginable scenarios” in large quantities with a world model.
2) The goal isn’t to make it “good enough to drive well in reality,” but to make it perfect at training in every scenario that can be imagined—and they say that is what they call the ‘end’ of autonomous driving.
3) And the training approach strongly shows a shift from relying on rules alone as in the past to centering on reinforcement learning + world-model-based simulation to further advance the driving agent.
Today, I’ll rework the contents of this interview into a “news-style” format and summarize Waymo’s learning philosophy and technical points at once.
News Summary (One-Line Briefing)
Waymo said that, alongside real-world data collected from road driving, it is pushing past the limits of autonomous driving by using world models to create large-scale simulation scenarios and training the driving agent end-to-end with reinforcement learning.
1) Waymo’s Core Learning Philosophy: “Infinite Expansion of Real-World Data + World-Model Scenarios”
1-1. The awareness that “Using only real-world data leaves risk behind”
What was emphasized in the interview is that, yes,
real-world driving data is incredibly important, but
there are structural limitations in covering “risky situations that can’t be encountered in the real world (or happen only very rarely).”
1-2. What the world model does: Generate sensor data to create training material
This is where the world model appears.
It’s said that, in general, the definition of a world model can still be ambiguous, but Waymo’s world model is specifically
a model that generates sensor data like camera and lidar.
In other words, its core role is to make it possible to use in training a simulation-ready form of “how the world looks (sensor inputs) and how it moves.”
1-3. Connecting the “number of scenarios” to autonomous driving performance
This is the most strongly striking part of the interview.
Waymo says that after creating truly many scenarios with the world model,it trains the driving agent across tens of thousands / hundreds of thousands / hundreds of millions of scenarios.
And the conclusion is clear.
“Train it perfectly in every scenario that can be imagined”—that’s what they see as the end of autonomous driving.
2) Summary of the Perspective: “Open-Loop Learning (World Model)” vs “Closed-Loop Learning (Fixed Environment)”
2-1. Limits of open-loop (fixed-environment) learning
The framework compared in the interview looks roughly like this.
- Open-loop training: Fix an environment such as a specific city/map and train the agent
- Example: Map Seoul, and if you crash into a wall you get a penalty; if you violate signals you receive reward/penalty, and so on
The problem is that with this approach, it’s difficult to create “many scenarios at the same time.”
2-2. Closed-loop (world-model-based) and scenarios explode in number
So Waymo explains that it creates scenarios first with a world model,and then moves toward a structure that puts the driving agent into large-scale reinforcement learning using those scenarios.
Why this matters in practice is that,
rare situations (unpredictable behaviors of other vehicles, sudden events that are hard to imagine)can be taught frequently and repeatedly.
3) Where foundation models (or giant models) enter autonomous driving
3-1. “LLMs are strong on reasoning”
The frame mentioned in the interview is simple.
Language models like LLMs are widely applied to reasoning.
3-2. The VLA flow: Connecting vision-language-action
The trend mentioned recently is the VLA (vision-language-action) viewpoint.
For example,if you see a red traffic light on the road, it means the system can connect to the interpretation+decision side—such as taking actions like stopping for that situation.
3-3. Why they want to expand rule-based approaches with foundation models
The core reason is that “it becomes more advantageous to train the driving agent in simulation/world-model environments.”
That means a model-based approach
can handle a wider variety of situationsmore effectively, and
it can also bring even scenarios that are hard to implement in the real world up as training material.
4) Where does Waymo place more weight: “R&D” vs “Product”?
4-1. The product image was stronger in the past, but recently they’ve greatly increased R&D too
According to the interview, there was an early 분위기 centered on product,but now that they’ve started using LM (large models), they answer that they’re also putting a lot of resources into R&D.
4-2. Summary of the atmosphere: “It’s tougher because it’s mission-driven and the cost of accidents is high”
As everyone knows, autonomous driving has a high risk of outcomes.So
because the reality is that if the LM fails, it can lead to an accident,they say the culture is strongly focused on the mission.
5) The infrastructure perspective on “making learning/development faster” (a story from an engineer’s experience)
5-1. Real-world problems in training large models: cost and efficiency
Engineer Tae-hwan Kim points to the difficulties he faced when working at Gemini:
the enormous compute cost required to train a single model.
Even if you optimize just 1%, the amount drops significantly, so
how to speed up training and increase training efficiency with fewer chipswas a crucial task, he says.
5-2. Bringing in an agent increases development speed, but “hallucination/verification costs” also grow
In the later part of the interview, you see practical work from the perspective of AI agent developers.
When you use an agent, information is compressed, so the coding process can be more efficient,but
because of the hallucination problem in LLM-based agents,
there are cases where it creates functions that don’t exist as if they do, or produces nonsensical results.
So you might waste 2–3 days, and
they summarize that you need a verification loop to collectively check whether the code actually exists (whether a function exists).
6) Hiring / ideal talent: “Reinforcement learning, physics/reward design” is more competitive than “API fine-tuning”
6-1. Simple “API-usage” engineers can be easier to replace
In the interview, they say very directly that “an engineer who just brings APIs and does fine-tuning only”has a high chance of being replaced by AI.
6-2. A competitive engineer: reinforcement learning + physical laws + reward design
Instead,
how to design rewards(reward) well in reinforcement learning
how to understand physical laws/environment constraints and incorporate them into the agent architecture
They believe that engineers with this kind of “their own expertise” are more competitive.
Main message they want to convey (“the real core” that often doesn’t get picked in other news)
The most important point in this interview isn’t the story about “data collection that every autonomous driving company is doing,” but rather
defining the “criteria for ending autonomous driving” as the completeness of scenario coverage.
That is,
- real-world data is essential, but there are limitations, so
- generate sensor-based training data with a world model, and
- repeat training with reinforcement learning across scenario counts from tens of thousands to hundreds of millions, and
- move toward a standard of “perfect across every situation that can be thought of”
This flow can also be read as a signal toward a future where, not only for autonomous driving but also for generative AI, “explosive productivity in training data” plays a role in real industries.
Connections seen through the lens of autonomous driving & AI trends (naturally reflecting 5 economic/industry keywords)
- Autonomous driving: An approach that extends limitations of real-world driving data through world-model simulation
- AI semiconductors: Optimization of training efficiency and chip cost repeatedly emerges as a key challenge
- Generative AI: The world model’s role in making “sensor input generation” to create training material
- Reinforcement learning: An engine that pushes performance beyond the constraints of open-loop and improves it across large-scale scenarios
- Foundation models: Suggesting a direction for expanding rule-based methods through reasoning and vision-action connections
< Summary >
– Waymo creates training scenarios at large scale not only with real road driving data, but also by generating sensor data using a world model.
– The goal is to “perform training perfectly in every scenario that can be imagined,” and they see that as the ‘end’ of autonomous driving.
– Open-loop (fixed-environment) reinforcement learning has limits in expanding scenarios, so there is a strong push to overcome it with a world-model-based approach.
– The reason for expanding from rule-based approaches to foundation-model-based approaches is also to train preparedness for diverse and rare situations.
– They emphasize that the ideal talent is less about only API fine-tuning and more about the ability to incorporate reinforcement learning, reward design, and physics/environment constraints into the architecture.
– AI agents can increase development efficiency, but hallucination verification costs (like checking for existence) also rise, making practical verification loops essential.
[Related Articles…]
- World-Model-Based Simulation Economics: How to Reduce Training Costs
- The Moment Reinforcement Learning Changes Autonomous Driving: Linking Reward Design and Safety
*Source: [ 티타임즈TV ]
– 웨이모가 자율주행의 끝으로 가는 방법 (웨이모 AI엔지니어 김태환)
