Compact
Addressable agent slots replace dense future-frame tokens as the planning interface.
NeurIPS 2026
AgentWorld
MoE Key Lab of Artificial Intelligence, Institute of AI
Shanghai Jiao Tong University
A compact world state. Grounded in agents. Built for driving.
The idea
What world state should a VLA world model reason over to support planning beyond pixel-level prediction?
Recent vision-language-action (VLA) world models reason about imagined futures, but dense visual tokens preserve many appearance details that are irrelevant to driving. AgentWorld instead represents the scene as compact, structured, temporally evolving agents.
Each discrete agent token is dynamically grounded to a surrounding participant and encodes semantic identity, 3D geometry, motion, and interaction context. Through explicit geometric grounding, reasoning-oriented supervised fine-tuning, and agent-oriented reinforcement learning, these tokens become a shared interface for reasoning and trajectory planning. Open-loop and closed-loop experiments show stronger planning and more interpretable world states.
Recent progress in integrating vision-language-action (VLA) models with world modeling has significantly advanced end-to-end autonomous driving by allowing systems to reason about imagined futures instead of merely reacting to current observations. However, most world VLA models generate future frames at the pixel level with plenty of dense visual tokens. Although such generated futures can be visually plausible, they preserve many driving-irrelevant appearance details, i.e., only a small portion of the tokens directly encode critical planning-relevant factors, such as agent geometry, motion dynamics, and interactions. This introduces a significant performance bottleneck. An important question thus arises: what world state should a world VLA model reason over to support planning beyond pixel-level prediction? We argue that planning-oriented world modeling should focus on a compact and structured agent-centric latent state that captures the interactions between objects in the driving scene over time. As such, we introduce AgentWorld, an agent-centric VLA world model that uses discrete agent tokens jointly for reasoning and trajectory planning. Each token is dynamically grounded to a surrounding agent associated with its object’s state, such as semantic identity, 3D geometry, motion, and interaction context. AgentWorld learns these tokens through three key processes: explicit geometric grounding, reasoning-oriented supervised fine-tuning, and agent-oriented reinforcement learning. Extensive open-loop and closed-loop experiments under diverse driving scenarios show that AgentWorld enhances planning capabilities, fully exploits visual evidence and agent dynamics, and yields interpretable agent-centric world states.
01 / A different world state
Represent the participants, their motion, and their interactions in a form the policy can reason over.
Addressable agent slots replace dense future-frame tokens as the planning interface.
Semantics, geometry, and future trajectories tie latent states to real traffic participants.
Agent-grounded reasoning connects surrounding participants to the ego trajectory.
02 / Method
Three learning stages turn discrete agent tokens into a useful world state for driving.
Anchor agent queries to offline 4D geometry and 2D visual features. Hungarian matching supervises category, 3D boxes, and future motion.
Teach the policy to identify relevant agents, explain interaction risks, and produce high-level actions and future ego waypoints.
Use GRPO with format, perception, action, trajectory, and interaction rewards, including collision, clearance, and time-to-collision terms.
Multimodal routed anchoring
The geometry branch provides spatial structure and temporal consistency; the vision branch provides appearance and semantic context. Independent token-wise gates and cross-attention fuse both into grounded agent tokens.
At inference, the policy uses the MLLM-only input-output format. Auxiliary grounding heads and routed anchoring modules are not required.
03 / Experiments
Evaluated on nuScenes for open-loop planning and Bench2Drive for closed-loop driving.
| Method | Driving Score ↑ | Success Rate (%) ↑ | Efficiency ↑ | Comfortness ↑ |
|---|---|---|---|---|
| AD-MLP | 18.05 | 0.00 | 48.45 | 22.63 |
| UniAD-Base | 45.81 | 16.36 | 129.21 | 43.58 |
| VAD | 42.35 | 15.00 | 157.94 | 46.01 |
| TCP-traj | 59.90 | 30.00 | 76.54 | 18.08 |
| DriveAdapter | 64.22 | 33.08 | 70.22 | 16.01 |
| Orion | 77.74 | 54.62 | 151.48 | 17.38 |
| AutoVLA | 78.84 | 57.73 | 146.93 | 39.33 |
| AgentWorld Ours | 81.17 | 61.36 | 147.82 | 54.39 |
Table 4 of the paper. Best values among the compared methods are bold. Higher is better for all four metrics.
| Method | ST-P3 metrics | UniAD metrics | ||
|---|---|---|---|---|
| L2 (m) ↓ | Collision (%) ↓ | L2 (m) ↓ | Collision (%) ↓ | |
| OmniDrive | 0.84 | 0.94 | — | — |
| FSDrive | 0.53 | 0.17 | 0.83 | 0.16 |
| VLA-World | 0.30 | 0.10 | 0.45 | 0.12 |
| AgentWorld | 0.27 | 0.08 | 0.45 | 0.12 |
| OmniDrive* | 0.33 | 0.30 | — | — |
| FSDrive* | 0.28 | 0.10 | 0.42 | 0.12 |
| VLA-World* | 0.26 | 0.08 | 0.42 | 0.12 |
| AgentWorld* | 0.25 | 0.07 | 0.40 | 0.11 |
Selected methods from Table 1. * uses additional ego-state information. — indicates an unreported result. Compare methods within the same ego-state setting and evaluation protocol.
Compact by design
The visual grounding ablation measures planning with and without images. AgentWorld shows a larger performance gap when visual evidence is removed, while using a shorter output sequence.
Latency: 0.31 s for AgentWorld vs. 0.40 s for VLA-World in the paper’s Table 7 setting.
04 / Qualitative results
Agent tokens make the participants influencing a driving decision explicitly addressable.
Inside a reasoning trace
The policy can refer to a specific agent and connect its predicted behavior to the planned maneuver.
Excerpt from Scene 2 in Figure 5; emphasis added.
“Main risk is<agent_2>and<agent_3>, two cars ahead-right with small future clearance and likely path proximity as the ego moves forward.<agent_1>is a lead car in front but keeps a stable gap.”
Reference
@inproceedings{wang2026agentworld,
title={Learning Agent-centric Vision-Language-Action World Models for Autonomous Driving},
author={Wang, Guoqing and Tang, Pin and Ren, Xiangxuan and Ma, Chao},
booktitle={Advances in Neural Information Processing Systems},
year={2026}
}