# RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

## TL;DR

RT-2 is the first large-scale Vision-Language-Action (VLA) model that directly applies internet-scale VLM knowledge to robotic control. By representing robot actions as text tokens and co-fine-tuning PaLI-X (55B) and PaLM-E (12B) on web and robot data together, RT-2 achieves over 3× generalization improvement versus RT-1 across 6,000 real robot trials — and spontaneously develops **emergent reasoning capabilities never present in robot training data.**

---

## Background: The Question After RT-1

RT-1 proved that scale works in robot learning. 130K demos, 700+ tasks, 97% success rate. But it had a fundamental limitation.

RT-1 couldn't understand concepts outside its training data. A command like "give the tired person a drink" requires knowing that energy drinks help with fatigue — knowledge that doesn't exist in robot demonstrations.

**RT-2's question: Can a large VLM already trained on trillions of web tokens transfer that knowledge into robot actions?**

The answer is yes — and it exceeded expectations.

---

## Core Idea: Actions as Text Tokens

The key insight of RT-2 is simple but powerful.

Robot actions (arm positions, gripper state, etc.) are **discretized into string tokens**. Moving the arm 0.03m along the x-axis becomes text token `256`. This transforms robot control into the **next-token prediction problem** that language models already excel at.

This means you can take a massive VLM off the shelf and **co-fine-tune** it on robot data. Web knowledge and robot behavior coexist inside a single model.

---

## Architecture: PaLI-X & PaLM-E as Backbone

RT-2 experiments with two base models:

| Base Model | Parameters | Characteristics |
|---|---|---|
| **PaLI-X** | 5B / 55B | Multilingual VLM, image-text understanding |
| **PaLM-E** | 12B | Embodied multimodal LLM |

![RT-2 Architecture: VLM backbone + action tokenization](https://i.imgur.com/YSittzj.png)
*Figure 1. RT-2 architecture — end-to-end VLA that takes image + text input and outputs action tokens*

### Input Structure

- **Image**: Robot camera (front view)
- **Text command**: Natural language instruction
- **Output**: Text token sequence (last N tokens = robot actions)

Where RT-1 had EfficientNet + TokenLearner + Transformer, RT-2 is just a single massive VLM. The architecture is actually simpler.

### Action Tokenization

Action space:
- Arm 7-DOF (x, y, z, roll, pitch, yaw, gripper): 256 bins each
- Base movement: 3-DOF

Each bin becomes an integer token → added to VLM vocabulary → treated identically to text tokens.

### Co-fine-tuning

The critical detail: RT-2 doesn't fine-tune on robot data alone. It trains on **web VQA data and robot trajectory data simultaneously**. This prevents catastrophic forgetting of web knowledge.

---

## Evaluation

6,000 real robot trials across three evaluation axes:

### 1. Standard Task Performance

| Model | Success Rate |
|---|---|
| RT-1 (seen tasks) | 97% |
| RT-2-PaLI-X 55B | **~90%** |
| RT-2-PaLM-E 12B | ~83% |

![RT-2 performance results across evaluation categories](https://i.imgur.com/xK7NPzQ.png)
*Figure 4. Success rate across evaluation categories — RT-1-level on seen tasks, 3× improvement on novel scenarios*

Slightly lower than RT-1 on familiar tasks. But RT-2's value is elsewhere.

### 2. Generalization to Novel Scenarios

| Condition | RT-1 | RT-2-55B |
|---|---|---|
| Novel objects | ~32% | **~62%** |
| Novel backgrounds | ~40% | **~69%** |
| Novel commands | ~23% | **~59%** |

![RT-2 generalization to novel objects, backgrounds, and commands](https://i.imgur.com/nEdOO68.png)
*Figure 2. Generalization evaluation — RT-2 handles unseen objects, backgrounds, and commands robustly*

**3× improvement.** RT-1 collapses outside its training distribution. RT-2 handles it robustly due to web knowledge transfer.

### 3. Emergent Capabilities — The Core Finding

This is what makes RT-2 remarkable. Capabilities **never present in robot training data** emerged spontaneously:

**Symbol Understanding**
"Place the object on the number 3" — robot data has no concept of written numbers, but the VLM recognizes them visually.

**Cross-concept Reasoning**
"Pick up a drink appropriate for someone who is drunk" → selects non-alcoholic beverage
"Give something to a tired person" → picks up energy drink

![RT-2 emergent capabilities evaluation](https://i.imgur.com/fK8Z1Zt.png)
*Figure 6. Emergent capabilities evaluation — success rates on concepts absent from robot training data (symbols, commonsense reasoning)*

**Chain-of-Thought** (RT-2-CoT)
Before acting, generates reasoning: "Plan: Energy drinks help with fatigue, therefore select energy drink. Action: [tokens]"

---

## Chain-of-Thought Extension

RT-2-CoT operates as:

```
Input: image + "What drink would help someone who is tired?"
Output: "Plan: Energy drinks help with fatigue. Action: 238 211 ... [action tokens]"
```

Adding CoT **meaningfully improves success on tasks requiring multi-step reasoning** — directly foreshadowing OpenVLA, π0, and Embodied CoT research.

---

## Scale Effects: 55B vs 5B

Larger base model consistently wins:
- PaLI-X 55B > PaLI-X 5B on emergent tasks
- Gap is largest on novel concept reasoning

Same robot data, different base model size → different emergent capabilities. **LLM scaling law applies to VLAs.**

---

## Limitations — A Field Engineer's View

**Paper acknowledges:**
1. Inference speed: 55B model too slow for real-time control
2. Still needs robot data: web knowledge alone is insufficient
3. Single robot morphology: Everyday Robots only

**Field perspective:**
- **Latency**: Running 55B on an onboard computer is impractical. Requires separate inference server + network latency management
- **Coarse action resolution**: Text token discretization still suffers from the same 256-bin problem as RT-1 — insufficient for fine manipulation (π0 addresses this with diffusion)
- **Catastrophic forgetting risk**: Getting the co-fine-tuning ratio wrong destroys web knowledge — hyperparameter sensitivity is real

---

## The Lineage — What RT-2 Enabled

| Paper | What it took from RT-2 |
|---|---|
| **RT-X** (2023) | Cross-embodiment + RT-2 architecture |
| **Octo** (2023) | Open-source VLA, democratizing RT-2 paradigm |
| **OpenVLA** (2024) | 7B LLM + RT-2 co-fine-tuning recipe |
| **π0** (2024) | VLM backbone + diffusion action head to solve resolution problem |
| **Embodied CoT** (2024) | Extended RT-2-CoT into deeper reasoning chains |

RT-2's proof: **LLM knowledge transfers to robotic action.** This single finding redirected the entire field.

---

## Summary — Key Takeaways

1. **Web knowledge transfers to robots** — co-fine-tuning alone unlocks emergent capabilities
2. **Actions as text tokens = plug-and-play with existing LLM infrastructure**
3. **Bigger base model = stronger emergent capabilities** — LLM scaling applies to VLA
4. **Chain-of-thought works for robot reasoning** — generating rationale before acting improves complex tasks
5. **RT-2 is the VLA paradigm shift** — from task-specific to general-purpose robot intelligence

---

📄 **Paper**: [arXiv:2307.15818](https://arxiv.org/abs/2307.15818)
🌐 **Project**: [robotics-transformer2.github.io](https://robotics-transformer2.github.io)

*Next: FAST-LIO2 — High-speed precision odometry through LiDAR-IMU fusion*