Embodied AI Explained: Robot Foundation Models and VLAs

Think of how a child in an Indore kitchen learns to roll a roti. Nobody hands her a manual on dough elasticity. She watches her mother a dozen times, tries, presses too hard, gets a torn roti, adjusts, and after a few weeks her hands simply know.

Reading every recipe online would not teach her that.

That is the problem embodied AI tries to solve for robots.

What is embodied AI?

Map the kitchen onto a robot. The child's hands and eyes are the robot's body: cameras, joints and a gripper. Watching her mother is learning from demonstrations. The torn roti is feedback from the physical world. Years of general life experience that make the next skill easier are what a foundation model brings.

Embodied AI is artificial intelligence that has a physical body, perceives its surroundings through sensors, acts through motors, and improves its behaviour from the consequences of those actions in the real world.

A chatbot can write an essay about folding a shirt; an embodied AI system has to fold it, with cloth that slides and a view that keeps changing. This is Moravec's paradox: what is easy for humans, like grasping a cup, is often harder for machines than chess.

What are robot foundation models?

For decades, each robot skill was programmed or trained separately. A robot foundation model is pretrained once on a very large and varied dataset, often from many different robots, and then adapted to a new robot or task with a small amount of extra data.

Two things made this possible. Vision-language models learnt general knowledge of objects and words from web data. And shared robot datasets appeared, such as Open X-Embodiment (2023), which pooled over a million robot episodes from 22 robot types across many labs.

What is a vision-language-action (VLA) model?

A vision-language-action model takes camera images, a natural-language instruction and the robot's current state as input, and outputs the robot's next actions, such as joint positions or gripper commands.

The backbone brings general knowledge; the action head turns it into motor commands the controller can follow.

Inside, most VLAs have three parts:

1. A vision encoder turns each camera image into a set of feature vectors.
2. A language model backbone combines those features with the instruction, for example "put the red cup in the tray".
3. An action head produces actions. Early VLAs wrote actions as tokens, just like words. Newer ones attach an "action expert" that predicts a smooth chunk of continuous actions using diffusion or flow matching.

How does a model write actions as tokens?

The token approach is simple: scale each action dimension into a fixed range, cut it into 256 bins, and use the bin number as a token. The language model then "writes" the action the way it writes a word.

import numpy as np
# One robot-arm action: change in x, y, z (m), roll, pitch, yaw (rad), gripper (0-1)
action = np.array([0.012, -0.004, 0.020, 0.05, -0.02, 0.10, 1.0])
low = np.array([-0.05, -0.05, -0.05, -0.25, -0.25, -0.25, 0.0])   # from the dataset
high = np.array([0.05, 0.05, 0.05, 0.25, 0.25, 0.25, 1.0])
bins = 256

def to_tokens(a):
    norm = (np.clip(a, low, high) - low) / (high - low)          # 0..1
    return np.minimum((norm * bins).astype(int), bins - 1)       # 0..255

def from_tokens(tok):
    return low + (tok + 0.5) / bins * (high - low)               # bin centres

tokens = to_tokens(action)
decoded = from_tokens(tokens)
print("action tokens :", tokens)
print("decoded action:", np.round(decoded, 4))
print("max error per dimension:", np.round(np.abs(decoded - action), 4))
print(f"position resolution: {(high[0] - low[0]) / bins * 1000:.2f} mm per bin")

Seven numbers become seven tokens, and decoding them gives back the action to within about 0.4 mm of position resolution. That is fine for picking up a cup and too coarse for threading a needle, which is one reason newer models predict continuous actions instead.

Why do VLAs predict action chunks?

A large model might take 100 to 300 ms to think, but a robot arm wants a new command 50 times a second or more.

If the robot waits for each answer, it moves in jerky steps. The common fix is action chunking: every inference returns the next several actions, and the robot executes them while the next chunk is being computed.

import numpy as np
rng = np.random.default_rng(2)
control_hz, infer_ms, chunk = 50, 150, 16  # 50 Hz motors, 150 ms model, 16-step chunks
k = int(np.ceil(infer_ms / (1000 / control_hz)))
print(f"control steps that pass during one inference: {k}")

t = np.arange(0, 2, 1 / control_hz)
ideal = 0.5 * (1 - np.cos(np.pi * t / 2))           # smooth 2 s joint motion (rad)

# Single-step policy: one target per inference, held until the next arrives
single = ideal[(np.arange(len(t)) // k) * k]
# Chunked policy: each inference predicts the next 16 targets (with small errors);
# the robot executes them while the next chunk is computed, then switches over
chunked = np.empty_like(ideal)
for start in range(0, len(t), k):
    pred = ideal[start:start + chunk] + rng.normal(0, 0.003, len(ideal[start:start + chunk]))
    chunked[start:start + k] = pred[:k]

for name, path in [("single-step", single), ("chunked", chunked)]:
    jumps = np.degrees(np.abs(np.diff(path)))
    err = np.degrees(np.abs(path - ideal)).mean()
    print(f"{name:12s} largest jump {jumps.max():5.2f} deg/step, "
          f"mean tracking error {err:4.2f} deg")

With one action per inference, the joint jumps by over 7 degrees at a time and lags the intended path. With chunks, the robot always has fresh targets, the largest jump drops to about 1.4 degrees and the tracking error falls more than tenfold. Real systems also blend overlapping chunks so the hand-over between them is smooth.

Fast and slow thinking: dual-system designs

Each layer runs at the speed its job needs, much like the edge-cloud tiers.

Several recent models split the work in two. A slower reasoning part (a vision-language model) understands the scene and the instruction a few times a second. A faster action part turns that understanding into motor chunks many times a second.

Below both, the robot's ordinary motor controllers still run their own loops at around 1 kHz, as in PID control explained. NVIDIA describes its Isaac GR00T N models this way, and Google DeepMind pairs a reasoning model (Gemini Robotics-ER) with its action models.

Notable robot foundation models (as of November 2026)

The field moves monthly, so check each model card before you build on one. The robotics and edge AI trends 2026 report tracks the news.

Running robot foundation models on edge hardware

A VLA has to run where the robot is, because its loop is too fast for an internet round trip (see edge-cloud robotics). That makes memory the first question, and memory bandwidth the second: every forward pass reads the weights.

# Can a robot foundation model fit and run on an edge board?
models = {"SmolVLA": 0.45e9, "GR00T N1.6": 3e9, "OpenVLA": 7e9}   # parameters
precisions = {"fp16": 2, "int8": 1, "int4": 0.5}                   # bytes/param
boards = {"Orin Nano Super": (8, 102), "AGX Thor": (128, 273)}      # GB RAM, GB/s

for model, params in models.items():
    for prec, b in precisions.items():
        gb = params * b / 1e9
        cells = []
        for board, (ram, bw) in boards.items():
            fits = gb < 0.6 * ram          # leave room for OS, camera, ROS 2
            passes = bw / gb               # full weight reads per second (upper bound)
            cells.append(f"{board}: {'fits' if fits else 'no  '} {passes:6.0f}/s")
        print(f"{model:10s} {prec:4s} {gb:5.2f} GB | " + " | ".join(cells))

A small model such as SmolVLA fits comfortably on an 8 GB Jetson Orin Nano Super. A 3-billion-parameter model fits once quantised to int8 or int4. A 7-billion-parameter model needs int4 there, or a bigger board such as the Jetson AGX Thor.

The "/s" column is an upper bound on full forward passes per second set by memory bandwidth alone; real numbers are lower, which is exactly why chunking matters. The model quantisation article explains the precision trade-off.

How do robots learn new skills from data?

Most VLA fine-tuning today is imitation learning: a person controls the robot by teleoperation, for example moving a "leader" arm that a "follower" arm copies, while cameras and joint sensors record everything. Fifty or so good demonstrations of a task are a common starting point for a small model.

With Hugging Face's open-source LeRobot library (v0.6, July 2026), fine-tuning SmolVLA on your recorded dataset is one command. Flags change between releases, so copy the current one from the LeRobot SmolVLA docs.

pip install -e ".[smolvla]"          # inside a clone of the lerobot repository
lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=${HF_USER}/pick_cup_50_episodes \
  --policy.repo_id=${HF_USER}/smolvla_pick_cup \
  --batch_size=64 \
  --steps=20000 \
  --output_dir=outputs/train/smolvla_pick_cup \
  --policy.device=cuda

Two other ideas complete the picture. Simulation generates huge amounts of cheap practice data, and domain randomisation (varying lighting, textures and physics) helps skills survive the move to the real world, the so-called sim-to-real gap.

Reinforcement learning lets a robot improve by trial and reward, usually in simulation first. Fleets can even learn together without sharing raw video, using federated learning.

Common misconceptions about embodied AI

• "A VLA replaces classic robotics." It sits on top of motor control, kinematics and safety limits, which still have to be engineered.
• "Bigger models are always better." A small fine-tuned model on good demonstrations often beats a large one on poor data.
• "Simulation is enough." Contact, friction and lighting differ in reality; always validate on the real robot.

Key takeaways

• Embodied AI learns by perceiving and acting through a body; physical skills are hard for machines.
• Robot foundation models are pretrained on many robots' data and fine-tuned with few demonstrations.
• VLAs combine a vision encoder, a language backbone and an action head that outputs tokens or continuous chunks.
• Action chunking and fast-slow designs bridge slow inference and fast control.
• On the edge, memory size and bandwidth decide which models fit; quantisation and small models help.