Edge AI Hardware: MCUs, NPUs, GPUs and FPGAs Compared

Think of a kitchen in Indore preparing poha. One cook with a knife can chop onions for four people. For a wedding of 400 guests, the same cook needs a vegetable chopper, and a caterer runs a commercial kitchen with a row of machines and a big power connection.

The right tool depends on the size of the job, the time you have and the electricity you can spare.

Edge AI hardware works the same way.

What is edge AI hardware?

Map the kitchen onto robots. Chopping onions is the repeated multiply-and-add maths inside a neural network. The cook with a knife is a general-purpose CPU. The chopper is a dedicated accelerator such as an NPU. The commercial kitchen is a GPU module, and the power connection is your robot's battery budget.

Edge AI hardware is the processors and accelerators that run trained machine learning models directly on a device, chosen to balance speed, power consumption, memory, cost and software support for a given model.

TinyML explained stayed at the smallest end. This article covers the whole range, so you can pick a board for a robot instead of guessing.

Why do neural networks need special hardware?

A neural network layer is mostly one pattern repeated millions of times: multiply an input by a weight and add it to a running total. This is called a MAC (multiply-accumulate). A small image classifier needs tens to hundreds of millions of MACs per frame.

A CPU core does a handful of MACs per clock cycle. An accelerator has hundreds or thousands of small multipliers side by side, so it finishes a frame in a fraction of the time and energy.

Doing it in 8-bit integers instead of 32-bit floats makes each multiplier smaller and cheaper, which is why most edge accelerators prefer quantised models (see model quantisation for the edge).

Speed is quoted in TOPS, trillions of operations per second, where one MAC counts as two operations.

The edge AI hardware ladder

Each step up multiplies compute and power by roughly ten; FPGAs sit to the side because they trade ease of programming for fixed, predictable latency.

Microcontrollers and micro-NPUs

Cortex-M4 and M7 chips have DSP instructions that the CMSIS-NN library uses for int8 maths. The Cortex-M55 was Arm's first Cortex-M core with Helium, a vector extension.

Paired with an Ethos-U micro-NPU, small vision and audio models run many times faster. Arm says the Ethos-U85 scales from 128 to 2,048 MACs, or 256 GOPS to 4 TOPS at 1 GHz, and adds operators used by transformer models.

Single-board computers and NPU add-ons

A Raspberry Pi 5 runs small int8 classifiers on its CPU. The official AI HAT+ adds a Hailo NPU rated at 13 or 26 TOPS.

The newer AI HAT+ 2 uses a Hailo-10H, rated at 40 TOPS for int4, with 8 GB of its own memory for small language models as well as vision. Rockchip RK3588 boards include a 6 TOPS NPU on the main chip.

GPU modules

NVIDIA's Jetson boards put a CUDA GPU and Arm CPU on one module. The Jetson Orin Nano Super Developer Kit (December 2024, US list price $249) is rated at 67 TOPS.

The Jetson AGX Thor developer kit (August 2025, $3,499 at launch) uses a Blackwell GPU, up to 128 GB of memory and a 40 to 130 W power range, and is quoted at 2,070 TFLOPS, a figure for FP4 with sparsity. Thor targets humanoids and the robot foundation models in embodied AI explained.

FPGAs

An FPGA is a chip whose circuits you configure yourself. It runs a vision pipeline with the same latency every frame and connects to unusual sensors directly, at low power. The cost is development effort with hardware tools rather than plain Python.

How fast will my model run? Estimate before you buy

Datasheet TOPS is a peak. Memory transfers, unsupported layers and pre-processing keep an accelerator only partly busy, so engineers estimate with a utilisation factor and then measure.

This calculator counts the MACs of a small detector-style network and estimates latency on five device classes. The effective-throughput figures are rough assumptions for illustration, not benchmarks.

# MACs of a conv layer = out_h * out_w * out_channels * (k * k * in_channels)
stages = [(160, 32, 3), (80, 64, 32), (40, 128, 64), (20, 256, 128), (10, 512, 256)]
macs = 0
for size, c_out, c_in in stages:          # 320x320 input, 3 conv layers per stage
    macs += size * size * c_out * 9 * c_in + 2 * size * size * c_out * 9 * c_out
ops = 2 * macs                            # one MAC = one multiply + one add
print(f"MACs per frame: {macs / 1e9:.2f} G  ->  {ops / 1e9:.1f} GOPs")

devices = {  # name: (peak TOPS, utilisation, fixed overhead ms, watts)
    "ESP32-S3 CPU":         (0.002, 0.5,  0, 0.3),
    "Cortex-M55 + Ethos-U": (0.25,  0.3,  2, 0.5),
    "Raspberry Pi 5 CPU":   (0.1,   0.4,  8, 7),
    "Pi 5 + 13 TOPS NPU":   (13,    0.2,  8, 9),
    "Orin Nano Super":      (67,    0.15, 4, 20),
}
for name, (tops, util, extra_ms, watts) in devices.items():
    ms = ops / (tops * 1e12 * util) * 1000 + extra_ms   # + pre/post-processing
    print(f"{name:22s} {ms:8.1f} ms/frame {1000 / ms:6.1f} fps "
          f"{1000 / ms / watts:6.2f} frames/J")

The 5.7 GOP network takes almost six seconds per frame on the ESP32-S3, about 150 ms on the Pi 5 CPU, about 10 ms with the NPU and under 5 ms on the Orin. On the NPU board, the fixed CPU overhead for resizing and decoding is most of the frame time.

And frames per joule, the number that matters on a battery, is about ten times better with the NPU than on the bare Pi CPU. The micro-NPU row looks efficient, but a 320 × 320 detector would not fit in its memory; such chips suit much smaller models.

Is my layer limited by compute or by memory?

An accelerator can only multiply as fast as numbers reach it. A useful measure is arithmetic intensity: operations performed per byte moved from memory. Convolution layers reuse each weight many times, so they have high intensity. Fully connected layers use each weight once per input, so they are often memory-bound, and adding TOPS does not help them.

peak_tops = 13            # accelerator peak, int8
bandwidth_gbs = 32        # memory bandwidth in GB/s (LPDDR5-class)

def attainable_tops(ops, bytes_moved):
    intensity = ops / bytes_moved                  # ops per byte
    return intensity, min(peak_tops, bandwidth_gbs * intensity / 1000)

# 3x3 conv, 56x56x64 -> 56x56x64, int8 weights and activations
conv_ops = 2 * 56 * 56 * 64 * 3 * 3 * 64
conv_bytes = 3 * 3 * 64 * 64 + 2 * 56 * 56 * 64
# Dense layer 4096 -> 4096 on a single input vector, int8
fc_ops = 2 * 4096 * 4096
fc_bytes = 4096 * 4096 + 2 * 4096

for name, o, b in [("3x3 conv", conv_ops, conv_bytes), ("dense 4096", fc_ops, fc_bytes)]:
    i, t = attainable_tops(o, b)
    kind = "compute-bound" if t >= peak_tops else "memory-bound"
    print(f"{name:11s} intensity {i:7.1f} ops/byte -> {t:6.2f} TOPS usable ({kind})")

The convolution can use the NPU's full 13 TOPS. The dense layer reaches only about 0.06 TOPS, because it spends its time waiting for weights. This is why large language models on edge boards are limited by memory bandwidth and memory size, and why the AI HAT+ 2 and Jetson boards advertise their memory as loudly as their TOPS.

How to choose edge AI hardware for a robot

Start from the sensor and the deadline, not from the most powerful board you can afford.

1. Write down the job and deadline, for example "detect a person in 640 × 480 video at 15 fps, within 100 ms".
2. Write down the budget: battery, watts, heat, weight and price.
3. Count the operations of a candidate model, as in the calculator above, and estimate with a cautious utilisation.
4. Check operator support. Layers the NPU cannot run fall back to the CPU and can wipe out the speed-up.
5. Check the software path. LiteRT, ONNX Runtime, OpenVINO (Intel CPUs, GPUs and NPUs; version 2026.3 at the time of writing), TensorRT on Jetson, or the NPU vendor's compiler. Pick hardware your team can actually program.
6. Buy one board and measure the whole pipeline, capture to post-processing.

Many robots use two levels: a microcontroller for motors and safety stops, and a Pi or Jetson for vision. The Arduino vs ESP32 vs Raspberry Pi guide covers the first level.

Common hardware-choice mistakes

• Comparing TOPS across precisions. An int4 or sparse TOPS figure is not comparable with a dense int8 one.
• Ignoring memory. A model that does not fit in RAM does not run, whatever the TOPS.
• Forgetting thermals. A board that throttles after five minutes in a closed robot body is slower than its benchmark.

Key takeaways

• Neural networks are mostly MACs; accelerators win by doing thousands in parallel, usually in int8.
• The ladder runs from MCUs (milliwatts) to micro-NPUs, Pi-class boards, NPU add-ons and GPU modules (tens of watts), with FPGAs beside it.
• TOPS is a peak. Utilisation, precision, sparsity and memory bandwidth decide real speed.
• Choose from the job and the power budget, check operator support, then measure on one board.