Edge NPU benchmarks lie. Here is why.
Vendor datasheets for edge NPUs list TOPS and power. If you are picking hardware for an inference appliance, those figures do not tell you what you need to know. A paper posted to arXiv on 10 November 2026 shows that the same NPU can rank first or last on the same workload depending on how the test is built.
The paper is “Benchmarking Vendor-Guided Optimizations on Edge NPUs: From NPU Only to End-to-End Performance” (arXiv 2611.13563), from Youngjin Jo and colleagues. They ran four NPUs against three vision workloads: ResNet50 for classification, YOLOv5m for detection, YOLOv8s-pose for pose estimation. Each board was measured in two configurations. The Vendor Guided NPU Only (VG-NPU) mode keeps model input and output handling plus host-side processing identical across all four boards. The Vendor Guided End-to-End (VG-E2E) mode follows each vendor’s own deployment pipeline, host-side steps included.
| NPU | INT8 TOPS | Host |
|---|---|---|
| Hailo-8 | 26 | Jetson Orin Nano, CPU pinned at 1.728 GHz |
| Hailo-10H | 20 | Jetson Orin Nano, CPU pinned at 1.728 GHz |
| DEEPX DX-M1 | 25 | Jetson Orin Nano, CPU pinned at 1.728 GHz |
| Mobilint Regulus | 10 | Vendor SOM, Yocto Linux, 1.2 GHz |
NPU-level rankings do not hold at the system level
Under VG-NPU, accuracy and runtime-call latency both improved in 10 of the 12 NPU and workload combinations. Switch to the full vendor pipeline and system-level latency fell in all 12, by an additional 10.4 to 70.7 percent over what VG-NPU achieved. The authors singled out YOLOv5m and YOLOv8s-pose on the Hailo-10H, all three workloads on the DX-M1, and YOLOv8s-pose on the Regulus as the largest gains.
The raw numbers show why the choice of test matters. Hailo-8 running YOLOv5m went from 43.1 ms at VG-NPU to 28.1 ms end to end. Hailo-10H on the same workload went from 114.8 ms to 33.7 ms. DX-M1 running ResNet50 went from 17.2 ms to 7.2 ms. Vendor pipelines change how the host handles pre-processing and post-processing, and how memory transfers are scheduled, so a slower accelerator with a tighter host path can finish first.
The paper’s own conclusion is blunt: NPU-level performance rankings do not always match end-to-end rankings.
The accuracy cost is invisible in a latency chart
End-to-end deployment improved accuracy in four of the 12 combinations. In the others it gave some back. Hailo-8 on YOLOv5m scored 41.68 at baseline, 43.33 under VG-NPU, then 39.21 under the vendor pipeline, which is below where it started. Hailo-10H on YOLOv8s-pose moved 50.58, 52.49, then 47.84. Those are the same runs that produced the latency wins in the table above.
For an integrator choosing between a Hailo-8 M.2 card and a DX-M1 for a multi-camera recorder, the chip-to-chip comparison tells you which accelerator is faster at matrix multiplication. It does not tell you the latency of a frame entering your pipeline and a detection leaving it.
What to test before you commit
Measure the whole path on the target host. Frame in, result out, at your resolution, with your model. NPU-only inference time is the number in the datasheet.
Quantise with the vendor’s own toolchain and revalidate accuracy on the target board rather than on the machine you trained on. Check the confidence score distribution, not only the mean average precision, because a model that compiles and loads can still return worse predictions than the float version it came from. The paper calibrates its models with 1,024 images and evaluates on 5,000.
Ask for the vendor’s deployment pipeline, or build it, then measure it yourself. The four platforms tested all run in the sub-40 ms range on some combination and past 100 ms on another, which is a wider spread than the TOPS ratings suggest.
Sources
- Youngjin Jo, Jihwan Kim and others, “Benchmarking Vendor-Guided Optimizations on Edge NPUs: From NPU Only to End-to-End Performance,” arXiv 2611.13563, submitted 10 November 2026. https://arxiv.org/abs/2611.13563
- “Edge-LLM-Bench: Characterising LLM Inference on Raspberry Pi 5 with Hailo-10H NPU Acceleration,” ACM MobiSys 2026 workshops, DOI 10.1145/3812836.3814998.