Start with a task, not a leaderboard
Write down the model, input size, target latency, and operating conditions. A board that excels at a supported vision network may be a poor match for a language model with a large context window.
Precision is part of the number
INT4 and INT8 refer to different numerical representations. Sparsity assumptions can also differ. Keep the precision, sparsity, and vendor’s measurement conditions beside every headline figure. Do not rank unlike figures as if they were one test.
Memory can decide feasibility
Weights, runtime overhead, activations, and cached context all use memory. Check whether the complete workload fits. If it does not, a theoretical compute advantage may never become a usable application.
Software compatibility comes first
A model must be supported by the hardware’s runtime and conversion tools. Look for the exact model family, operator coverage, quantization path, and a reproducible deployment example.
Measure the experience you need
For interactive text, capture time to first token and subsequent generation speed. For vision, measure end-to-end latency and missed frames. Add steady-state wall power and temperature to understand the cost of that experience.
Primary sources
Source check: November 1, 2026. Product and service details can change.