Choose a beginning and an end
State what event starts the clock and what event stops it. An application might care about input arrival, while a runtime microbenchmark starts with a prepared tensor. Both can be useful if named clearly. End the application measurement when the result is available in the form its consumer needs. Otherwise preprocessing, transfers or output formatting can disappear from the figure while remaining fully visible to the user who is waiting for the response.
Account for asynchronous work
If an API returns before device execution is complete, the duration of that call alone cannot establish completed inference time. Use the runtime’s documented completion or synchronization mechanism for the measurement being made. Keep this separate from the application’s normal scheduling design, since forcing additional synchronization can change behaviour. ONNX Runtime’s profiling tools provide host and, with appropriate support, device-level information. Interpret those events within their timing boundaries instead of treating every duration in a trace as interchangeable.
Report a distribution with context
Retain individual request durations and separate initialization from steady operation. Include the input shapes and arrival pattern so the variation has a defined setting. A median describes an ordinary observation, while a slower portion of the distribution can expose a service problem that an average obscures. Do not discard slow requests without explaining the rule. Failed or timed-out work belongs in the application result, even when it cannot be assigned a normal successful completion time.
Check the instrumented result
Compare the profiled run with an otherwise equivalent run using lighter observation. Detailed tracing is useful for explanation, but the service requirement concerns the deployed application. Keep the exact workload and measurement definition with both records. When publishing a comparison, preserve these boundaries across devices and software changes. The reader should be able to tell whether a claimed improvement concerns dispatch overhead, model execution or the complete response; those are different achievements with different consequences for the project.
Sources & publication notes
Retrospective period: August 2026. Written and published November 2, 2026. Product announcements establish release timing; support pages are linked as current references.