Define the promise to one caller
An interactive local service needs a response expectation before it needs a larger batch. Decide how long a caller may wait and what the application does when that expectation cannot be met. A request that starts late can be unacceptable even if the device processes many requests over a longer interval. Separate time spent waiting for admission from time spent executing. That distinction makes the cost of sharing a small accelerator visible to the person designing the service.
Bound work in more than one dimension
A request count alone may hide very different memory demands. For a language model, prompt length and output allowance matter; Hugging Face documents that generation caches can grow as tokens are added. For vision, image shape and the number of streams can change the load. Set limits that reflect the work the service actually receives. Record how an oversized request is handled before it reaches a resource-intensive stage, where rejecting it may already be too late to preserve responsiveness.
Choose an overload response
Waiting, rejecting and dropping old work each have legitimate uses. An interactive document assistant can ask a caller to retry; a live camera application may prefer the newest frame. Avoid an unlimited queue whose only policy is to consume whatever resources remain. Include cancellation and a maximum wait in the application design, and decide whether cancelled work can actually stop. Otherwise the visible queue may shrink while the device continues spending time on requests that no longer have a recipient.
Measure a mix of arrivals
Evaluate both isolated requests and a short burst that resembles expected use. Retain per-request waiting and completion times, not just an aggregate rate. Include long and short inputs so a favourable average cannot conceal a caller stuck behind expensive work. Change concurrency or batching only after the baseline is understood. The appropriate setting is the one that meets the service promise within its memory budget; maximum occupancy is not an independent application requirement.
Sources & publication notes
Retrospective period: October 2025. Written and published November 2, 2026. Product announcements establish release timing; support pages are linked as current references.