The file is only the first entry
The size of a model download is a starting point for planning, not an estimate of the complete running application. Make separate entries for model weights, runtime allocations, generation state and the services needed around inference. Then record which figures are estimates and which come from observation. This avoids turning a neat total into false precision. The important early result is discovering which components remain unknown before committing to a model or a maximum request size.
Context has its own cost
Hugging Face documents the key-value cache as stored attention information reused during generation, and notes that it can become a significant memory burden for long contexts. Treat the prompt and output limits as resource settings. A configuration that loads successfully can still fail when a longer request arrives. Keep a few deliberately different request lengths in the acceptance workload, including the longest request the application promises to support, rather than inferring that capacity from a short greeting.
Reserve room for useful surroundings
On a small edge system, the user interface, camera process, logging and remote access are part of the product. Write down the services that must remain responsive while generation runs. A terminal that becomes unusable during a difficult request is an operational problem even if tokens eventually arrive. Prefer explicit limits on accepted work to a memory plan that assumes every other process will remain idle. Keep recovery access in the workload definition, where it can be checked.
Change one budget line at a time
Begin with a modest context and one request, then expand the intended workload in controlled steps. Record peak usage and failures with the exact model and cache settings. Cache quantization or offloading may offer different tradeoffs where the model and runtime support them, but they need their own evaluation. Decide whether reducing context, using a smaller model or changing the cache policy best serves the task. The ledger should explain the choice without promising that compression creates unlimited capacity.
Sources & publication notes
Retrospective period: March 2025. Written and published November 2, 2026. Product announcements establish release timing; support pages are linked as current references.