Treat First-Frame Latency as Four Measurable Runtime Phases

Flow

Resident Slot Request Loop

A request either finds proven-compatible prepared state or takes an explicit slower path.

1. 1Parse requirements⌄

   Resolve model revision, precision, layout, and context identity.

2. 2Find resident slot⌄

   Select only a fully prepared compatible slot.

3. 3Reserve execution⌄

   Serialize or isolate route-sensitive activation.

4. 4Bind buffers⌄

   Activate device state without reloading the whole model.

5. 5Lookup context⌄

   Require exact cache identity; treat uncertainty as a miss.

6. 6Prefill missing work⌄

   Compute only state that could not be reused.

7. 7Decode and flush⌄

   Emit output while recording phase timings.

8. 8Release ownership⌄

   Return slot and memory resources safely.

A local runtime should not treat every generation request as a cold start.

The cold path is too expensive:

Illustrative anonymized example
request
  -> load model
  -> allocate host memory
  -> transfer weights
  -> bind device buffers
  -> build context
  -> prefill
  -> decode first token
  -> flush first chunk

The better path separates preparation from serving.

Before request:

Illustrative anonymized example
prefetch model state
reserve pinned host buffers
prepare device buffers
mark slot resident
maintain context pages

During request:

Illustrative anonymized example
route to slot
activate slot
look up context page
prefill missing state
decode first token
flush first chunk

The first design object is the model slot.

# Illustrative anonymized example
from dataclasses import dataclass
from typing import Literal

SlotStatus = Literal[
    "empty",
    "prefetching",
    "resident",
    "active",
    "evicting",
]

@dataclass(frozen=True)
class ModelSlot:
    slot_id: str
    model_revision: str
    precision: str
    layout_revision: str
    status: SlotStatus
    host_buffer_id: str | None
    device_buffer_id: str | None

A slot lets the runtime talk about prepared model state without pretending every request owns the model.

A request should not allocate and prepare everything from scratch. It should route to an existing resident slot when possible.

# Illustrative anonymized example
class SlotRegistry:
    def __init__(self) -> None:
        self._slots: dict[str, ModelSlot] = {}

    def require_resident(self, slot_id: str) -> ModelSlot:
        slot = self._slots[slot_id]

        if slot.status not in {"resident", "active"}:
            raise RuntimeError(f"slot {slot_id} is not resident")

        if slot.device_buffer_id is None:
            raise RuntimeError(f"slot {slot_id} has no device buffer")

        return slot

Pinned memory belongs to the preparation side.

# Illustrative anonymized example
@dataclass(frozen=True)
class PrefetchPlan:
    model_revision: str
    slot_id: str
    host_buffer_id: str
    target_device: str
    byte_count: int

def plan_prefetch(slot: ModelSlot, byte_count: int) -> PrefetchPlan:
    if slot.status not in {"empty", "evicting"}:
        raise ValueError(f"slot {slot.slot_id} is not available")

    host_buffer = pinned_pool.reserve(byte_count)

    return PrefetchPlan(
        model_revision=slot.model_revision,
        slot_id=slot.slot_id,
        host_buffer_id=host_buffer.buffer_id,
        target_device="cuda:0",
        byte_count=byte_count,
    )

The careful claim is:

Pinned memory can help when CPU-to-GPU transfer is part of the bottleneck.

Not:

Pinned memory is always faster.

Pinned memory can have costs. Overusing it can put pressure on host memory and reduce system flexibility. The runtime should treat it as a transfer strategy, not a universal optimization.

Activation is a separate phase from preparation.

# Illustrative anonymized example
class SlotRouter:
    def __init__(self) -> None:
        self._active_slot_id = "identity"
        self._activation_lock = AsyncLock()

    async def activate(self, slot_id: str) -> None:
        async with self._activation_lock:
            if slot_id == self._active_slot_id:
                return

            slot = slot_registry.require_resident(slot_id)

            await runtime.bind_device_buffers(slot.device_buffer_id)

            self._active_slot_id = slot_id

The lock matters.

If active_slot_id is process-global, concurrent generation can become unsafe. One request may activate slot A while another request is still decoding from slot B.

The runtime needs one of these disciplines:

• serialized generation
• lock around route-sensitive execution
• worker isolation
• request-scoped binding
• per-request device-buffer dispatch

Context paging is a cache, so identity must be strict.

# Illustrative anonymized example
@dataclass(frozen=True)
class ContextPageKey:
    model_revision: str
    tokenizer_revision: str
    layout_revision: str
    slot_revision: str
    page_index: int
    token_hash: str

Lookup should fail closed.

# Illustrative anonymized example
def get_context_page(key: ContextPageKey) -> "ContextPage | None":
    page = context_pages.get(key)

    if page is None:
        return None

    if page.layout_revision != key.layout_revision:
        return None

    return page

A false miss means the runtime recomputes.

A false hit can corrupt the output.

Strict identity is worth it.

Streaming should be tracked as a delivery phase.

# Illustrative anonymized example
async def stream_chunks(request: "GenerationRequest"):
    slot = await slot_router.route(request.model_hint)
    page_state = context_pager.lookup(request.context_key)

    async for chunk in runtime.generate_chunks(
        input_ids=request.input_ids,
        slot_id=slot.slot_id,
        page_state=page_state,
    ):
        yield {
            "type": "chunk",
            "slot_id": slot.slot_id,
            "text": chunk.text,
        }

Streaming improves perceived responsiveness, but it does not erase prefill, decode, or transfer work. Benchmarks should track both first chunk and full response.

A better benchmark shape:

Illustrative anonymized example
{
  "benchmark": "first-frame-runtime-path",
  "runs": [
    {
      "case": "cold_slot_no_context_page",
      "slot_hit": false,
      "context_page_hit": false,
      "phase_ms": {
        "route": null,
        "slot_activation": null,
        "context_lookup": null,
        "prefill": null,
        "decode_first_token": null,
        "first_chunk_flush": null
      },
      "first_chunk_ms": null,
      "full_response_ms": null
    },
    {
      "case": "resident_slot_context_page_hit",
      "slot_hit": true,
      "context_page_hit": true,
      "phase_ms": {
        "route": null,
        "slot_activation": null,
        "context_lookup": null,
        "prefill": null,
        "decode_first_token": null,
        "first_chunk_flush": null
      },
      "first_chunk_ms": null,
      "full_response_ms": null
    }
  ]
}

The engineering point:

Do not optimize latency as one vague number. Break it into route, activation, context lookup, prefill, decode, and flush phases.

Illustrative anonymized example
Illustrative anonymized example:

# Illustrative anonymized example
def handle(request, cache, queue):
    result = cache.get(request.resource_key)
    if result is None:
        queue.enqueue({"kind": "prepare", "key": request.resource_key})
        return {"status": "pending"}
    return {"status": "ready", "result": result}