Four Boundaries for Making Transformer Runtime Research Measurable

Flow

Boundary-First Experiment Sequence

Later optimizations consume contracts and evidence established by earlier stages.

1. 1Instrument split forward⌄

   Define intervals, tensor schemas, and per-phase timings.

2. 2Validate route identity⌄

   Carry adapter choice through cache, prefill, decode, and streaming.

3. 3Model residency⌄

   Represent prefetch, resident slots, activation, and eviction explicitly.

4. 4Capture reference traces⌄

   Bind middle-layer inputs and outputs to exact model identity.

5. 5Train candidate transition⌄

   Fit only against validated trace partitions.

6. 6Compare downstream behavior⌄

   Evaluate logits, rollouts, tasks, latency, and memory.

7. 7Promote incrementally⌄

   Use explicit gates, limited rollout, and rollback.

This article maps a sequence of transformer runtime experiments.

The common pattern is not "make it faster."

The common pattern is:

Illustrative anonymized example
make the hidden boundary explicit
give it identity
measure it
test it
then consider replacing or optimizing it

The four stages are:

1. split forward -> layer boundary
2. adapter routing -> route boundary
3. pinned-memory slots -> residency boundary
4. recurrent middle -> transition boundary

Stage 1: Split Forward

The first boundary is the model execution interval.

Instead of treating the model as:

# Illustrative anonymized example
logits = model(input_ids)

we split execution:

# Illustrative anonymized example
from dataclasses import dataclass

@dataclass(frozen=True)
class ForwardSplit:
    model_revision: str
    tokenizer_revision: str
    architecture_revision: str
    layout_revision: str
    precision: str
    early_end_layer: int
    middle_end_layer: int

Teaching pseudocode:

# Illustrative anonymized example
def split_forward(model, input_ids, split: ForwardSplit):
    early_state = model.run_layers(
        input_ids=input_ids,
        start=0,
        end=split.early_end_layer,
    )

    middle_state = model.run_layers(
        input_ids=early_state,
        start=split.early_end_layer,
        end=split.middle_end_layer,
    )

    logits = model.run_layers(
        input_ids=middle_state,
        start=split.middle_end_layer,
        end=model.num_layers,
    )

    return logits

The point is not that this is production code.

The point is that a layer interval is now an explicit object.

Once this exists, the runtime can ask:

• How expensive is the early phase?
• How expensive is the middle phase?
• How expensive is the late phase?
• Can the middle state be traced?
• Can a boundary be cached?
• Can a candidate block be tested at this interface?

The split creates the condition for later experiments.

Stage 2: Adapter Routing

The second boundary is behavior selection.

A slow model says:

Illustrative anonymized example
different behavior -> different model load

A better adapter runtime says:

Illustrative anonymized example
different behavior -> different route through resident base model

Route object:

# Illustrative anonymized example
@dataclass(frozen=True)
class AdapterRoute:
    name: str
    adapter_revision: str
    anchor_revision: str
    compatible_model_revision: str

Selector:

# Illustrative anonymized example
def select_adapter_route(requested: str | None) -> AdapterRoute:
    route_name = requested or "identity"

    if route_name not in adapter_routes:
        raise ValueError(f"unknown adapter route: {route_name}")

    return adapter_routes[route_name]

Generation path:

# Illustrative anonymized example
async def generate_with_route(request):
    route = select_adapter_route(request.adapter)

    async with runtime.bind_adapter(route.name):
        async for token in runtime.generate_stream(
            input_ids=request.input_ids,
            adapter_revision=route.adapter_revision,
        ):
            yield token

The route has to remain stable across the whole generation path:

Illustrative anonymized example
request route
= cache lookup route
= prefill route
= decode route
= streaming route

This stage turns behavior into a request boundary.

Stage 3: Pinned-Memory Slots

The third boundary is model residency.

If every request performs model preparation, the user waits for work that could have happened earlier.

Slot state makes that visible:

# Illustrative anonymized example
@dataclass(frozen=True)
class SlotResidency:
    slot_id: str
    model_revision: str
    layout_revision: str
    precision: str
    host_buffer_id: str | None
    device_buffer_id: str | None
    status: str

Activation check:

# Illustrative anonymized example
def can_activate(
    slot: SlotResidency,
    requested_model_revision: str,
) -> bool:
    return (
        slot.model_revision == requested_model_revision
        and slot.device_buffer_id is not None
        and slot.status == "resident"
    )

This separates:

• preparation
• residency
• activation
• generation
• streaming

A useful critical-path trace:

Illustrative anonymized example
{
  "critical_path_trace": {
    "prefetch_before_request": true,
    "slot_resident": true,
    "activation_ms": null,
    "prefill_ms": null,
    "decode_first_token_ms": null,
    "first_chunk_flush_ms": null
  }
}

The null values matter. They show the schema without inventing benchmark numbers.

This stage turns model readiness into a measurable boundary.

Stage 4: Recurrent Middle

The fourth boundary is the model's internal transition.

This is the riskiest stage because it changes computation inside the model.

The safe framing is:

Illustrative anonymized example
reference transition:
  hidden_start -> transformer middle -> hidden_end

candidate transition:
  hidden_start -> recurrent middle -> candidate_hidden_end

Experiment object:

# Illustrative anonymized example
@dataclass(frozen=True)
class MiddleTransitionExperiment:
    model_revision: str
    tokenizer_revision: str
    architecture_revision: str
    layout_revision: str
    precision: str
    layer_start: int
    layer_end: int
    candidate_revision: str

Comparison shape:

# Illustrative anonymized example
def compare_middle_transition(
    experiment: MiddleTransitionExperiment,
    trace,
):
    validate_trace_identity(experiment, trace)

    reference_hidden = trace.hidden_end
    candidate_hidden = recurrent_middle(trace.hidden_start)

    return {
        "hidden_state_error": measure_hidden_error(
            reference_hidden,
            candidate_hidden,
        ),
        "next_token_match": compare_next_token(
            reference_hidden,
            candidate_hidden,
        ),
        "distribution_kl": compare_distribution(
            reference_hidden,
            candidate_hidden,
        ),
    }

This stage should not be described as a replacement until it passes measured gates:

• hidden-state error
• next-token match
• distribution drift
• rollout quality
• latency delta
• memory delta

The builder-level takeaway:

Each experiment is useful because it creates a boundary that can be identified, tested, measured, and audited.

That is how a runtime path moves from serving optimization toward model-computation research without overstating what has been proven.

Illustrative anonymized example
Illustrative anonymized example:

# Illustrative anonymized example
prefix_state = prefix.forward(tokens)
transition_state = bridge.forward(prefix_state)
output = suffix.forward(transition_state)

assert transition_state.shape == suffix.expected_shape
return output