Switch Adapters by Routing Requests Through Resident, Verified State

Flow

Implementation Decision Loop

A cache hit is accepted only after route identity and concurrency safety are established.

1. 1Parse request⌄

   Normalize the requested adapter and reject invalid input.

2. 2Check registry⌄

   Confirm the adapter was prepared outside the request path.

3. 3Construct key⌄

   Bind model, tokenizer, layout, prefix, and adapter-sensitive identity.

4. 4Hit or recompute⌄

   Reuse only proven-compatible state; otherwise run prefill.

5. 5Generate under binding⌄

   Prevent another request from changing shared route state.

6. 6Measure phases⌄

   Capture lookup, prefill, first chunk, and completion separately.

A multi-adapter runtime should not treat adapter activation as model reloading.

The architecture should look more like this:

The request should select a route. It should not load files, rebuild adapter metadata, or mutate global runtime state without isolation.

A simple adapter state object should be immutable.

# Illustrative anonymized example
from dataclasses import dataclass
from typing import Mapping

@dataclass(frozen=True)
class AdapterState:
    name: str
    adapter_revision: str
    anchor_revision: str
    compatible_model_revision: str
    fused_weight_refs: Mapping[str, object]
    fused_count: int
    skipped_count: int

The registry is prepared before the request path.

# Illustrative anonymized example
adapter_registry: dict[str, AdapterState] = {
    "identity": AdapterState(
        name="identity",
        adapter_revision="none",
        anchor_revision="base-v1",
        compatible_model_revision="model-v1",
        fused_weight_refs={},
        fused_count=0,
        skipped_count=0,
    ),
    # other adapters registered at startup
}

Then request-time adapter selection is just a lookup.

# Illustrative anonymized example
def select_adapter(requested: str | None) -> AdapterState:
    adapter_name = requested or "identity"

    if adapter_name not in adapter_registry:
        raise ValueError(f"unknown adapter: {adapter_name}")

    return adapter_registry[adapter_name]

That small function is the core design shift.

Adapter selection becomes:

requested adapter name -> immutable route state

not:

requested adapter name -> reload model

The dangerous version is a shared mutable pointer.

# Illustrative anonymized example
active_adapter = "identity"

def apply_adapter(name: str) -> None:
    global active_adapter

    if name == active_adapter:
        return

    state = adapter_registry[name]

    for module_name, weight_ref in state.fused_weight_refs.items():
        modules[module_name].weight.data = weight_ref

    active_adapter = name

This can only be safe under strict conditions:

• generation is serialized
• or protected by a lock for the full generation region
• or isolated per worker
• or implemented with request-scoped adapter dispatch

Without that, two concurrent requests can interfere.

Example:

1. Request A asks for code-review.
2. Request A starts prefill.
3. Request B asks for legal.
4. Runtime mutates the global active adapter to legal.
5. Request A continues decode.
6. Request A now emits tokens under the wrong adapter.

That is a correctness failure.

A safer teaching shape is request-scoped binding.

# Illustrative anonymized example
async def handle_generation_request(request):
    route = select_adapter(request.adapter)

    cache_key = build_cache_key(
        route=route,
        input_ids=request.input_ids,
    )

    cached_state = middle_cache.get(cache_key)

    async with runtime.bind_adapter(route.name):
        async for token in runtime.generate_stream(
            input_ids=request.input_ids,
            cached_state=cached_state,
        ):
            yield sse_chunk(token)

The important invariant is:

Illustrative anonymized example
adapter selected by request
= adapter used in cache key
= adapter used in prefill
= adapter used in decode
= adapter used while streaming

Cache identity should be strict.

# Illustrative anonymized example
from dataclasses import dataclass

@dataclass(frozen=True)
class MiddleCacheKey:
    model_revision: str
    tokenizer_revision: str
    layout_revision: str
    prefix_hash: str
    adapter_anchor_revision: str

def build_cache_key(route: AdapterState, input_ids: list[int]) -> MiddleCacheKey:
    return MiddleCacheKey(
        model_revision=MODEL_REVISION,
        tokenizer_revision=TOKENIZER_REVISION,
        layout_revision=LAYOUT_REVISION,
        prefix_hash=hash_tokens(input_ids),
        adapter_anchor_revision=route.anchor_revision,
    )

The cache path should not hide correctness logic.

# Illustrative anonymized example
cached_state = middle_cache.get(cache_key)

if cached_state is None:
    prefill_state = runtime.run_prefill(
        input_ids=request.input_ids,
        adapter=route.name,
    )
    middle_cache.put(cache_key, prefill_state)
else:
    prefill_state = cached_state

for token in runtime.decode(prefill_state, adapter=route.name):
    yield token

A cache miss is not failure. It means the runtime refused to reuse state it could not prove was valid.

The public claim should stay narrow:

Adapter switching can be fast when the base model is resident, adapter state is already registered, route identity is request-scoped, and cache reuse is guarded by strict identity keys.

That is stronger than saying adapters are instant, and safer than implying arbitrary cache reuse across adapters.

Illustrative anonymized example
Illustrative anonymized example:

# Illustrative anonymized example
def route(request, registry):
    adapter = registry.get(request.adapter_kind)
    if not adapter.is_ready():
        raise ExampleStatus("adapter unavailable")
    return adapter.handle(request.payload)