1.1 The full system

Each layer has a clear responsibility:

Sources [AGT-01] [CTX-01] [OBS-01]Author synthesis: the cited sources inform this layered operating model.

1.2 Prompt engineering is one part of the harness

Prompt engineering handles the wording and structure of instructions. Context engineering selects and maintains every token supplied for each model step. Harness engineering runs the software system around the model.

A production harness needs to own:

1. Bind every run.
   1. Request identity, tenant, authorization, and policy version.
   2. Prompt, tool, schema, model, and retrieval-index versions.
2. Control execution.
   1. Deterministic context assembly and compaction.
   2. Model routing and fallback compatibility.
   3. Parse, schema, semantic, and policy validation.
   4. Retry classification and idempotency.
3. Bound and account for outcomes.
   1. Time, token, tool, turn, cost, and risk budgets.
   2. Success, refusal, blocked, degraded, and failed terminal states.
   3. Traces, outcome metrics, and cost attribution.

Enforce any rule about safety, money, permissions, or termination in code. A sentence in a system prompt can steer the model, but it cannot serve as a security boundary.

Sources [AGT-01] [TOOL-01] [AUTH-01]Author synthesis: the taxonomy separates prompt, context, and harness responsibilities.

2.1 Context engineering, not long prompts

Treat context as finite working memory with diminishing marginal value. Adding more can increase prefill cost and time while drawing effective attention away from the material that matters. A long context window does not justify loading everything into it.

Build the context in this order:

1. Invariant contract: role, hard policies, output/tool protocol, and current schema version.
2. Current task: the user's latest intent and explicit constraints.
3. Authorized state: only the tenant, resource, and conversation state needed now.
4. Fresh evidence: retrieved passages with source, version, timestamp, and access metadata.
5. Available actions: only relevant, clearly distinct tool contracts.
6. Useful history: compact decisions and unresolved state, not an unfiltered transcript.
7. Examples: only when they improve the target eval more than the tokens and anchoring risk cost.

Practical rules:

• Optimize signal per token, not context-window utilization.
• Retrieve details just in time. Keep durable decisions in explicit state.
• Label and separate untrusted content from instructions, but do not mistake roles, delimiters, or typed fields for isolation. Enforce a deterministic source-to-sink policy so data derived from an untrusted source cannot authorize actions, expand permissions, or select new egress destinations.
• Keep provenance attached through summarization. A summary without its source and version creates a future stale-fact bug.
• Use structured state instead of repeatedly asking the model to infer state from prose.
• Give tools concise, non-overlapping names and descriptions. Expose only the tools relevant to the current step.
• Version and hash the context manifest so the team can reproduce a bad result.
• Test context ordering and truncation. Models can use relevant information less reliably when it is buried in a long input, a behavior documented by the Lost in the Middle^[CTX-02] study.

Treat a token budget as a policy. There is no universal percentage. One practical version looks like this:

fixed contract

hard cap; cache-friendly prefix

task and state

enough to make the current decision

retrieved evidence

top-ranked, deduplicated, attributable

tool definitions

relevant tools only

working history

compacted decisions and open questions

output reserve

protected before adding more input

When the budget gets tight, remove redundant and low-confidence material first. Compress authoritative instructions or evidence only after those cuts.

Treat provider-managed compaction as a state transition, not invisible housekeeping. Persist the signed summary block exactly as returned. Remove only the messages the provider says it replaced, and keep system instructions and tool definitions compatible across the handoff. Account for the summarization request separately when usage is reported outside the top-level totals. Compaction can silently omit details and remove earlier images, documents, uploads, or URL content. Preserve durable decisions and required evidence outside the transcript, and advance the conversation only after the compacted continuation succeeds. Anthropic's compaction contract^[CTX-03] makes those failure modes explicit.

Sources [CTX-01] [CTX-02] [CTX-03]

2.2 Four caches that must not be confused

Prompt caching and semantic caching do different work:

• Prompt/prefix caching is exact and reuses prefill work. It skips recomputation for an identical rendered prefix, while still running prefill for the uncached suffix and a full decode. Put stable system instructions, examples, and tool definitions first. Put user-specific material later. Small differences in tokenization, message order, tool serialization, or timestamps can destroy a hit. The cache can reduce TTFT and billed input cost only when provider-specific read/write pricing and reuse frequency make the tradeoff worthwhile.
• Semantic caching is approximate and reuses an answer. It decides whether a new request is close enough to an old one to return the old answer. The reuse decision acts as both a quality classifier and a security classifier. The critical metric is false-hit rate by risk slice, not hit rate alone.

Use semantic caching only when two paraphrased requests should receive the same answer. Avoid it, or require verification, for personalized, permission-sensitive, rapidly changing, transactional, high-stakes, and multi-turn-dependent answers. Azure's own semantic-cache policy warns that similarity hits can be incorrect, outdated, or unsafe; see the official policy reference^[CACHE-02].

A defensible application cache key or namespace usually needs:

Request scope

tenantprincipal/ACL scopetask typenormalized state

Runtime contract

model snapshotprompt versiontool-contract versionoutput-schema version

Policy

safety-policy version

Retrieval context

retrieval corpus/index versionlocalefreshness window

Track cache and token hit rates; false-hit, stale-hit, and eviction rates; cache age; latency and cost saved; and cross-scope rejection count. Never share private semantic responses globally based on embedding similarity.

Cache breakpoints determine which shared prefix can actually be reused. Where the API exposes them, place and test an eligible boundary after the stable prefix. An implicit boundary after a changing suffix may never reuse that prefix. Verify with requests that keep the prefix fixed while changing the suffix, and inspect cache-read and cache-write usage. OpenAI documents this distinction^[CACHE-01] for its newer caching contract. Include writes, uncached suffixes, compaction, and retention in the net-cost calculation.

Sources [CACHE-01] [CACHE-02] [SERV-01]Author synthesis: the four caches are grouped by what they reuse and how they fail.

3.1 Prefill and decode optimize differently

Autoregressive inference has two phases that place different demands on the system.

Break the latency budget into its actual components:

Track p50, p95, and p99 for end-to-end latency, queue time, TTFT, TPOT/ITL, and tool latency. A low average can still hide an unusable tail.

Chunked prefill can reduce interference with active decodes and improve tail inter-token latency. Chunking can also increase TTFT for the long prompt being split. Use prefill/decode disaggregation only when its SLO benefit outweighs the cost of duplicated model weights, KV transfer, interconnect, routing, and operations.

Sources [SERV-03] [SERV-04] [SERV-05]

3.2 KV cache management and memory pressure

For a conventional transformer, use this approximation for KV footprint per sequence:

Grouped-query or multi-query attention reduces KV_heads, which can dramatically reduce the footprint. For example, 80 layers, 8 KV heads, head dimension 128, and BF16 state use about 320 KiB per token, or roughly 10 GiB for a 32K-token sequence. Exact layouts and engine overhead vary. The scaling behavior does not: tokens, concurrency, and precision multiply.

Watch for memory pressure in:

• rising KV utilization and allocation failures;
• evictions, preemption, recomputation, or CPU offload;
• falling prefix-cache hit rate under churn;
• queue growth even when compute looks underutilized;
• throughput cliffs and out-of-memory failures on long-tail prompts;
• one tenant's long contexts starving other tenants.

Apply the controls in this order:

1. Enforce admission control and explicit maximum input/output lengths.
2. Reserve memory for active decodes and protect latency-critical traffic classes.
3. Use paged/block allocation to reduce fragmentation. Prefix sharing is a separate, explicitly keyed capability layered on top.
4. Use per-tenant concurrency and KV quotas.
5. Where prefix caching is enabled, reuse exact full-prefix blocks with ownership, refcounts, and copy-on-write semantics.
6. Evict only refcount-zero reusable blocks. Reclaim active KV by preempting whole sequences and explicitly choosing recomputation or offload; do not apply eviction only by request age without workload evidence.
7. Evaluate KV quantization or offload on long-context quality and tail latency before enabling it.
8. Shed or degrade load before allowing a node-wide OOM.

The PagedAttention paper^[SERV-01] applies virtual-memory-like block management to KV state. This reduces allocation waste and enables sharing. Paging manages memory more efficiently, but the underlying KV capacity limit remains.

KV retention policies decide which past state survives; paging decides how retained state is allocated. Measure retained bytes, physically allocated memory, and peak memory across the whole invocation, including prefill. A mask or eviction after prefill may reduce later state without lowering the prefill peak. Evaluate long-context evidence use with the same retention policy and workload used for memory tests. HeadWiseKV^[SERV-11] illustrates these distinctions in a bounded hybrid-model study. Its results do not establish a universal quality or memory gain. Revalidate after changing the model, cache format, context distribution, or retention policy.

For bursty multi-tenant traffic, evaluate a reservation policy against real prompt-length, output-length, and arrival traces instead of reserving each request's maximum possible KV footprint. A reservation policy can admit more useful work when it protects against plausible distribution shifts and keeps an explicit overload path. Cheng et al.'s robust KV-cache reservation study^[SERV-06] provides one trace-driven formulation. The formulation is a planning method, not a substitute for per-tenant quotas, active-decode reserves, or load shedding.

Sources [SERV-01] [SERV-04] [SERV-06] [SERV-11]The KV footprint example follows directly from the model architecture.

3.3 Continuous batching, paged attention, and goodput

Static batching waits for a fixed group of requests. Continuous, or iteration-level, batching releases completed sequences at each engine iteration and may admit queued work into the capacity they free, subject to token, KV, deadline, and fairness limits. Continuous batching removes static-batch head-of-line blocking, although requests can still queue. The Orca paper^[SERV-02] established iteration-level scheduling as a core serving technique.

Large prefills can still stall active decodes. Chunked prefill splits prompt processing into schedulable pieces and interleaves those pieces with decode work. The tradeoff is lower decode interference and potentially longer TTFT for the split request. Sarathi-Serve^[SERV-03] analyzes this prefill/decode interference and stall-free scheduling.

The scheduler needs controls for:

• maximum batched tokens and active sequences;
• prefill chunk size and prefill/decode priority;
• request class, deadline, and fairness policy;
• maximum model length and output reservation;
• prefix locality and cache-aware routing;
• preemption and load-shedding policy.

Raw tokens per second do not tell you whether the service works for users. Optimize goodput: the maximum arrival rate per provisioned GPU or resource at which the required fraction of requests satisfies both TTFT and TPOT/ITL SLOs plus the application-quality gate. A larger batch can increase throughput while pushing interactive latency past an acceptable limit.

For multi-stage agent workflows, measure task-level completion time alongside request goodput and prefix-hit rate. Evaluate dependency and critical-path progress, prefix residency, transition and preemption cost, and starvation on real traces. Optimizing prefix locality or the progress of one stage in isolation can make the complete workflow slower.

The current vLLM documentation^[SERV-04] is a useful implementation reference for PagedAttention, continuous batching, prefix caching, chunked prefill, quantization, and speculative decoding. Engine defaults are starting points. Tune them against production prompt-length and output-length distributions.

Sources [SERV-02] [SERV-03] [SERV-04] [SERV-07]

4.1 Speculative decoding, quantization, and distillation

Speculative decoding, quantization, and distillation change different parts of the inference stack and solve different bottlenecks.

When the rejection sampler and target-model sampling semantics are implemented correctly, speculative decoding preserves the target distribution. Use it to reduce latency, not to approximate an answer. Floating-point or batching nondeterminism still prevents any guarantee of byte-identical runs. Benchmark acceptance, draft and verification cost, TPOT, and SLO goodput at production concurrency. Low acceptance or saturated compute can make the system slower. Leviathan et al.^[SPEC-01] describe the method.

Distillation changes the deployed model and usually changes its behavior. Distillation can greatly lower steady-state cost, but the target workflow first needs to be stable enough for teacher data and evals to represent production. Hinton, Vinyals, and Dean^[DIST-01] describe the foundational method.

Sources [SPEC-01] [QUANT-02] [DIST-01]Author synthesis: the comparison maps each technique to the bottleneck it changes.

4.2 Quantization: formats, methods, and quality risk

AWQ and GPTQ describe quantization methods. Numeric formats are a separate choice. Always spell out the precision tuple: W8A8, W8A16, W4A16, or KV8, for example. INT8 and INT4 leave too much unspecified on their own.

Weight storage is approximately parameters × bits / 8 + scales + zero-points + unquantized modules. W4A16 reduces weight storage, but it does not shrink activations or KV. Calling a model “4-bit” therefore does not imply 4× lower total VRAM. Any latency reduction depends on the actual kernel and hardware.

Primary references: FP8 formats^[QUANT-01], SmoothQuant for W8A8^[QUANT-02], AWQ^[QUANT-03], and GPTQ^[QUANT-04].

Quality risk generally rises below 4-bit, in smaller models, on harder reasoning slices, with unrepresentative calibration, and when activation or KV outliers are aggressively quantized. The risk pattern is empirical, not guaranteed. Specific risks include:

• calibration data does not match production languages, domains, lengths, or modalities;
• the task depends on small logit margins, exact numbers, code, rare tokens, or precise tool arguments;
• activation or KV outliers are aggressively quantized;
• long-context, multilingual, rare-token, safety, or tool-use behavior is more sensitive than the aggregate benchmark suggests.

Short-context or aggregate-only evals can hide regressions. Missing native kernels can erase the performance gain even when quality holds. Benchmark the exact checkpoint, packing, runtime, hardware, batch shape, and parallelism plan.

Run every quantized candidate through the same golden and adversarial suite as the baseline. Include exact structured output, function selection and arguments, safety refusals, multilingual slices, long context, and p95/p99 latency. Measure end-to-end performance. Model file size alone proves nothing.

Sources [QUANT-01] [QUANT-02] [QUANT-03] [QUANT-04] [SERV-04]

5.1 Structured outputs: syntax is only the first gate

Constrained decoding and provider structured-output features can greatly improve syntactic conformance. Syntactic conformance does not establish whether the values are true, authorized, internally consistent, current, or safe to execute.

Treat every structured-output request as a fault-containment boundary. Cap grammar or schema compilation work, isolate it from the shared inference engine, and reject malformed or unsupported constraints without terminating other tenants' requests. A request-scoped validation error must remain request-scoped; the vLLM structured-output advisory^[SAFE-17] shows how an unhandled request error can otherwise kill a shared engine process.

Use this pipeline:

Build the schema so failures are explicit:

• Version the schema and reject unknown versions.
• Use required fields, enums, ranges, string formats, and additionalProperties: false where appropriate.
• Make invalid states hard to represent with discriminated unions rather than nullable field combinations.
• Distinguish null, omitted, unknown, and refused.
• Validate dates, identifiers, money, units, and cross-field relationships in deterministic code.
• Store hashes, versioned references, validation errors, and repair lineage by default. Retain raw candidates only under an explicit privacy policy with minimization or redaction, encryption, restricted access, and short retention.
• Treat schemas from providers, tool servers, or user-controlled sources as untrusted input. Do not automatically resolve network $ref targets. If remote resolution is explicitly enabled, allowlist hosts; block loopback, link-local, and private ranges; cap bytes and time; log every fetch; and reject unresolved references. Bound schema depth, subschema count, and validation time so composition keywords cannot turn validation into an SSRF or denial-of-service path.

Keep repair bounded:

1. Classify transport/status, refusal, incomplete/truncated, parse, schema, semantic, authorization, and policy failures separately.
2. Repair only bounded syntax or shape failures when the result is non-consequential or read-only. Never use repair to override a refusal, invent semantically missing values, or make a mutation executable.
3. Revalidate the complete normalized payload from the beginning.
4. For consequential payloads, regenerate from trusted inputs, request clarification, require review, or fail safely. Alternate models must satisfy the same data policy and contract; deterministic defaults must be explicitly product-defined and surfaced to the user.

Do not create an unbounded “please fix the JSON” loop. Track first-pass validity and final validity separately. Otherwise repair can hide a model regression. See the OpenAI structured output guide^[OUT-01] and the JSON Schema documentation^[OUT-02].

Sources [OUT-01] [OUT-02] [MCP-03] [SAFE-17]Author synthesis: repair policy is separated from the rules for consequential actions.

5.2 Reliable function and tool calling

Treat a tool call as an untrusted proposal crossing from a probabilistic system into deterministic software.

A strong tool contract includes:

1. one clear purpose and a distinct, namespaced name;
2. typed, unambiguous arguments such as customer_id, not customer;
3. a strict input schema and documented semantic invariants;
4. explicit read/write, destructive, network, and open-world declarations for UX and discovery; make security decisions from a trusted server-side policy registry, not annotations supplied by a dynamic or untrusted tool server;
5. bounded pagination and concise high-signal output;
6. stable error codes, retryable status, and a typed result envelope;
7. timeouts, rate limits, authorization requirements, and audit semantics;
8. a contract version and compatibility policy.

The harness then enforces the contract:

1. Reject unknown tool names; never dynamically dispatch a hallucinated string.
2. Parse and validate all arguments.
3. Resolve identity and authorize the exact resource and action server-side.
4. Create or accept one idempotency key per logical mutation before first dispatch. Scope it to tenant, actor, and operation; persist it across retries, resumes, handoffs, and model/provider fallbacks; reject reuse with a different normalized request fingerprint.
5. Require a single-use, expiring approval for consequential actions, bound to the normalized action, actor, resource, destination, disclosed data, and state version. Reauthorize immediately before execution; any material change requires new approval.
6. Execute with least-privilege, short-lived credentials.
7. Validate tool output and treat returned text as untrusted data, not new instructions.
8. Retry only failures known to be safe, using the same idempotency key.
9. Record the proposed action, policy decision, execution result, and resulting state.

A useful response envelope is:

{
  "ok": false,
  "code": "VERSION_CONFLICT",
  "retryable": false,
  "recovery": "REFRESH_STATE_AND_REPLAN",
  "data": null,
  "state_version": "42"
}

Design for at-least-once delivery. Control it with idempotent effects, uniqueness constraints, or compare-and-swap state. Atomically record the idempotency key, normalized request fingerprint, side effect, and resulting receipt/state.

When a stable, repeated procedure is narrow enough to specify and test, compile it into a versioned deterministic tool instead of making the model re-reason through every mechanical step. Keep judgment, exception handling, and stop decisions explicit. In a production alarm-triage study, Kujanpaa et al. compiled repeated SOP steps into tools^[TOOL-04] and reported lower median latency and fewer errors in that workflow; treat the result as domain evidence, not a universal performance guarantee.

A network timeout can leave the execution status unknown. Reconcile through the idempotency receipt or authoritative external state before retrying or changing routes. The HTTP specification explains why automatic retry is safe only when the semantics are idempotent or the client knows the original action was not applied; see RFC 9110^[TOOL-03].

Asynchronous tool calling adds a pending-job lifecycle. Register each dispatched job before waiting on it. Persist its handle and original call ID for the conversation, and deliver the eventual result against that call. Keep job status separate from the result; a scheduled job is not a completed action. OpenAI's async-tool contract^[TOOL-05] makes these distinctions explicit. Apply the same tenant binding, authorization, deadlines, and cancellation rules to callbacks and late results. Preserve completed handle records so a later call cannot reuse their identity.

A late tool result is still untrusted input. Reauthorize its tenant, principal, conversation, run, call, resource, and state version at delivery time; discard results for cancelled, expired, superseded, or completed work. Never let a delayed callback revive an old plan, silently spend a new budget, or attach to a reused call identifier.

If tools can be added or removed mid-conversation, treat the available-tool set as versioned runtime state. Build the registry server-side and authorize every change. Persist a canonical schema digest with the turn, and reject a call whose tool version no longer matches. Keep security policy outside model-supplied tool descriptions, and test whether removing a tool truly makes it unavailable. Anthropic's dynamic-tool contract^[TOOL-06] is one current example of this pattern.

Schema-valid tool output can still be plausibly wrong. For consequential actions, verify the returned entity, provenance, freshness, units, and relevant state against an authoritative source or an independent deterministic check before using it. Test tools that return fluent, internally consistent falsehoods, not only exceptions and malformed payloads. The study Agents' Overreliance on Unreliable Tools^[ORIG-13] found that agents often relied on corrupted but plausible tool results and that explicit verification prompts detected more corrupt results than the baseline prompts.

Evaluate tool and protocol behavior directly. Test correct selection, correct non-selection, argument validity, authorization denial, duplicate delivery, timeout-after-commit, partial failure, hostile tool output, protocol-version negotiation, and session-state assumptions. For the source behind this guidance, see Anthropic's tool design guidance^[TOOL-01].

Give every agent and workload a distinct identity; never lend it a human credential or allow it to impersonate the delegating user. Bind short-lived, audience-restricted authority to the operating principal, attenuate it at every delegation, and let the least-authorized hop bound the chain.

Protocol migrations are architecture migrations. MCP 2026-07-28 removed protocol sessions, Mcp-Session-Id, and the initialization handshake. A connection or stdio process is not a conversation. Carry protocol version and capabilities on every request, discover server support before selection, and pass cross-call state through explicit handles. Treat clientInfo and serverInfo as self-reported diagnostics, never authenticated identity or policy input.

Model protocol results as a state machine. resultType: "input_required" is nonterminal; preserve the logical operation, policy decision, approval scope, and budget across the next round. If a response stream breaks, issue a new protocol request ID, but preserve the application idempotency key and reconcile external state before repeating a mutation. Pin the protocol and SDK, test mixed-version clients and servers, and roll through a compatibility window.

Treat protocol deprecations as owned migration work, not optional cleanup. The MCP 2026-07-28 changelog^[MCP-02] marks Roots, Sampling, Logging, HTTP+SSE, the "thisServer" and "allServers" includeContext values, and OAuth Dynamic Client Registration as Deprecated. Do not add new dependencies on them. Inventory current use, assign an owner and removal date, and test the named replacement before the compatibility window closes.

Sources [TOOL-01] [TOOL-02] [TOOL-03] [TOOL-04] [MCP-01] [MCP-02] [MCP-03] [AUTH-01] [AUTH-02] [AUTH-03] [TOOL-05] [TOOL-06] [TOOL-07] [MOD-01] [ORIG-13]Author synthesis: approval, idempotency, and execution receipts form one tool-safety contract.

6.1 Agent guardrails, budgets, and termination

An agent is a state machine whose next transition is proposed by a model. Apply guardrails at every transition. Final-text checks alone are not enough.

Enforce these budgets outside the model:

Budget counters need to be hierarchical, atomic, and persistent across retries, resumes, handoffs, fallbacks, and subagents. A retry or alternate route must not reset the run's deadline, cost, risk, or mutation budget.

Give every run explicit terminal conditions:

• success predicate verified against external state;
• user cancellation, after stopping new work and reconciling any already-dispatched mutation whose commit status is unknown;
• policy or authorization denial;
• dependency unavailable or irrecoverably blocked;
• budget exhausted;
• repeated state, repeated action, or no measurable progress;
• insufficient evidence under an externally measured sufficiency check, or a calibrated product score that requires abstention; never use the model's self-reported confidence alone.

Broken or impossible tasks must terminate safely. Persistence or additional reasoning may not unlock wider tools, egress, or credentials. Define severity triggers for boundary circumvention and unauthorized cross-agent coordination. Provide a reliable fleet or workload kill path, name the authority to pause, contain, restart, and restore, and fail closed when a severe alert cannot be cleared.

Treat awaiting_input and awaiting_approval as persisted, resumable interrupts rather than success or failure terminals. Keep their deadlines and reserved budgets explicit.

The model's claim that it finished does not count as the success predicate. Verify the outcome: the record exists, tests pass, the message was sent once, or the cited evidence supports the answer.

When the run ends, return a typed state such as succeeded, partial, blocked, refused, cancelled, budget_exhausted, or failed. Include the completed work and a safe next action. A typed result makes degraded UX and operations much clearer than a generic error.

Sources [AGT-01] [EVAL-02] [AUTH-02] [EVAL-05] [EVAL-06] [ORIG-05]Author synthesis: agent limits are organized as budgets and explicit terminal states.

6.2 Model routing and degraded mode

Route against measured task requirements, not provider reputation or prompt length alone. Include:

• task class and required capability;
• risk and consequence of error;
• modality, context length, language, and tool requirements;
• latency deadline and cost ceiling;
• tenant policy and data residency;
• current provider health and rate limits;
• router confidence and out-of-distribution detection.

Maintain a compatibility matrix for every route: exact model snapshot or negotiated protocol version; provider API, client SDK, and serving-runtime versions; lifecycle state and retirement date; supported request parameters and turn rules; tool protocol; JSON Schema support; context and output limits; refusal semantics; modality; processing geography and data policy; and tested quality slices. Managed agent runtimes and built-in tools belong in that matrix too. Record the agent runtime, tool versions, approval and sign-in behavior, state-retention rules, and retirement date. Then test the actual replacement before a preview or dated model shuts down. Treat aliases, defaults, and wrappers as mutable. Google, Anthropic, OpenAI, and AWS publish changes on different schedules, and platform-specific retirement dates can differ from a model creator's dates. Google's Gemini deprecation schedule^[MOD-07] demonstrates why a product alias or managed surface needs its own migration owner even when the underlying model family remains available. A fallback that cannot honor the same contract puts the product in a different mode.

The resolved model settings and preserved model-generated state belong in the migration record. Test omitted parameters separately from explicit values, record supported reasoning levels and their output-token budget, and verify tool/API support and cache controls on the exact route. Parse responses by block type. Bind signed thinking, reasoning, or summary blocks to the exact provider, model, account, conversation history, and API contract. Return them unchanged when required, and keep the history append-only. If a route drops or rejects that state, fail or restart from an explicitly compatible checkpoint instead of silently continuing with different semantics. Migration tests must cover default reasoning behavior, forced-tool restrictions, structured-output parameters, refusal categories, and a fresh effort sweep because the same effort label can change across model generations. OpenAI's migration guidance^[MOD-05] documents changes to these contracts, while AWS's adaptive-thinking correction^[MOD-06] shows how omission can enable billable reasoning. Anthropic's Sonnet 5.5 migration guide^[MOD-08] documents the signed-state and request-contract changes. Reject unsupported settings rather than silently dropping them. Compare the resolved behavior, latency, and cost before rollout.

Treat provider SDK upgrades as production migrations. Diff request headers and response types, exercise every mutating method against disposable state, and canary the pinned upgrade before rollout.

Choose the fallback based on the failure class. Never route around a safety refusal, an authorization or policy denial, or a semantic validation failure. After a timeout that may have committed a mutation, reconcile through the idempotency receipt before retrying or changing routes. Every attempt shares one deadline and retry budget. Use circuit breakers to prevent retry and fallback storms.

Overload and traffic ramp limits need separate recovery paths. Classify the HTTP status together with the provider error code. OpenAI distinguishes a 429 slow_down ramp limit from a 503 server_is_overloaded capacity failure in its error-code reference^[REL-05]. Honor Retry-After when present. Otherwise, use bounded backoff with jitter and coordinate pacing across workers instead of letting each retry independently. Keep retries inside the shared deadline and budget, and use a qualified fallback or queued outcome when waiting would exceed them.

A safe chain might be:

In degraded mode, tell the user exactly what changed: unavailable capability, reduced freshness, partial scope, read-only behavior, or delayed completion. Do not silently swap to a materially weaker model and present the output as equivalent.

Evaluate the router directly: route accuracy, unnecessary escalation, unsafe down-routing, fallback success, quality by route, added latency, and cost per successful task. Shadow new routing policies only on eligible, authorized traffic. Disable all side effects, redact where possible, and never send data to a provider or region outside the request's approved residency and retention policy.

Sources [OPT-01] [SERV-05] [MCP-01] [MOD-01] [MOD-02] [MOD-03] [MOD-04] [MOD-05] [MOD-06] [MOD-07] [MOD-08] [REL-05]Author synthesis: routing decisions are tied to the user's degraded-mode experience.

7.1 RAG architecture

RAG gives a generator access to an external retrieval system. If ingestion loses evidence or retrieval misses it, the generator cannot recover it.

Ingestion and chunking

• Build chunks around retrievable answer units: sections, procedures, clauses, table regions, code symbols, or conversation turns.
• Preserve title, hierarchy, source ID, version, timestamps, page/line anchors, and ACL metadata.
• Keep tables, lists, and code structures intact where splitting destroys meaning.
• Use overlap only to preserve boundary meaning; indiscriminate overlap increases duplicate retrieval and context waste.
• Consider small child chunks for recall and larger parent sections for generation context.
• Build versioned tombstones and deletion propagation. A deleted source remaining in the vector index is a security and freshness bug.

Embeddings

• Benchmark embedding models on the product's actual queries, languages, jargon, document types, and relevance units. No model dominates every retrieval task.
• Use a pinned model, version, preprocessing configuration, and distance function for both document and query vectors.
• Treat an embedding change as an index migration. Build and evaluate a side-by-side index, then switch atomically or through a measured rollout. Do not mix incomparable vector spaces.
• Dimension, precision, and index parameters trade storage and query cost against recall. Measure the full retrieval system rather than ranking embedding models by a generic leaderboard alone.

Retrieval

• Lexical retrieval is strong for exact names, identifiers, acronyms, error codes, and rare terms.
• Dense embeddings are strong for paraphrase and conceptual similarity.
• Hybrid retrieval combines complementary candidate sets. Reciprocal rank fusion is a practical starting point when raw scores are not directly comparable; see the Elastic hybrid-search guide^[RAG-02].
• Enforce tenant and document ACL constraints inside the source query and before or during vector candidate generation and reranking. Post-filtering a global top-k result is not an isolation boundary and risks leakage through scores, logs, caches, or context.
• Treat retrieval queries, embeddings, candidate identities, scores, access patterns, and selected results as sensitive data. ACLs control what may be returned; they do not hide these artifacts from the retrieval operator. When that operator is outside the trust boundary, define the adversary and validate the security, latency, and bandwidth tradeoffs of the private-retrieval design.
• Rerank a wider candidate set with a more precise model, then pack a smaller nonredundant evidence set.
• Tune k, fusion, reranking, and context packing against end-to-end evals instead of intuition.

Freshness

Track source_version, indexed_at, validity interval, and deletion state. Measure source-to-index lag and define a freshness SLO. Use change-data capture, event-driven reindexing, or scheduled reconciliation based on the source. Invalidate retrieval and semantic caches with corpus/index versions, not only TTLs.

The foundational RAG paper^[RAG-01] emphasizes non-parametric memory, provenance, and updatable knowledge. A production system also needs authorization, freshness, hybrid retrieval, reranking, context packing, and supporting operations.

Sources [RAG-01] [RAG-02] [RAG-03] [RETR-06] [TEN-01]

7.2 Retrieval and answer evals

Evaluate retrieval separately from generation. A correct answer can hide a retrieval miss when the model already knew the fact. Perfect evidence can still produce a bad answer.

Define the relevance unit for every metric: document, deduplicated source, chunk, claim, or exact evidence span. Overlapping chunks can inflate item precision and recall, so report deduplicated source recall and claim coverage as well. When relevance judgments are incomplete, Recall@k and Precision@k are estimates against the judged pool. They do not represent corpus-wide truth.

A useful golden record contains:

Request context

queryuser/tenant/ACL identitytime snapshot

Expected outcome

expected answer or outcomeanswerable/unanswerable label

Evidence and attribution

relevant source IDsexact evidence spansallowed and forbidden sourcesexpected citation mapping

Coverage and risk

important slice labelsrisk class

Test typo, acronym, exact-identifier, paraphrase, multi-hop, conflicting-source, stale-source, newly updated, deleted, unauthorized, poisoned-document, no-answer, and long-tail queries.

Use BEIR^[RETR-01] for heterogeneous retrieval evaluation, RAGAS^[RETR-04] for component-level RAG evaluation, and ALCE^[RETR-05] for citation quality. Treat every retrieval or evaluator score as decision-specific evidence. Validate it against the comparison, training choice, system selection, downstream prediction, or filtering decision it will drive. Test whether evaluator conclusions remain stable across the relevant slices. A metric that ranks retrievers well may still fail to predict answer quality or justify a different generator. Automated metrics surface problems; calibrated human review remains part of the evaluation.

For iterative search agents, evaluate retrieval in trajectory context. Condition the retriever and its evaluation on the original task and prior search interactions, not only the latest sub-query. Measure marginal evidence gain, duplicate resurfacing, and task-level outcome. A locally relevant result can still waste a step or steer the investigation away from the remaining evidence gap.

Sources [RETR-01] [RETR-02] [RETR-03] [RETR-04] [RETR-05] [RETR-07] [RETR-09]

8.1 Evals as the release contract

Build the evals before you optimize. Without them, every caching, routing, quantization, prompt, retrieval, or model change can quietly exchange quality for speed or cost.

Use these layers together:

1. Deterministic unit tests: parsers, schemas, invariants, ACL filters, idempotency, budgets, termination, and cost math.
2. Component evals: retrieval, reranking, routing, tool selection/arguments, structured output, safety classifiers.
3. Golden workflow regressions: representative production tasks and high-value slices with known outcomes.
4. Adversarial evals: injection, poisoned retrieval, malformed tools, duplicate delivery, timeouts, stale indexes, long contexts, conflicting evidence, and runaway-loop traps.
5. Stochastic trials: repeated runs to measure consistency, not one lucky completion.
6. Human evaluation: subjective quality, novel failures, safety, and high-stakes domain correctness.
7. Production monitoring and A/B tests: real distribution, user outcomes, drift, and unexpected interactions.

Keep capability evals separate from regression tests. Capability sets should stay challenging enough to measure improvement. Regression sets protect behavior already known to work and generally need a much higher pass rate.

Keep the iteration set separate from a blinded release holdout. Deduplicate related prompts, source documents, and generated variants across splits. Prevent fine-tuning data, few-shot examples, and prompt-optimizer inputs from contaminating the holdout.

For nondeterministic workflows, report both first-try success and consistency. pass@k answers whether at least one of k trials succeeds; pass^k asks whether all k trials succeed. Customer-facing reliability often depends much more on the latter than benchmark demos do. Always report pass@1, the exact k, trials per task, estimator, and confidence interval. Do not compare pass@k or pass^k across different k values. Repeated retries do not equal first-attempt product reliability.

LLM-as-judge is useful when:

• the rubric is explicit and examples are calibrated;
• candidates are anonymized and pair order is randomized or swapped;
• deterministic checks grade everything they can grade;
• judge agreement is measured against domain experts;
• disputed, high-risk, and distribution-shifted cases go to humans.

An LLM judge is not an independent source of truth. Judges exhibit position, verbosity, self-preference, and reasoning biases; see the MT-Bench judge study^[EVAL-01].

Calibrate a black-box judge on the exact endpoint, model version, rubric, language, and time window used for the release. Repeat representative judgments, record disagreement and rank stability, and require a human or deterministic fallback when the judge drifts. Do not transfer a calibration result across endpoints or silently updated aliases. A 52,988-attempt study^[ORIG-12] of shared judge endpoints found substantially lower repeat and next-day rank stability than a controlled endpoint. This makes local calibration an operating requirement rather than a one-time benchmark.

Release gates

Make release gates slice-aware. A one-point global gain cannot offset a large regression in safety, structured outputs, a language, a tenant type, or a critical workflow. Treat any High-severity safety, authorization, or side-effect failure as its own stop condition instead of averaging it into a global score.

Version the test data, graders, judge prompts/models, harness code, and environment. Report task count, trial count, confidence intervals, and per-slice support. Define non-inferiority margins before the run. Then read the failed traces; the score cannot tell you whether the model or the evaluation infrastructure failed.

For multimodal injection, report both attempt rate and completed harmful action by carrier and execution stage. Test visible text, hidden metadata, audio, video, files, screenshots, and retrieved media. Distinguish whether the model noticed, planned, called a tool, and produced an external effect. Low end-to-end completion can conceal a much higher rate of attempted unsafe actions. MMPIBench, for example, reports six carrier types and a large gap between attempted and completed attacks across 720 runs (paper^[ORIG-15]).

Treat agents and model-generated code in training and evaluation as hostile workloads. Isolate every run and its writable supporting services. Apply independent egress controls to the sandbox and every package mirror, cache, metadata service, relay, and shared store. Make direct escape, transitive escape, and cross-run communication tested failure modes.

Grade the trajectory and external effect, not only the reward or final answer. Include broken or impossible tasks, grader and tool tampering, unauthorized peer instructions, side channels, boundary probing, safe clarification or stop behavior, and cross-run coordination. Keep grader secrets outside agent-visible surfaces.

When model-generated code changes the system under evaluation, bind approval to an immutable artifact digest. Any edit invalidates approval. Keep held-out data inaccessible to the agent through operating-system isolation. Have a separate evaluator refuse unapproved artifacts, use capability regressions as hard gates, and review sampled trajectories for gaming. Monitoring is detection, not a substitute for isolation or deterministic enforcement.

Anthropic's agent-eval guidance^[EVAL-02] and OpenAI's eval guide^[EVAL-03] both emphasize evaluating the full system or harness instead of stopping at isolated model text.

Sources [EVAL-01] [EVAL-02] [EVAL-03] [EVAL-05] [EVAL-06] [ORIG-05] [EVAL-09] [ORIG-12] [ORIG-15]

9.1 LLM observability as a first-class discipline

Give each bounded workflow execution its own trace. Use a stable journey or group ID and span links to connect retries, background jobs, queued continuations, and multi-session journeys. Do not stretch one trace across an unbounded conversation. One span structure looks like this:

Capture the following data, with the right privacy controls in place:

• trace, run, and step IDs, including parent relationships;
• feature, workflow, route reason, and pseudonymous tenant/user dimensions;
• model provider, requested model, actual model snapshot, and decoding settings;
• prompt/context manifest, tool contract, schema, policy, and index versions;
• per-model-call input, cached input, cache-write, output, and reasoning-token counts when available; derive agent and workflow totals from those child spans rather than attaching rolled-up cache subtypes to an agent span;
• cache decision, similarity score, age, namespace, and invalidation reason;
• retrieval query, filters, source IDs/versions, ranks, and reranker scores;
• TTFT, mean TPOT, p95/p99 inter-token latency and stall rate, plus queue, retrieval, rerank, tool, repair, and end-to-end latency;
• validation errors, retries, the fallback path, finish reason, and terminal state;
• outcome, quality score, user feedback, and attributed cost.

Do not use source IDs, trace IDs, pseudonymous user or tenant IDs, prompts, or free-form route reasons as time-series metric labels. Keep those values in traces or analytical storage. Use bounded enumerations for metric labels, then join the high-cardinality attributes during offline analysis.

By default, do not record raw prompts, retrieved private documents, tool arguments, or model outputs. Use redaction, sampling, encryption, access control, short retention, and content-free hashes or manifests. The observability system cannot become the largest data-leak path.

Build drift dashboards from directly observed distributions and sampled, versioned evaluator estimates for retrieval relevance, citation support, grounding, and outcome quality. Show evaluation coverage, grader version, calibration status, and confidence intervals. Put them alongside input length, route mix, cache behavior, tool errors, validation and repair rate, tokens, latency, and cost. Alert on user-impacting SLOs and regression slices instead of every noisy model event.

OpenTelemetry's GenAI semantic conventions^[OBS-01] provide a vendor-neutral starting vocabulary for inference, agents, tools, tokens, and timing. The conventions are evolving, so pin the version your telemetry contract uses.

Agent-, runtime-, and tool-authored transcripts are evidence, not the source of truth. Capture policy decisions, tool invocations, network events, receipts, and terminal state outside the agent-controlled environment in append-only or tamper-evident storage. Then reconcile traces against authoritative external state. Reasoning traces, when available, may supplement detection, but neither hidden reasoning nor any single transcript can be a required enforcement dependency.

Monitor recalibration belongs in every model or reasoning-configuration migration. Compare action-only, reasoning-only, and combined monitoring where those signals are available and permitted. Measure detection, false alarms, and evasion on the same task slices. OpenAI's GPT-6 Astra system card^[EVAL-10] reports monitorability changes and adversarial evasion in its evaluation setting. Treat those results as a reason to revalidate local monitoring, not as a production detection rate. Keep external receipts and enforceable policy checks independent of the monitor.

Sources [OBS-01] [OBS-03] [OBS-02] [EVAL-06] [ORIG-05] [EVAL-10]

9.2 Attribute cost to the whole workflow

Use the same reliability trace to attribute cost:

The dimensions you usually need are feature, workflow, tenant, user journey, route, model, environment, and terminal outcome. Use pseudonymous identifiers where possible.

Track:

• cost per successful task, not only per request;
• cost per grounded/correct answer;
• retry and repair amplification;
• semantic and prefix-cache savings net of lookup/storage cost;
• cost of fallback and escalation paths;
• idle and reserved accelerator cost for self-hosting;
• for self-hosted serving, total inference energy per request and per successful outcome alongside energy per output token, sliced by model, phase, batch, context length, and output length; a falling per-token figure can hide rising request energy;
• marginal cost and fully loaded cost separately.

Reconcile the trace-derived estimates with provider invoices or Costs APIs. Allocate reserved capacity, idle accelerators, storage, observability, and shared platform cost through documented drivers. Keep an explicit unallocated or residual bucket instead of inventing false precision. Preserve the price-table version and effective timestamp behind every calculation.

Compare complete workflows across quality, latency, cost, and reliability. Model price is only one input. A cheaper model can cost more when it doubles retries or sends more work to humans. A higher-priced model can cost less per successful outcome when it shortens the loop, uses tools correctly, or prevents failure.

Sources [COST-01] [COST-02] [OBS-01] [OBS-03] [COST-03]Author synthesis: the journey-cost formula ties spend to a completed user outcome.

10.1 Safety engineering and permission boundaries

A stronger system prompt, fine-tuning, or RAG will not solve prompt injection. User text, web pages, documents, emails, images, and tool outputs can all carry adversarial instructions. Assume a probabilistic detector will eventually miss an attack. Deterministic controls must limit the blast radius when that happens.

Build the defense in layers:

1. Authenticate the human and workload identity.
2. Authorize every data retrieval and tool action against the exact resource.
3. Label and separate untrusted content from instructions, but treat the separation as model guidance rather than isolation; enforce deterministic source-to-sink policy.
4. Expose only the tool and data surface the current step needs. Keep credentials and capability tokens inside the trusted executor.
5. Use allowlisted tools, egress destinations, and parameter constraints.
6. Never place raw secrets or bearer tokens in model context. Inject scoped, short-lived credentials server-side only after authorization.
7. Sandbox code, files, and network access.
8. Require a single-use approval bound to the exact consequential action and state version; reauthorize immediately before execution.
9. Apply data minimization and DLP at ingestion, context assembly, tool arguments and egress, final output, logs, and traces. Reauthorize sensitive data at every sink.
10. Audit proposals, policy decisions, actions, and resulting state.
11. Red-team direct, indirect, multimodal, encoded, and retrieved injections.
12. Have containment, revocation, incident response, and replay procedures.

Human approval is scarce attention and approval frequency is a security property. Use bounded action plans and risk-triggered reapproval so consent fatigue does not turn review into a rubber stamp. Never use elicitation to collect passwords, bearer tokens, or other secrets.

Govern long-lived API credentials as production identities. Require an owner, purpose, environment, allowed service surface, maximum lifetime, rotation path, last-used visibility, and revocation drill. Prefer short-lived workload credentials. When a static key is unavoidable, enforce expiry and organization-level lifetime policy, alert before expiration, and prove that rotation does not require an outage. OpenAI's API changelog^[MOD-04] now documents expiry and maximum-lifetime controls, turning credential age from an inventory field into an enforceable policy.

Treat agent skills, instruction bundles, and reusable workflow packages as executable supply-chain inputs. Pin the exact source and digest. Review transitive files and scripts, minimize installation and runtime privileges, and re-evaluate the package after any update. Test for instruction smuggling, credential access, unexpected egress, and delayed or cross-run behavior. SkillShift^[ORIG-14] demonstrates that apparently useful agent skills can carry persistent behaviors that activate outside the installation moment.

For OAuth or MCP tools, validate the issuer, audience, scopes, actor, and expiry. Prohibit token passthrough. Use separately issued downstream tokens and per-client consent. Treat OAuth metadata, discovery documents, and registration endpoints as untrusted network input. Allowlist schemes and destinations, block private and link-local ranges after DNS resolution, disable unsafe redirect following, and revalidate every redirect hop. The MCP authorization specification^[AUTH-01] and security best practices^[AUTH-02] make these boundaries explicit.

The OWASP LLM prompt-injection guidance^[SAFE-01] notes that RAG and fine-tuning do not fully mitigate injection. Microsoft's defense-in-depth pattern^[SAFE-02] recommends content isolation, least privilege, monitoring, and human approval. For the broader governance and risk context, use the NIST Generative AI Profile^[SAFE-03].

Sources [SAFE-01] [SAFE-02] [SAFE-03] [AUTH-01] [AUTH-02] [AUTH-03] [EVAL-06] [ORIG-05] [SAFE-09] [MOD-04] [ORIG-14] [ORIG-15]

10.2 Multi-tenant isolation and cache safety

Carry tenant identity through every layer of the system. Appending it at the UI does not create an isolation boundary.

Use this checklist:

• derive tenant and principal from authenticated server-side identity;
• enforce tenant and document ACL constraints inside the source query and before or during vector candidate generation and reranking; post-filtering a global top-k result is not an isolation boundary;
• use tenant-scoped credentials, storage namespaces, vector indexes or filters, and encryption boundaries appropriate to risk;
• namespace exact, semantic, and retrieval response caches by a nonempty authenticated tenant/principal or ACL scope plus every response-affecting policy, corpus, prompt, model, schema, and freshness version; reauthorize provenance on every hit and purge entries after ACL, role, user, or document changes;
• require exact-token matching plus runtime ownership and refcount isolation for prefix/KV reuse; a provider organization or project is not an application tenant boundary;
• put an authenticated and authorized ingress boundary in front of every model-server route; treat built-in API-key flags as endpoint-scoped until verified, inventory every reachable route, disable unused compatibility, administration, metrics, and profiling APIs, and negative-test access;
• treat model repositories, processor and tokenizer code, custom operators, and conversion steps as executable supply-chain inputs; allow only reviewed, digest-pinned artifacts, and load or convert them in an isolated environment without production credentials or broad egress; a runtime flag such as trust_remote_code=false is not a security boundary for an unreviewed loader;
• enforce decoded-byte, duration, frame, pixel, and accelerator-memory limits across every audio, image, and video path; derive limits from bounded decode output rather than trusting caller-controlled container metadata, and reject the request before it can exhaust shared worker or GPU capacity;
• enforce resource limits at the earliest acquisition step: cap request-field length and label cardinality before hashing or metrics, bound compressed and remote media before full download or decompression, recheck decoded duration, frames, pixels, and sampler output after transformation, and keep per-request failures from poisoning a shared scheduler or engine;
• validate caller-supplied token IDs on the host before accelerator work: require integers within both the lower bound of zero and the loaded model's vocabulary range on every token-ID input path; reject malformed input without poisoning shared engine state, and regression-test the affected endpoints under concurrent tenants; the vLLM negative-token-ID advisory^[SAFE-11] demonstrates why checking only the upper bound is insufficient;
• do not let request fields select a decoder, kernel, plugin, adapter, or other privileged backend unless an operator allowlisted it and included it in startup resource reservation and isolation policy;
• treat agent runtimes, sandbox provisioners, gateways and proxies, bridges, installers, updaters, and inference services as privileged policy-enforcement code; pin and patch them, verify certificates and artifact integrity, normalize paths before layer-7 policy, avoid building shell commands from untrusted data, and test escape, injection, traversal, and authorization-bypass paths;
• treat the serving engine, custom kernels, and batching scheduler as tenant-isolation code; pin and inventory the runtime, monitor its security advisories, patch within the risk window, and test response ownership under concurrent tenants; a model-server flaw can cross-contaminate responses even when cache namespaces are correct;
• record the requested processing geography and the provider-resolved organization, project, workspace, and region when the API exposes them; reject unexpected resolution before sensitive data crosses the boundary;
• exclude sensitive content and identifiers from shared scheduler metadata, and address timing or resource side channels with quotas, traffic shaping, or physical silos where the threat model requires;
• bind every request, tool call, callback, queue item, and result to immutable tenant, principal, conversation, run, and call IDs; reject mismatched or late results and clear or zero pooled worker, sandbox, and request-local state before reuse;
• isolate background jobs, queues, temporary files, sandboxes, logs, traces, and eval environments;
• enforce tenant quotas for requests, context, KV memory, tools, and cost;
• test deleted users, changed roles, revoked documents, cache hits after ACL changes, and concurrent tenants;
• seed cross-tenant canaries and alert if they are retrieved, generated, logged, or cached outside their scope.

Physical isolation is not always required. The isolation boundary still has to match the data and threat model. High-risk workloads may require separate projects or accounts, networks, encryption keys, compute, and tool servers. Google's multi-tenant agentic AI reference architecture^[TEN-01] shows what that stricter topology looks like.

Sources [TEN-01] [AUTH-01] [AUTH-02] [SAFE-03] [SAFE-04] [SAFE-05] [SAFE-06] [SAFE-07] [SAFE-08] [SAFE-09] [SERV-08] [MOD-02] [MOD-04] [SAFE-11] [SAFE-12] [SAFE-13] [SAFE-14] [SAFE-15] [SAFE-16] [SAFE-17] [SAFE-18]Author synthesis: the isolation checklist spans the cited permission and tenancy boundaries.

11.1 In-context learning, RAG, fine-tuning, and distillation

Choose the method based on what has to change.

These methods can work together. A practical sequence is:

1. Define the evals and establish a strong-model baseline.
2. Improve the harness, context, tools, and deterministic validation.
3. Add RAG for changing or attributable knowledge.
4. Route simpler cases to smaller models only when evals show that it is safe.
5. Fine-tune persistent behavior gaps with clean representative data.
6. Distill only when the volume and task stability justify a student model.

Use each method for the job it can actually do: fine-tuning changes stable behavior rather than current facts, RAG supplies knowledge without enforcing behavior, in-context examples do not create durable state, and distillation is not lossless. OpenAI's model optimization guide^[OPT-01] starts with evals because model behavior varies across families and snapshots.

Sources [OPT-01] [CTX-01] [RAG-01] [DIST-01]Author synthesis: the selection matrix matches the intervention to the constraint.

11.2 Diagnosing the four-way tradeoff

Diagnose the symptom, measure the relevant stack signals, and track where the fix moves risk.

Change one major variable at a time when you can. If you combine a model swap with a prompt rewrite, a new retriever, quantization, and a router change, you will have almost no way to attribute a regression.

Sources [SERV-05] [SERV-04] [OPT-01] [OBS-01]Author synthesis: the diagnostic matrix keeps quality, latency, cost, and reliability in one decision.

12.1 Production failure matrix

Before launch, define a detection signal, containment path, and regression test for every known failure.

Sources [OUT-01] [TOOL-01] [SERV-01] [EVAL-02] [SAFE-01] [TEN-01]Author synthesis: the failure matrix combines the controls in sections 4–20.

12.2 Production readiness checklist

Use this checklist before launch and again after any material change.

Contract and harness

• Success, partial, refused, blocked, degraded, and failed states are defined.
• End-to-end quality, latency, cost, reliability, freshness, and safety SLOs exist.
• State, versions, retries, idempotency, budgets, and termination are harness-owned.
• Consequential actions have deterministic authorization and approval boundaries.

Context and retrieval

• Context is minimal, authorized, source-attributed, versioned, and reproducible.
• Retrieval recall and generation grounding are measured separately.
• The retrieval strategy (lexical, dense, or hybrid), plus chunking, k, fusion, and reranking when used, is selected and tuned through evals.
• Updates, deletions, ACL changes, and cache invalidation meet a freshness SLO.

Inference and caching

• Queue, TTFT, mean TPOT, p95/p99 ITL, stall rate, goodput, and p95/p99 end-to-end latency are load-tested on real length distributions.
• KV capacity, fragmentation, eviction, preemption, quotas, and OOM behavior are known.
• Prefix and semantic caches have distinct policies and metrics.
• Any speculative, quantized, or distilled route passes the full regression suite.

Outputs, tools, and agents

• Structured outputs pass syntax, schema, semantic, authorization, and result checks.
• Tool names, arguments, results, errors, and risk annotations are strict and versioned.
• Retryable side effects are idempotent and tested under unknown commit status.
• Idempotency keys survive retries, resumes, handoffs, and fallbacks and are recorded atomically with mutations.
• Concurrency, fan-out, mutation, and shared fallback budgets are enforced atomically with turn, tool, token, time, cost, risk, retry, and no-progress budgets.
• Refusal, incomplete/truncated, repair, and fallback paths are tested; repair cannot authorize mutations.

Evals, operations, and economics

• Golden, regression, adversarial, stochastic, and human eval layers exist.
• Release gates are slice-aware and evaluator versions are pinned.
• Bounded traces cover routing, context, retrieval, model, validation, tools, and outcome, and linked journey IDs connect asynchronous or multi-session work.
• Cost is attributed per feature, workflow, tenant, journey, route, and successful outcome.
• Drift dashboards and rollback procedures are exercised.

Security and tenancy

• Untrusted content is never treated as authority merely because it is in context.
• Least privilege, sandboxing, egress controls, DLP, and approvals limit blast radius.
• Tenant identity scopes retrieval, caches, tools, logs, traces, queues, and quotas.
• Cross-tenant canaries, revoked-access tests, and incident procedures exist.
• Fallback cannot bypass refusal, authorization, residency, retention, or unknown-commit reconciliation.
• Credentials and bearer tokens never enter model context; OAuth audiences are validated and token passthrough is prohibited.
• Approvals are bound to the exact action and state version.
• Cache hits fail closed on missing tenant scope, are reauthorized, and are invalidated after access changes.

Sources [SAFE-03] [OBS-02] [EVAL-02] [TEN-01]Author synthesis: the operating guidance is recast as a production-readiness checklist.

12.3 Recommended learning order

1. Define success and build a small golden set.
2. Build the deterministic harness: state, schemas, tools, budgets, and termination.
3. Learn context construction and retrieval with provenance and ACLs.
4. Add traces, outcome metrics, and full-workflow cost attribution.
5. Threat-model injection, leakage, side effects, and multi-tenant boundaries.
6. Load-test prefill, decode, batching, KV memory, and queue behavior.
7. Only then optimize with caching, routing, quantization, speculative decoding, fine-tuning, or distillation.

Do the optimization work after the system is measured and safe. Otherwise, the change can improve speed while leaving the outcome wrong or unsafe.

Sources [EVAL-02] [EVAL-03] [OPT-01]Author synthesis: the learning order follows the sequence I would use to build the system.