GPT-6 makes prompt caching easier to observe and control, but savings still depend on context architecture. Here is a production workflow for structuring prefixes, diagnosing misses, testing behavior, and deciding whether caching complexity pays.
Long agent prompts tend to grow according to organizational convenience. A system policy is pasted first, tool schemas arrive from a registry, repository notes are inserted wherever the orchestration layer can reach them, and timestamps or user metadata get attached near the top. The resulting request may remain readable and even produce good answers. It can still be economically hostile to prompt caching.
OpenAI’s GPT-6 prompt-caching update makes this problem more actionable. The company says eligible shared prefixes can receive cache discounts when reused within a 30-minute window, while a new dashboard, diagnostics, explicit breakpoints, and prewarming controls expose more of the mechanism. The examined announcement says the improved system launched with the GPT-6 family, but it does not provide a separate model release date. OpenAI’s GPT-6 prompt-caching announcement
The important change is not simply a larger discount. Prompt caching is becoming an application-architecture concern. Context ordering, message boundaries, tool stability, compaction, model routing, and configuration changes can determine whether thousands of tokens are reused or processed again. A shared passage is not necessarily a reusable cached prefix.
The useful response is a workflow, not a prompt trick: map context by mutation rate, establish cached and uncached baselines, place deliberate boundaries, inspect production misses, and verify agent behavior on both paths. This turns cache optimization into the same kind of controlled engineering work applied to databases, CDNs, and build systems.
What the cache actually reuses
OpenAI’s documentation says the cache preserves key-value states produced while processing an unchanged prompt prefix. It does not merely store a text response, and the model still processes new input and generates a new output. Reuse requires the rendered beginning of a later request to match an eligible earlier prefix. The prompt-caching guide also says cached prompts do not guarantee identical outputs.
That distinction explains why ordinary prompt tidiness is insufficient. Suppose two requests contain the same 8,000-token policy and tool catalog. If request-specific metadata is inserted before that material, the common content no longer begins at the same place. The application has semantic reuse but not prefix reuse. If a tool schema changes near the beginning, everything after the changed token may also become ineligible.
Order context by stability, not by discovery time Put stable, broadly reused material first. Put workspace-stable and session-stable material next. Append request-specific and rapidly changing content last. A breakpoint is a commitment about the end of a reusable prefix. It is not a hint that arbitrary matching text elsewhere should be cached.
A practical ordering for a long-running coding agent Application invariant Safety policy, operating procedure, response contract Release-level Place first; version deliberately Tool surface Tool names, descriptions, schemas, ordering Low to medium Keep definitions stable; restrict availability without deleting definitions Repository invariant Architecture guide, conventions, test commands Commit or workspace-level Cache when shared across enough requests Session memory Accepted plan, compressed discoveries, prior messages and tool results Turn-level Append rather than rewrite; use later breakpoints where useful Retrieved evidence Files, search results, diagnostics, logs Request-level Usually place after stable prefixes Current task User request, timestamp, request ID, selected files Every request Keep in the uncached suffix unless repeated
Cache boundaries are design boundaries
GPT-5.6 and later support implicit and explicit caching according to the retrieved OpenAI guide. In implicit mode, OpenAI selects eligible message boundaries. In explicit-only mode, the developer marks selected content-block boundaries, and a request without an explicit breakpoint does not use prompt caching. The guide states that GPT-5.6 and later permit up to four cache writes per request.
Explicit control is valuable when a large stable prefix is followed by a volatile suffix. Without a boundary after the stable content, an implicit write may extend through the current user message. A later request with a different user message can fail to match that longer endpoint. The text is shared, but no eligible boundary identifies the useful shared portion.
Multiple boundaries support a hierarchy. One can follow global instructions and tools, another can follow repository guidance, and another can follow durable session state. The service can look for the longest eligible match and fall back to an earlier boundary. This resembles layered memory more than a single cache switch.
That memory hierarchy connects directly to pxpipe and the future of agent memory. The design question is not only what the agent should remember. It is also which representation should remain stable, where it belongs in the context, and how often rebuilding it is economically justified.
Worked example: restructuring a coding agent
A hypothetical repository maintenance agent The numbers and workload below are illustrative, not measured results. They show how to reason about a migration without assuming that a provider’s maximum discount will appear in production.
Consider an agent that receives 14,400 input tokens on its first turn. Its orchestrator currently assembles the request in this order: 150 tokens of timestamp and request metadata; 3,200 tokens of developer instructions; 2,100 tokens of tool definitions; 2,800 tokens of repository guidance; 1,600 tokens of compressed session memory; 3,700 tokens of retrieved files; and an 850-token task.
The first 150 tokens change on every call. They therefore poison reuse for everything after them. Tool selection also removes unused tool definitions, so even requests with similar instructions frequently diverge near the beginning. The agent’s context is logically repetitive but physically unstable.
Build a mutation map. Classify each component by its expected lifetime: deployment, tool-catalog version, repository commit, session, turn, or request. Record the component’s position and token count. Do not optimize yet. Create an invariant prefix. Move the 3,200-token instructions first and the 2,100-token tool catalog second. Freeze tool ordering and schema serialization. Follow them with the 2,800-token repository guide. Version this 8,100-token prefix as an application artifact. Place the first boundary. Add an explicit breakpoint after the 8,100 stable tokens. Move timestamps, request IDs, selected-tool restrictions, and the current task after it. Use allowed-tools controls rather than deleting definitions when the provider and model support that approach. Treat session state as a second layer. Append the 1,600-token session memory after the stable prefix. If it remains unchanged for several turns, place a second boundary after it. When memory changes, create a new appended state or accept that this layer will be rewritten without disturbing the earlier 8,100 tokens. Keep evidence and the task late. Place retrieved files, live diagnostics, timestamps, and the current request after the last useful breakpoint. These items often have low reuse probability, so writing them can add cost without producing future reads.
Assume the redesigned agent handles ten requests while the stable prefix remains eligible. Using the documented GPT-5.6-and-later multipliers as an illustrative cost model, one write of 8,100 tokens costs the equivalent of 10,125 ordinary input tokens at 1.25×. Nine complete reads cost another 7,290 token-equivalents at 0.1×. The stable portion therefore totals 17,415 token-equivalents, compared with 81,000 if all ten requests processed it at the ordinary input rate.
That is not the application’s total savings. The variable suffix, output tokens, reasoning tokens, misses, and any additional writes still cost money. The calculation also assumes full reuse within the eligibility period. Its purpose is to expose the variables: stable-prefix length, write multiplier, read multiplier, number of successful reads, and miss frequency.
What must remain behaviorally equivalent The tool definitions available to the model remain semantically identical even when callable tools are restricted separately. Instruction precedence remains correct after stable rules move earlier and dynamic overrides move later. Repository guidance is tied to the intended commit or version rather than silently surviving a code change. Conversation items, tool calls, and tool results preserve their original order. The agent still reaches the same authorization, verification, and stopping decisions across cached and uncached requests.
The cache layout belongs inside the agent harness, alongside permissions, tool execution, context control, and verification. Building a coding agent from scratch explains why those orchestration choices, rather than the model call alone, determine whether an agent is dependable.
Measure economics, not only hit rate
A cache-hit percentage can improve while the system becomes worse. An application might pad a prefix with marginal material to cross a minimum cacheable length, write prefixes that are never read, or retain a huge history that increases total input despite a strong hit rate. It might also reduce time to first token while increasing end-to-end latency through unrelated retrieval or tool work.
OpenAI recommends tracking cached tokens, cache-write tokens, total input tokens, latency, and realized cost. Its documentation defines token cache-hit rate as cached tokens divided by total input tokens, aggregated over a useful group such as user, workspace, or day. OpenAI’s monitoring guidance
Evaluate cache architecture at the workload level Cached-token ratio How much submitted input was served from cache? A high ratio does not prove lower total cost Cache-write tokens How much new state was written and billed? Ignoring writes exaggerates savings Input cost per successful task Did the whole workflow become cheaper? Per-request averages can hide retries and failures Time to first token Did prefix reuse improve initial responsiveness? It is not end-to-end task latency Task completion and policy checks Did restructuring preserve behavior? Output similarity alone can miss tool or authorization changes Miss reason by release Which mutations or settings broke reuse? Aggregate hit rate can conceal one damaging deployment
Compare at least three cohorts: the current production layout, the proposed layout with caching disabled or intentionally missed, and the proposed layout under representative reuse. The uncached run of the new layout matters because reordering instructions can change behavior independently of caching. The cached and uncached versions of the same layout then test whether the cache path introduces any operational difference.
Token counts should also be treated as production telemetry rather than static prompt estimates. Tokenizer or model changes can alter thresholds and economics. The migration discipline in Treat Tokenization as Production Infrastructure applies here: measure with the actual model, settings, schemas, and traffic distribution.
Diagnose misses systematically
OpenAI’s diagnostic facility compares a current request with an earlier completed response and can classify changes involving the model, cache key, service tier, tools, output format, reasoning effort, verbosity, compaction, or input. The comparison option does not load the earlier conversation or alter caching behavior. Actual reuse must still be read from the response usage fields. Prompt cache diagnostics documentation
Select a valid baseline Use a recent response from the same organization whose prefix the new request was expected to reuse. Save response IDs with deployment, model, schema, and prompt-layout versions. Inspect usage before explanation Record input tokens, cached tokens, cache-write tokens, model, service tier, latency, and cost. A diagnostic cache-hit classification can still accompany newly processed suffix tokens. Request a comparison Pass the baseline response ID through the documented comparison option. Store the returned reason and estimated missed tokens with the trace. Fix one class of mutation Stabilize tool serialization, move timestamps, preserve message boundaries, or align settings. Diagnostics report the first classified reason, so another cause may appear after the first is corrected. Replay representative traces Run several requests from the affected traffic shape. Compare cost and behavior, not just whether one diagnostic changed from miss to hit.
A useful release artifact is a prefix fingerprint calculated by the application for each intended boundary. The fingerprint should cover the application-controlled serialized content and relevant configuration, without pretending to reproduce the provider’s hidden rendering. When a hit-rate regression appears, operators can ask whether the application’s own prefix changed before investigating routing or provider behavior.
Mutations that look harmless
Editing instead of appending Extending an existing message can remove the earlier message-end boundary from consideration. Preserve the original message and append a new one when the conversation permits. Rebuilding the tool array Adding, removing, reordering, or changing descriptions and schemas can invalidate reuse. Keep definitions stable and control callability separately. Embedding volatile metadata early Timestamps, request IDs, user names, experiment assignments, and workspace state can alter every following token. Move them after the durable boundary. Changing output configuration A structured-output schema or verbosity change can affect the rendered prefix. Treat these as versioned configuration, not invisible transport metadata. Compacting history Compaction can reduce total tokens while lowering cache reuse because it replaces earlier context. Evaluate total cost and task quality rather than defending hit rate. Changing reasoning effort at request level The retrieved documentation says supported GPT-6 models can instead append a configuration update while retaining the original request-level effort, preserving the earlier prefix.
Limits and tradeoffs
Caching does not remove input from rate-limit accounting. OpenAI’s guide says cached input tokens still count toward tokens-per-minute limits. It also does not guarantee a hit: eligibility depends on prefix compatibility, lifetime, routing, and model behavior. For GPT-5.6 and later, the retrieved guide documents a minimum cacheable visible prefix of 1,024 tokens and a 30-minute minimum lifetime after the latest write or reuse, while noting that model behavior differs for earlier generations.
Prewarming moves prefix processing ahead of a user request, but it is not free. The guide says prewarmed tokens are billed at the cache-write rate. It therefore fits predictable bursts or startup paths where a later request is likely to arrive within the eligibility period. Prewarming low-probability prefixes merely converts possible latency into certain write cost.
Privacy and tenancy also need explicit treatment. The documentation says caches are not shared across organizations or regional processing boundaries. It describes optional cache keys on GPT-5.6 and later for separate accounting by customer or user, including protection against probing for another group’s cache activity. That is an accounting and cache-observability decision, not a general security-isolation boundary.
Most importantly, a more stable prompt can become a less current prompt. Repository guidance must be invalidated when its source changes. Policies need versions. Tool schemas must remain synchronized with executors. Stability should come from controlled versioning, not from allowing stale context to persist.
Use this decision process next
Quantify reusable mass. Sample production traces and measure how many leading tokens could remain identical across requests. Segment by workflow, tenant, model, and session duration. Estimate reuse within the lifetime. Count likely reads per write inside the supported eligibility window. A long prefix used once is not a caching opportunity. Choose implicit or explicit boundaries. Use implicit caching for append-only conversations where eligible message endings align with reuse. Use explicit boundaries when volatile suffixes would otherwise be written or obscure a stable prefix. Version the prefix. Give instructions, tools, reference material, output format, and relevant settings an application-level layout version. Log that version on every response. Run behavioral evaluations. Test task completion, tool selection, authorization, structured output, and stop conditions on the old layout, new uncached layout, and new cached layout. Roll out with traces. Start with a small traffic slice. Monitor cached tokens, writes, cost per completed task, latency percentiles, diagnostic reasons, and quality regressions. Keep or remove the complexity. Keep explicit cache architecture only if realized savings or latency improvements justify versioning, observability, and invalidation work. Otherwise retain the simpler prompt and rely on the provider’s defaults.
The broad lesson is vendor-aware but not vendor-bound. Any prefix cache rewards stable beginnings, intentional boundaries, reuse locality, and disciplined invalidation. Provider-specific controls determine how those principles are expressed, but the architectural work belongs to the application.
GPT-6’s diagnostics and controls make that work easier to see. They do not make it automatic. The teams that benefit most will not be the ones with the longest prompts or the highest advertised discount. They will be the ones that can explain exactly which context is stable, why it is safe to reuse, how frequently it is reused, what invalidates it, and whether the resulting agent still behaves correctly.
Subscribe free to Harshith's Newsletter to read every article in the interactive edition.
Harshith Vaddiparthy works with founders, operators, and teams on practical AI products, workflows, advisory, training, and mentorship. This no-JavaScript version preserves the page's core information and navigation.