AI can create a convincing demo quickly. That demo should not become a production system until named people own its permissions, testing, data handling, deployment, recovery and long-term maintenance.

A prototype proves that a path might work. Production requires evidence that the system remains useful when inputs are ambiguous, services fail, permissions change and somebody other than the original builder has to operate it.

That distinction is becoming more important as code generation gets faster.

A working demo can hide unfinished decisions

The happy path is usually the easiest part of a software project.

A prototype might summarize a document, answer a product question or create a customer record. In the demonstration, the input is clean, the account is authorized, the connected systems respond and the presenter knows which questions not to ask.

Production receives misspelled requests, incomplete records, hostile instructions, duplicate submissions, expired credentials and users with different permissions.

The risk is not simply that the model might produce a bad sentence. A connected system can disclose private information, overwrite a record or trigger an action that is difficult to reverse.

OWASP describes “excessive agency” as a combination of excessive functionality, permissions or autonomy. Its examples include granting a document-reading agent an extension that can also modify and delete documents. See OWASP’s excessive agency guidance.

The practical rule is simple: the model may help decide what to propose, but the application must decide what is authorized.

Require six production proofs

Before moving an AI-assisted workflow into production, ask for evidence in six areas.

Six production proofs
Production proofThe questionEvidence worth reviewing
PurposeWhat exact job is the system allowed to perform?Defined inputs, outputs, exclusions and escalation rules
PermissionWhat can each user and system identity read or change?Server-side authorization, narrow tool scopes and approval boundaries
PerformanceHow does it behave across normal, difficult and hostile inputs?Representative test set, failure categories and release thresholds
PrivacyWhat information is collected, transmitted, logged and retained?Data map, vendor settings, retention rules and deletion process
RecoveryHow do we contain a failure and restore a safe state?Kill switch, rollback procedure, backups and tested incident roles
OwnershipWho maintains the system after launch?Named owners, monitoring responsibilities, dependency register and exit plan

This is an editorial framework, not a published standard. Its purpose is to force a team to prove the operational system instead of presenting another polished conversation.

A missing proof does not always mean cancel the project. It may mean narrowing the first release until the risk is supportable.

Authorization belongs outside the prompt

“Only show the user information they are allowed to see” is a useful instruction. It is not an access-control system.

The application should authenticate the requester, determine permissions using trusted server-side rules and constrain every downstream operation. The model should receive only the tools and data needed for the current task.

For an action with financial, legal, privacy or account consequences, consider requiring an explicit human approval after the proposed change is shown in a clear, reviewable format.

Logs should identify the requester, tool, relevant parameters, authorization decision and outcome. Sensitive values should be removed or protected rather than copied indiscriminately into traces.

The current OWASP MCP Top 10 remains in beta, which is important context rather than a reason to ignore it. It identifies insufficient authentication and authorization, privilege expansion and missing audit telemetry as distinct MCP risks. Review the OWASP MCP Top 10.

Test the system, not just the prompt

A few impressive conversations are anecdotes. They do not establish reliability.

Create a test set from the work the system will actually encounter:

  • common requests;
  • incomplete or contradictory information;
  • attempts to obtain restricted data;
  • instructions hidden in retrieved documents;
  • unavailable dependencies;
  • duplicate actions;
  • requests that should be refused or sent to a person.

Record the expected outcome before testing. Then classify failures: unsupported answer, wrong source, incorrect action, permission violation, unusable escalation or operational error.

An OpenAI evaluation recipe published October 6, 2025 describes an analyze–measure–improve cycle built around reviewing failures, establishing a measurable baseline and making targeted changes. The recipe is now marked archived, so its model and API examples may be outdated; the evaluation discipline remains the relevant principle. Read the evaluation flywheel recipe.

Do not reduce everything to a single accuracy percentage. A system can perform well overall and still fail consistently on the small category that causes the most damage.

Set separate release thresholds for consequential behavior. A customer-facing knowledge assistant might tolerate an occasional harmless “I don’t know.” It should not tolerate exposing another customer’s documents.

Privacy is a lifecycle, not a checkbox

A prototype often begins with convenient access to a large data collection. Production should start by asking what the system genuinely needs.

Map information from collection through transmission, processing, logging, retention and deletion. Include prompts, retrieved documents, tool results, feedback, analytics and human-review queues.

NIST’s Privacy Framework treats privacy risk across the complete data lifecycle, from collection through disposal, and distinguishes privacy harm from cybersecurity incidents alone. See NIST’s Privacy Framework introduction.

That distinction matters. A perfectly secured system can still collect more information than it needs, retain it too long or use it in ways people do not reasonably expect.

Minimize data before trying to protect it. Document the remaining flows. Confirm provider settings and contractual terms rather than assuming that a model’s marketing label resolves every obligation.

A code rollback is not a time machine

Teams frequently say a release can be rolled back. Ask what that means.

Cloudflare Workers supports gradual deployments that split traffic between versions while teams monitor errors and can return to a stable version. See Workers gradual deployments.

Its rollback documentation also warns that connected resources are not changed by a code rollback and that earlier code can fail when data structures have changed. Read Workers rollback limitations.

That is the larger lesson: application code, prompts, models, schemas, stored records and external actions have different recovery paths.

If an agent sent an email, changed a price or deleted a record, restoring yesterday’s code does not undo the action. Production planning needs idempotency, reversible operations where possible, backups, data-migration plans and procedures for correcting downstream consequences.

A rollback should be rehearsed before it becomes an emergency.

Three illustrative scenarios

These are hypothetical examples, not DeCort client results.

Lead intake: An assistant can summarize an inquiry and recommend a category. Deterministic code should validate required fields, prevent duplicate submissions and route the record. A person should review high-value or ambiguous leads before consequential outreach.

Enterprise knowledge: An assistant can retrieve approved internal material and cite its sources. Access control must be applied to every retrieval using the requester’s identity. The assistant should not receive an unrestricted document index and then be asked to remember the confidentiality rules.

B2B commerce: An assistant can help a customer find compatible products or prepare a draft quote request. Inventory, account-specific pricing, tax status and order submission should come from authoritative systems and controlled transactions. The model should not invent a commercial term or independently place an order.

In each case, deterministic automation is often the safer choice for validation, routing and final execution. Use model judgment where language or ambiguity makes it valuable.

Portability is part of ownership

A production system needs an exit path.

Keep prompts, tool definitions, evaluation cases, content schemas and operating procedures in repositories the business controls. Record which behaviors depend on a particular model or vendor. Export important logs and data in usable formats where contracts and privacy requirements allow.

Avoid letting the only working version of the system live inside one person’s account.

NIST’s Secure Software Development Framework recommends integrating secure practices into the development lifecycle rather than treating security as a final-stage addition. It also provides a common vocabulary that software purchasers can use with suppliers. Read NIST SP 800-218, SSDF version 1.1.

Apply that thinking to AI systems: document components, dependencies, ownership and verification throughout development.

Before launch, run a failure rehearsal

Choose one meaningful workflow and conduct a short tabletop exercise:

  1. A user provides manipulated source material.
  2. The model attempts a tool call outside the user’s authority.
  3. The downstream service accepts the request but times out before confirming it.
  4. The team notices unexpected records the next morning.
  5. The original developer is unavailable.

Ask who detects the problem, how the workflow is stopped, which logs answer what happened, how affected records are found and who decides when service resumes.

If the answers depend on the person who built the demo remembering everything, the product is not ready.

AI can make the first version arrive faster. It does not remove the need for engineering, governance or accountable ownership.

The question after every successful demo should be: Who owns the system when the demonstration ends?

Sources