What an AI production-readiness review should actually test

AI prototypes are optimized to demonstrate possibility. Production systems have to survive ambiguity, malicious inputs, stale context, provider outages, cost limits, user mistakes and operational pressure. That changes what a useful review must test before a generative-AI feature or agent is released.

This framework draws on NIST's Generative AI Profile, the OWASP Top 10 for LLM Applications 2025, and current responsible-AI deployment guidance from Microsoft and Google Cloud. It is a practical release-gate framework rather than a certification claim or a promise that every AI failure can be eliminated.

1. Define the use case and harm model before testing

A production review should begin by documenting what the system is intended to do, who will use it, what data it can access and which outcomes would be unacceptable. NIST's AI risk-management approach emphasizes context because risk cannot be separated from intended use and consequence.

• Who are the intended users, and which users are explicitly out of scope?
• Which tasks may the system perform, and which tasks must it refuse or escalate?
• What private, regulated or business-sensitive data can the system access?
• Can the system only answer questions, or can it also call tools and change external systems?
• Which failures would create meaningful financial, legal, safety or reputational harm?

2. Test representative tasks, not only generic benchmarks

Generic benchmark scores can be informative, but they do not prove that a system works for a specific workflow. The review should include a representative test set drawn from real task patterns, expected edge cases and known failure modes. The acceptance criteria should be tied to the use case rather than copied from another product.

• Normal production tasks and common user phrasing.
• Ambiguous or incomplete requests.
• Inputs that should trigger abstention or escalation.
• Conflicting context and stale reference material.
• Repeated runs where consistency matters operationally.

3. Test groundedness and unsupported claims

For retrieval-augmented or knowledge-based systems, production testing should examine whether the model uses the right source material, whether it can distinguish between supported and unsupported claims, and what happens when the available context is incomplete or contradictory.

The goal is not to promise that hallucinations disappear. The goal is to define the types of unsupported output that matter in the workflow, measure them against representative cases and design fallback or escalation behavior where confidence is insufficient.

4. Test security at the application boundary

OWASP's LLM risk guidance highlights prompt injection, sensitive-information disclosure, supply-chain weaknesses and excessive agency. Those risks are not limited to the model itself. They often emerge from how the application connects retrieval, tools, permissions and external content around the model.

• Direct prompt injection intended to override system behavior.
• Indirect prompt injection embedded in retrieved documents, web pages or messages.
• Attempts to reveal secrets, hidden instructions or confidential context.
• Tool calls that exceed the user's or system's intended permissions.
• Malicious or compromised external dependencies that can influence model behavior.

5. Review action permissions and reversibility

The risk profile changes sharply when an AI system can do more than generate text. Agents and tool-using assistants may send messages, change records, create transactions or modify infrastructure. Microsoft responsible-AI guidance recommends deciding explicitly which consequential actions require human approval.

A production review should classify actions by impact and reversibility. Read-only access, reversible drafts and high-impact irreversible changes should not automatically receive the same approval model. Least privilege should apply to tools in the same way it applies to people and services.

6. Verify privacy and data handling

Teams should verify what user inputs are logged, how long they are retained, whether they may be reused by providers, which subprocessors receive them and whether confidential or personal data can enter prompts or retrieved context. The answer should come from the actual application and provider configuration, not from an assumption based on a product brochure.

• Prompt and response logging behavior.
• Retention and deletion controls.
• Provider training or reuse settings where applicable.
• Access controls around conversation history and retrieved context.
• Cross-border or regulated-data considerations relevant to the use case.

7. Treat generated output as untrusted where appropriate

Google Cloud's responsible-AI guidance warns that automatically rendered model output such as Markdown or HTML can create indirect prompt-injection and data-exfiltration risk. Production reviews should therefore include the presentation layer: sanitization, safe external links, image handling and Content Security Policy where relevant.

8. Test operational limits and degraded modes

A production system also has to behave sensibly when the model provider is slow, unavailable or unexpectedly expensive. Test timeouts, rate limits, context-size failures, provider errors and fallback behavior. If the feature is non-essential, graceful degradation may be safer than repeatedly retrying an unavailable dependency.

• Provider outage and timeout behavior.
• Rate-limit handling and backoff.
• Unexpectedly long prompts or tool chains.
• Cost and token-usage guardrails.
• Fallback model or non-AI path where one exists.

9. Add observability before launch

Microsoft and Google both emphasize post-launch monitoring. A system that cannot show how often model calls fail, how tools behave, whether safety or groundedness problems are increasing, or which versions are active is difficult to operate responsibly after release.

• Model and tool errors.
• Latency, rate-limit and timeout patterns.
• Safety or policy escalations.
• User feedback and reports of incorrect output.
• Model, prompt, retrieval and tool versions associated with an incident.

10. Prepare incident response and rollback

NIST's Generative AI Profile treats incident disclosure and lifecycle risk management as part of responsible operation. Before release, the team should know who can disable the feature, revoke tool permissions, switch models, roll back prompts or retrieval changes and communicate an incident to affected stakeholders.

Production readiness is therefore not a permanent label. Changes to the model, prompt, retrieval corpus, tool permissions or application boundary can alter the risk profile. Material changes should trigger targeted re-evaluation rather than assuming the original release review still applies.

A practical release gate

• Use case, users and prohibited outcomes documented.
• Representative test set approved.
• Task-quality and safety acceptance criteria defined for the use case.
• Prompt-injection, sensitive-data and tool-abuse tests completed.
• Tool permissions and human-approval gates reviewed.
• Privacy, retention and provider-data behavior verified.
• Rendering and client-side output handling reviewed.
• Monitoring, fallback and rollback paths tested.
• Incident owner and disable path known.
• Material model, prompt, retrieval or tool changes trigger re-review.

Production readiness is an operating capability

The useful production-readiness review is not a ceremony and it is not a single benchmark score. It is evidence that the team understands how the AI system can fail, has tested the highest-impact failure modes, has constrained what the system can do, and can observe, disable or recover the feature when reality differs from the test environment.

What to check before you commit

1. Define the use case and unacceptable outcomes before selecting evaluation metrics.
2. Test the application boundary — retrieval, tools, permissions and rendering — not only the model.
3. Use human approval where consequential actions are difficult to reverse.
4. Add production monitoring and rollback before launch, not after the first incident.
5. Re-evaluate when models, prompts, retrieval data or tool permissions materially change.

A note on measurement

Teams that treat AI production readiness as an engineering project usually measure the wrong thing. Instrument the business outcome first — cycle time, cost per transaction, resolution rate, revenue retention — then work backwards to the technical metrics that move it.