Enterprise RAG evaluation should answer a business question: can this system respond safely, reliably, and traceably within its intended job? Before production, define what it may answer, which sources it may use, when it must abstain, who may see each document, and its operational limits. Test those commitments with representative and adversarial cases; one “accuracy” score cannot provide that evidence.
This checklist turns production readiness into observable acceptance gates for product, engineering, security, and domain teams. The outcome should be a documented go, conditional go, or no-go decision—not a collection of attractive sample conversations.
Start with an acceptance contract, not a benchmark
RAG retrieves passages from an external knowledge source and supplies them as context for a model-generated response. If you need the wider content lifecycle and governance picture, begin with this guide to AI-powered knowledge management. For production acceptance, however, keep the scope narrower: prove that the system retrieves authorized evidence, uses it correctly, shows usable provenance, and fails safely when evidence is missing.
Write a one-page acceptance contract before creating test cases. It should identify:
- supported user groups and decisions;
- approved repositories, document types, languages, and freshness;
- prohibited content and actions;
- required answer behavior, including citations and abstention;
- security, privacy, service, and cost constraints;
- the owner who can approve each gate and the evidence they require.
The contract must reflect organizational risk tolerance. The NIST Generative AI Profile is useful for structuring governance, measurement, and risk-management responsibilities, but it does not replace use-case-specific acceptance criteria.
Classify the use case before choosing metrics
The same failure has different consequences in an employee handbook assistant and a system that influences consequential decisions. Describe the plausible impact of unsupported answers, omitted qualifications, stale policies, permission leaks, and outages. That classification determines test depth, reviewers, escalation, and whether the assistant may answer directly or only locate evidence.
Set explicit use-case gates—for example, every sensitive repository needs an owner, every high-risk answer needs a source, and unsupported queries need a safe response. These illustrate gate types, not universal thresholds; your organization must set its own measurable limits.
Also define scope exclusions. If the system was tested for HR policy lookup, passing those tests does not approve it for employee-performance decisions. If only English and Persian documents were evaluated, do not imply support for every language. Boundaries belong in the product interface, operating procedure, and test report.
Build a representative golden set
A golden set is a versioned collection of realistic queries with expected evidence and behavior. Assemble it from actual user tasks, approved support records, search logs that can be used lawfully, and interviews with subject-matter experts. A document-processing pipeline may supply important source metadata; validate that pipeline independently if your corpus depends on intelligent document processing.
Do not create a dataset containing only clean questions with one obvious answer. Include at least these case families:
- **Answerable:** the approved corpus contains sufficient, current evidence.
- **Unanswerable:** no approved source supports an answer, so abstention is expected.
- **Ambiguous:** the query requires clarification about location, role, date, or meaning.
- **Conflicting:** two sources disagree and the system must prefer an authoritative version or expose the conflict.
- **Stale:** an older policy is retrievable but has been superseded.
- **Multilingual:** queries and sources cross the languages the product actually promises to support.
- **Permission-negative:** a relevant document exists, but the test identity is not authorized to retrieve it.
- **Adversarial:** the query or a retrieved document contains instructions intended to redirect the model.
Each case should store the query, test identity, expected and forbidden documents, essential answer points, acceptable variations, citation expectation, and required abstention or clarification. Preserve corpus, index, prompt, model, and application versions with every run so changed scores remain diagnosable.
Split the set into development and held-out acceptance portions. Add reviewed production failures to a regression set. Map coverage across users, repositories, permissions, languages, risk levels, and failure modes so blind spots stay visible.
Evaluate retrieval separately from generation
An incorrect final answer does not reveal whether search missed the right evidence or the model misused good evidence. Capture the retrieved document and passage identifiers before generation, then score retrieval on its own.
Choose retrieval metrics that fit the task
Use **Precision@K** when irrelevant passages in the context are a major risk or cost. Use **Recall@K** when missing any of several relevant sources could omit a critical condition. Use **MRR** when finding the first useful result early is important. Ranking measures such as NDCG can be helpful when relevance is graded rather than binary. These metrics require reviewed relevance labels; they should not be treated as interchangeable.
Inspect results by segment. An overall score can hide poor retrieval for a small department, minority language, new document type, or permission-constrained users. Compare keyword, vector, hybrid, filtering, and reranking changes against the same frozen set. Microsoft’s guidance on RAG evaluators likewise separates retrieval-process evaluation from final-response evaluation.
Diagnose misses before changing the model
For every failed query, label the likely cause: missing source, extraction error, bad chunk boundary, incomplete metadata, vocabulary mismatch, permission filter, query transformation, ranking, or overly small context budget. Fixing generation cannot recover evidence that was never retrieved. Conversely, increasing K without analysis can add noise, latency, and token cost.
Evaluate the generated answer against evidence
Once retrieval passes its gate, assess what the model does with the supplied context. Use a rubric with distinct dimensions:
- **Groundedness:** are claims supported by retrieved evidence?
- **Correctness:** does the answer match the approved interpretation of that evidence?
- **Completeness:** are required conditions, exceptions, and steps present?
- **Citation validity:** does each citation resolve to the claimed source and supporting passage?
- **Relevance and clarity:** does the response answer the user’s actual question without distracting detail?
- **Abstention and clarification:** does the system decline or ask a useful question when evidence is inadequate?
Groundedness is not the same as correctness. A response can faithfully repeat an obsolete or incorrectly selected document. Completeness is also different from verbosity: an answer may be short and complete if it contains every required point. Microsoft’s RAG language-model evaluation guidance emphasizes selecting metrics according to the workload rather than assuming one score fits every system.
Deterministic checks should verify citation URLs, document IDs, required disclosures, output structure, and prohibited strings. Humans should judge nuanced correctness. Calibrate model-based graders against expert decisions, version their prompts, and examine disagreements; no grader should become the sole authority for a high-risk gate. OpenAI Evals can organize datasets, criteria, graders, and repeatable runs, but acceptance design remains your responsibility.
Prove access control before retrieval
Permission filtering must happen before restricted content enters the model context. Hiding a citation or redacting the final answer is too late: sensitive text may already have influenced generation, appeared in logs, or entered a cache. If deployment constraints require a controlled environment, review the operational choices in this guide to private AI deployment, but treat authorization testing as a separate gate.
Build an identity matrix covering ordinary users, managers, temporary and departed staff, service accounts, cross-tenant identities, and multi-group users. Test public, direct-grant, group-grant, denied, inherited, revoked, and out-of-tenant documents. Results, snippets, answers, citations, memory, exports, logs, and caches must respect the same boundary.
Test permission changes as lifecycle events. Revoke access, refresh the index, and verify the document disappears within the stated window. The Azure AI Search document-level access-control guidance illustrates permission-aware retrieval and reinforces that authorization belongs at retrieval, not after generation.
Test prompt injection and poisoned knowledge
RAG introduces an indirect instruction channel: documents themselves can contain text that attempts to override system behavior. Add harmless attack fixtures to a segregated test corpus, including instructions to ignore policy, reveal secrets, follow an external link, change the requested task, or rank a document above authoritative sources. Include hidden or obfuscated variants that your ingestion pipeline could extract.
The system should treat retrieved text as evidence, not authority over its instructions. Verify that it does not disclose unauthorized context, execute unapproved actions, or conceal suspicious sources. Test direct and multi-turn injection too. OWASP’s prompt-injection guidance notes that RAG does not eliminate this vulnerability.
Poisoning tests should also cover compromised, duplicated, low-authority, and manipulated documents. Validate source allowlists, ownership, integrity signals, moderation or quarantine paths, and the ability to remove affected embeddings. OWASP’s vector and embedding risk guidance highlights data leakage, cross-context exposure, poisoning, and the need for permission-aware stores and monitored retrieval.
Verify freshness, deletion, and traceability
Production readiness requires a controlled content lifecycle. Test adding, updating, superseding, and deleting a source. Record when each change occurred, ingestion detected it, the index updated, caches expired, and the answer cited the new version.
Create conflict cases in which an old and new policy coexist. The system should follow a documented authority rule using fields such as source owner, effective date, status, jurisdiction, and version—not merely semantic similarity. A deletion test passes only when the removed material is absent from retrieval, generated answers, citation endpoints, caches, exports, and user-visible history according to the retention policy.
Every answer trace should support debugging without exposing unnecessary data. Record a request ID, identity class, query transformation, retrieved document and chunk IDs, source versions, prompt and model versions, filters, latency, token use, and disposition. Apply retention and access rules to traces.
Set operational gates for latency, cost, and availability
Measure the complete path under realistic concurrency: authentication, query processing, retrieval, reranking, model generation, citation assembly, guardrails, and rendering. Report distributions such as median and tail latency rather than a single average. Include cold starts, large permission groups, long conversations, degraded dependencies, rate limits, and index updates.
Set organization-defined budgets for latency, cost, availability, timeout rates, and capacity; there is no universal threshold. Test behavior when search, the model, identity service, or a citation source is unavailable. Safe degradation may mean retrying, returning sources without synthesis, declining, or routing to a person—never inventing an answer.
Run a practical HR policy acceptance scenario
Suppose an assistant answers employee questions about parental leave. The golden set includes permanent and temporary employees, two jurisdictions, an outdated handbook, a current policy, a manager-only exception document, and Persian and English queries.
For “What leave applies to me?”, the assistant should clarify role and jurisdiction. Retrieval should return the current authorized policy and exclude manager-only material. The answer should state the rule and material conditions, cite the current source, and avoid unsupported legal interpretation. If that policy is missing, it should abstain and name the approved channel.
Now revoke access to one policy, publish a superseding version, insert a document containing a hostile instruction, and repeat the test under load. The scenario passes only when authorization, freshness, citation, safe behavior, latency, and audit evidence all satisfy the acceptance contract. A fluent answer alone is insufficient.
Establish regression, review, and observability
Automate deterministic and repeatable evaluation in the delivery pipeline. Run a fast critical subset on every material change and the full suite before release. Treat changes to embeddings, chunking, metadata, retrieval settings, reranking, prompts, models, safety policies, and source connectors as candidates for regression testing.
For human review, define a rubric with examples. Double-review a sample, examine disagreement, and let a domain owner resolve consequential conflicts. Store the rationale, not just the score, and refresh the golden set as policies and failure patterns change.
In production, monitor empty retrievals, abstention, citation failures, denied access, latency, cost, freshness, feedback, and sampled quality. Alert on meaningful deviations by segment and turn incidents into reproducible evaluation cases, while privacy controls limit logging.
Use a documented go/no-go decision
The release review should produce evidence for every gate:
| Gate | Evidence required | Stop condition | |---|---|---| | Scope and risk | Approved use case, exclusions, owners, escalation | An unowned high-impact failure mode | | Retrieval | Segmented metrics and reviewed failure analysis | Critical evidence is consistently missed | | Answer quality | Groundedness, correctness, completeness, citations, abstention | Unsupported or materially incomplete high-risk answers | | Authorization | Identity and permission-negative test matrix | Any cross-role or cross-tenant disclosure | | Security | Direct, indirect, and poisoned-document tests | Unsafe instruction following or uncontained source | | Freshness | Add, update, supersede, delete, and cache tests | Stale or deleted evidence remains available beyond policy | | Operations | Load, failure, latency, cost, and availability results | Defined service budget or safe-degradation rule is unmet | | Governance | Owners, monitoring, incident response, rollback | No accountable operator or tested rollback path |
A conditional go requires a recorded residual risk, temporary control, owner, deadline, and rollback trigger. Never convert a failed security or permission gate into “monitor after launch.” Teams needing help with the system and evidence package can review EasySaz’s enterprise AI solutions.
Roll out gradually and keep evaluating
Production approval starts monitoring; it does not end evaluation. Begin with a limited cohort or shadow mode when appropriate. Restrict repositories and capabilities to the approved scope, make feedback easy, and assign incident review. Add new production query types to supervised tests after review.
Predefine rollback triggers for permission anomalies, source-integrity incidents, sustained regressions, budget breaches, and unsafe behavior. Keep a known-good configuration, verify that rollback includes index dependencies, and rerun critical cases after emergency changes.
Enterprise RAG evaluation is credible when every release claim can be traced to a test, a source version, an accountable owner, and a decision rule. Separate retrieval from generation, treat authorization and hostile content as first-class gates, test the knowledge lifecycle, and use organization-specific operating limits. If you want a production-readiness review tailored to your repositories, risks, and users, contact EasySaz to define an evidence-based acceptance plan.