An assistant may answer most product questions correctly while mishandling one request involving another customer's account. Averaging these outcomes into a single quality score hides the distinction between an imperfect answer and an authorization failure.
Security evaluation asks whether the application respects defined boundaries under specified conditions. Accuracy evaluation asks whether an answer matches the expected information. A release process can use both, with separate acceptance criteria and evidence.
Define the prohibited outcome
NIST's Generative AI Profile discusses risk management across the AI lifecycle, including measurement and evaluation. It provides a framework for organizing risks rather than a universal passing score for every product. The NIST profile supplies that broader context.
For an account assistant, define failures such as retrieving another customer’s records, making an unapproved change, or displaying a secret returned by a tool. Decide how to observe each one. Some appear in the answer; others require retrieval or service logs.
Build cases around business permissions
Use synthetic users with distinct roles and data. Include ordinary authorized tasks, unauthorized requests, ambiguous requests, and external content that attempts to redirect the workflow. The expected result can be a completed task, a request for missing information, or a service-side denial.
Keep the permission model fixed while changing the phrasing. This helps distinguish a brittle conversational response from an enforced control. A user asking indirectly for another account's record does not gain access merely because the request resembles a normal support question.
Record the complete configuration
An evaluation result belongs to a model version, system instructions, tool definitions, permissions, retrieval index, and application build. If any of those change, the result may no longer describe the deployed system. Store enough configuration detail to reproduce the run without putting credentials or customer data into the test package.
Also record repeated trials where behavior is variable. One successful refusal is an observation, not an estimate of a universal failure rate. Reporting the number of cases, number of trials, and observed outcomes makes the limits of the evaluation visible.
Separate severity from frequency
An authorization failure in a rarely exercised case can still require release attention. A high frequency of harmless formatting errors may instead affect usability. The product owner can decide on separate release conditions for these categories rather than allowing a large set of easy questions to dilute a significant security result.
For a scoped assessment, the deliverable can list tested boundaries, failed cases, service traces, and retest results. It can also state which tools, languages, document formats, or user roles were excluded. Those exclusions describe remaining coverage, not evidence that the untested areas are safe.
At the release meeting, review the tested permissions, failures, and retest results alongside the quality score. Identify the tools and roles the evaluation did not cover so the team knows where evidence is still missing.
Sources
OWASP: AI Agent Security. The account-assistant evaluation is an illustrative application of lifecycle risk assessment.