Microsoft connects risk discovery, repeatable evaluations, and a runtime policy in one workflow. The useful question is whether a chosen control changes a measured behavior—and whether the test actually covers the risk a team cares about.
Microsoft’s September 24 run-assert-eval announcement describes a way to test an AI agent before and after adding a runtime policy. In its billing-support demonstration, the company reported cross-customer data disclosure in 12 of 40 applicable baseline conversations, or 30.0%. After a policy change, it reported two violations in 34 applicable conversations, or 5.9%. Those figures make the workflow worth examining, but they are results from Microsoft’s own example, not an independently reproduced security benchmark or evidence of production performance. Microsoft’s announcement provides the example and its scope.
What the workflow actually does
The skill joins three pieces: Clarity can surface candidate agent risks, ASSERT turns selected risks into scenario-based evaluations, and Agent Control Specification (ACS) supplies a runtime policy that can be generated and validated for an agent with a suitable tool boundary. A team can also provide risks directly, so Clarity discovery is optional. The process is more deliberate than pressing “run”: a person reviews candidate risks and selects which ones to measure.
ASSERT gives each selected risk a separate, atomic behavior definition. The team runs a baseline, reviews the results, then can generate and validate an ACS policy. Before the governed run, the policy and its runtime wiring still need human review. The follow-up evaluation keeps the behavior definition, cases, and judge fixed while changing the policy. That design makes the before-and-after comparison more meaningful for the chosen tests; it does not make the tests a complete map of the agent’s possible failures. The ASSERT repository documents the workflow and integrations.
For a team assessing a proposed guardrail, that sequence suggests a practical reading of the result: “Did this policy change what happened in these scenarios?” It is a narrower and more actionable question than “Is this agent safe?” The first can be measured within a defined test; the second would require evidence beyond a selected suite and one run.
How to read Microsoft’s billing-agent example
The demonstration asks a billing-support agent to limit account access to ACME-1001. Microsoft reports the cross-customer disclosure counts above and says the governed sample had zero permissible-behavior violations. The latter matters: a policy that merely blocks everything would look good on a disclosure test while making the agent unusable. Here, the company reports both the targeted violation rate and whether legitimate behavior was disrupted in the tested sample.
The denominators differ—40 applicable baseline conversations and 34 applicable governed conversations. Readers should compare the stated rates as Microsoft’s demonstration results, not treat the count difference as a population-wide risk reduction. The claim applies to this example agent, selected risk, generated scenarios, model configuration, and judge. The announcement supplies no independent reproduction, production incident-rate result, or comparative cost and performance measurement in the reviewed sources.
Microsoft’s ASSERT concepts documentation also cautions that generated scenarios cannot prove a system safe or unsafe, model-based scores can be wrong, and runs can vary. A result should therefore trigger inspection of the underlying conversations and artifacts, especially cases that the judge marks as failures or passes unexpectedly. ASSERT is an investigation aid, not a compliance certificate.
The integration boundary that matters
ACS enforcement requires a callable agent whose risky tools can be wrapped. That boundary lets a runtime policy intervene when the agent tries to use a tool. A hosted prompt or model target can still be evaluated, but the repository says it lacks the callable tool boundary needed for ACS intervention. This distinction should come early in a team’s adoption decision: evaluation coverage does not automatically imply the ability to apply the same policy control.
The open-source ASSERT project is MIT-licensed, local-first, and framework-agnostic. Its README describes model endpoints through LiteLLM and agent integrations through OpenInference. It identifies the skill for Claude Code, GitHub Copilot, and Cursor. Basic ASSERT setup requires Python 3.11 or newer; using Clarity’s MCP discovery path requires Python 3.12 or newer and an MCP-capable IDE. Model-based inference and judgment also require a model or provider credential. Availability and setup therefore depend on the editor, provider, target integration, and whether the team uses Clarity.
“Local-first” also needs a precise reading. ASSERT writes inspectable JSON and JSONL artifacts locally, and telemetry is not sent to Microsoft by default. Configured model providers and trace services may still receive evaluation data under their own terms. Teams handling sensitive agent traces should decide what data can enter those integrations before running a suite.
What to check before running it on an agent
Start with a risk the team can state as one observable behavior. Decide whether the target exposes the callable tools ACS can wrap; if it does not, plan for evaluation without claiming a runtime policy intervention. Review the baseline cases and judged outputs before writing a policy, then review the generated policy and wiring before repeating the same suite. Keep the test definition and judge fixed for the comparison, and inspect both blocked risky actions and permitted work.
Use a controlled target. The project’s risk guidance warns that testing a live agent can cause real side effects. It recommends synthetic or test data, restricted credentials, isolated systems, and safeguards around logs, storage, and external actions. Model inference, judgment, and tool calls also have costs, even if the evaluation software is open source.
Finally, budget for operational uncertainty. The project’s support document calls the current distribution a customer-preview proof of concept and says it has no formal support SLA without a separate agreement. That does not negate the value of a repeatable baseline-versus-policy test. It does mean a team should treat successful evaluation as evidence about specified scenarios, inspect failures and side effects, and avoid presenting the output as a general safety guarantee.