How to evaluate an AI governance platform in one workflow
Use one realistic workflow to test discovery, policy, approval, failure behavior, and evidence. This run sheet works for any AI governance platform, including Halo.
By Halo Research Team
A useful AI governance evaluation starts with work your organization needs to govern. Pick one AI system, one accountable owner, and one consequential action. Then ask each platform to follow that example from discovery to reviewable evidence.
This post gives you a run sheet you can reuse across vendors. The same scenario, the same data, and the same questions make the results comparable.
Evaluate connected outcomes, coverage, and operating effort with the same synthetic workflow for each platform.
Define the scenario
Consider a support agent that reads an order, proposes a refund, and can send messages. Use synthetic customer data in a controlled environment. Write the permitted outcomes down before the demonstration starts.
| Item | Scenario card |
|---|---|
| AI system | Support agent for order questions |
| Owner | Support operations lead |
| Allowed | Look up an order |
| Needs approval | Issue a refund |
| Prohibited | Send customer details to an unapproved destination |
| Data | Synthetic customers and orders only |
Routine lookups proceed, the refund needs the right approval, and customer details must not leave through an unapproved destination. An invented order book gives each vendor the same footing.
Decide who should be in the room
Each role brings a different question to the same run.
- Security architect. Judges the boundary tests and the bypass paths. Security teams often start with AI policy enforcement.
- Governance or risk lead. Judges the decision record and the export. Governance teams can start with AI compliance evidence.
- Platform or IT owner. Judges setup effort and coverage. Platform teams can start with shadow AI discovery.
- Business owner of the agent. Judges whether routine work flows without friction.
Agree on the scoring before the run starts, so nobody adjusts the criteria after seeing the results.
The run sheet
1. Check what the platform can observe
Start with three questions.
- How did the platform find the agent?
- Which identity runs it?
- Which connections are active, and when did they last report?
Then ask the vendor to name a path outside coverage. An inventory entry should not imply that each action is monitored or controlled. Write down which devices, browsers, gateways, and agent environments the demonstration covered, and which it did not.
2. Configure a rule and find where it runs
Configure a rule for the scenario and ask where it executes. Is it checked before a tool call, inside the agent environment, or after an event arrives? List the components and licenses the demonstration required.
Record the setup work, so a polished demonstration does not hide implementation effort. Our post on where AI policy runs explains the difference between observing and enforcing.
3. Run the ordinary case first
The agent should complete its permitted lookup without unnecessary interruption. Unnecessary interruptions teach teams to work around a control, so the ordinary case deserves as much attention as the difficult one.
4. Test approval and a changed request
Request a refund, approve a specific amount, and then change that amount before execution. Observe whether the original approval still authorizes the changed request.
Our post on an agent's permission and the action decision explains why an approval should stay bound to the request that was reviewed.
5. Try the prohibited transfer
Try the prohibited transfer through each supported path in scope. A block, a warning, and a later alert are different results. Record which one occurred and whether the destination actually received anything.
6. Interrupt a dependency
Repeat an important case while a control dependency is unavailable, using the agreed test procedure. Record whether the action ran, waited, or failed. Our post on what happens when an AI control goes offline provides a procedure.
7. Finish with an export
A colleague who missed the demonstration should be able to identify the agent, the delegating actor, the policy version, the requested action, and the outcome. The record should also explain missing context or periods without telemetry.
Keep this artifact alongside the test instructions. See what an AI audit log should contain for the fields to look for.
Record outcomes with plain labels
Use a results table that the whole evaluation team can read.
| Case | What you expect | What to record |
|---|---|---|
| Order lookup | It proceeds | Allowed |
| Refund request | It waits for the right approver | Pending, then approved or rejected |
| Changed refund amount | It needs a new decision | Pending, or the observed behavior |
| Prohibited transfer | It does not reach the destination | Blocked, warned, or alerted, as observed |
| Dependency outage | It follows the failure rule you chose | Control error, plus the final state of the action |
Allowed, blocked, pending, and control error are different results. A platform that reports them as one number hides the information you need most.
Score outcomes, coverage, and effort
Score the result against your required outcome, your coverage, and the operating effort. Avoid awarding points because a vendor uses the same feature name as your checklist. An impressive capability outside your environment may contribute little to the decision.
- Outcome. Did each case end the way your policy says it should?
- Coverage. Which paths, devices, and tools were in scope, and which were left out?
- Operating effort. Which modules and licenses were required, how long did setup take, and who will maintain the rules?
Use the same scoring for each vendor, and ask each one to rerun any case that surprised you.
Watch for common evaluation pitfalls
A few patterns can make a demonstration look stronger than the product.
- The demonstration uses the vendor's sample data, so your scenario never runs.
- A coverage statement names no specific devices, browsers, or gateways.
- A block is shown on screen with no evidence that the destination received nothing.
- The export was assembled by hand and cannot be produced again on request.
Ask the vendor to repeat the run on your scenario, with your reviewers watching, and to produce the export on demand.
Keep the artifact
Store the scenario card, the run sheet, and the exported record together. A repeatable run lets you compare platforms, and it gives you a regression test when the platform or your agent changes later.
Our overview of what an AI governance platform is lists the four jobs a platform should connect, which is a useful checklist for the debrief.
If the platform includes an assistant that drafts assessments or recommends controls, add it to the run. Our post on who reviews the governance assistant lists what to check.
How Halo helps
Bring this workflow to a Halo walkthrough. Halo Browser and Halo Endpoint support discovery on supported surfaces, the policy engine runs in audit and enforce modes, the MCP and tool gateway applies human approvals to agent traffic, and audit and evidence logs export as CSV and JSON. Browser and endpoint rollout is guided, so setup effort is part of the conversation. You can also see how Halo compares with other approaches.