When should an AI system be reviewed again?
A review records a decision under particular conditions. Learn which changes should reopen it, how to scope the work, and how to keep both decisions on the record.
By Halo Research Team
An AI review records a decision about a system under particular conditions. Those conditions can change. A new tool, broader data access, or a different business purpose can make the original approval incomplete even when the agent keeps the same name.
This post covers how to decide which changes should reopen a review, how to scope the work, and how to keep the old and new decisions on the record.
A document assistant gains a tool
Imagine a document assistant approved to search an internal knowledge base. Its owner later adds a tool that sends summaries to external recipients.
The original assessment may still explain the retrieval risks well. The new destination and the authority to send need their own decision.
Reopen a review when a change can invalidate the decision that was made earlier. Keep the scope proportionate to the change.
Define the changes that matter
Possible triggers include a new model, tool, data source, owner, purpose, permission, or autonomy level. A significant incident or a failed evaluation can also justify another review. The right list depends on the system and the decision it needs to protect.
| Change | Why it can invalidate the earlier decision | Typical scope |
|---|---|---|
| New model | Behavior, quality, and data handling can differ | Evaluation cases and data-handling terms |
| New tool or connection | The agent can take actions the review never considered | Tool authorization, destinations, approvals |
| New data source | Sensitivity and access rules can differ | Data classification and access |
| Broader permission | The set of reachable actions grows | Authorization scope and approval thresholds |
| New owner | Accountability may have shifted | Owner confirmation and handover |
| New purpose or user group | The risk context changes | Use-case classification and reviewer |
| Incident or failed evaluation | The earlier assumptions did not hold | Root cause and affected controls |
These scopes are examples. Your own table should reflect the controls your organization uses.
Make the trigger specific
"Review whenever anything changes" creates work that nobody can finish. Identify which fields, events, or thresholds warrant attention and which updates can be recorded without reopening approval.
Detection depends on a current inventory. Our post on inventory coverage and freshness explains how to know when a source has stopped reporting.
Keep a periodic review as a backstop when important changes cannot be detected automatically. A calendar date catches the changes that detection missed.
Use relationships to narrow the work
If several agents depend on the same tool, a material change to that tool may affect each of them. Show those connections to reviewers. Avoid assuming that each dependent system carries the same risk or needs the same decision.
A read-only reporting agent and a customer-facing agent may use the same tool in very different ways. Review each at the level its use warrants.
Create a review task someone can act on
A review task should carry enough context to act. Include these items.
- What changed. The field, event, or version difference that triggered the task.
- The earlier decision. The approval, its conditions, and its date.
- Affected controls. The policies, approvals, and limits that relate to the change.
- Available evidence. Logs, test results, and configuration records.
- Responsible owner. A named person who can decide.
Avoid repeatedly opening identical tasks for the same unresolved event. Make missed triggers and failed automation visible to the team running the process.
Decide the operating state while review is pending
Some changes can proceed with monitoring, and others may need a temporary restriction or an approval before deployment. Record the rationale, any exception, and its expiry. This is a governance choice, and it should not hide inside automation defaults.
Here is a hypothetical trigger rule for the document assistant.
- Trigger. A tool with an external destination is attached to an agent approved as read-only.
- Scope. Tool authorization, the destination list, the approval threshold, and an evidence refresh.
- Owner. The application owner, with the governance lead as reviewer.
- Operating state. The tool runs in observe mode, and external sends wait for approval until the review closes.
Preserve both decisions
Keep the evidence and configuration behind the original approval, then link the new review and its outcome. Replacing the old record with today's answers makes it harder to explain what was authorized when an earlier action occurred.
Questions about a past action, such as why an agent was allowed to send a summary last quarter, need the decision that applied last quarter. Our post on what an AI audit log should contain explains how to keep version references in the record.
Where frameworks and law touch on change
External frameworks and laws point in the same direction. NIST AI RMF 1.0 describes risk management as continuing work across its Govern, Map, Measure, and Manage functions. ISO/IEC 42001 is a management system standard, and management systems run through planned, repeated cycles.
The EU AI Act uses the idea of a substantial modification for high-risk AI systems, which can call for a new conformity assessment. Ask your legal team what counts as a substantial modification for your systems, and confirm the current official timeline for the Act because amendments have been proposed.
Our post on the EU AI Act timeline for governance teams walks through the dates. The glossary entries for the NIST AI RMF, ISO/IEC 42001, and the EU AI Act give more background.
How Halo helps
Halo's AI inventory classifies assets as sanctioned, monitored, blocked, shadow, or unknown and records the source and confidence behind each one. Configuration changes, approvals, and decisions are recorded in audit and evidence logs that export as CSV and JSON, policy bundles are versioned, and human approval gates cover higher-impact actions. Halo records evidence that supports governance programs aligned to the EU AI Act, NIST AI RMF, and ISO/IEC 42001, and it does not automate compliance mapping or make anyone compliant. Bring one realistic change to a walkthrough and ask to see what the platform records about it, or read about AI compliance evidence.