Skip to Content
Welcome to the Novantra documentation.
GuidesGovernanceAI GovernanceRuns & Evidence

Runs & Evidence

Every AI invocation in your workspace produces a run record: the AI system invoked, the profile version used, the authorization that permitted it, the inputs, the model output, the suggestions, the reviewer decisions, and any changes that were eventually applied. This is the audit trail of AI work, and it’s the basis for the AI oversight rollup that leadership and auditors examine.

Runs and their suggestions are also a primary source of evidence claims for the rest of the governance program. An AI-drafted summary that a reviewer signed off on becomes evidence for an obligation. An AI-classified document becomes evidence for a retention rule. The connection is explicit and auditable.

When you would reach for this

You consult Runs & Evidence when:

  • An auditor asks “show me every AI invocation that touched this control in the last quarter.”
  • A reviewer is investigating an unexpected applied change to a record.
  • The team is examining a Copilot or Document Intelligence run that produced surprising output.
  • Leadership wants the oversight rollup: how much AI activity, what proportion was reviewed, what was approved, what was rejected.
  • An AI provider incident triggers a forensic look at affected runs.

You don’t reach for this for live AI work. The live work happens in the Copilot, Document Intelligence, and other AI surfaces. Runs & Evidence is the historical record.

What lives in runs and evidence

Multiple record types:

AI run is one invocation of an AI system. It carries: the system, the profile version, the use authorization, the inputs, the model output, the policy in effect, the timestamp, the actor.

AI suggestion is one proposed action from a run. A single run may produce multiple suggestions (e.g., a document intelligence run extracting many fields). Each suggestion has a status: pending-review, accepted, modified, rejected, applied.

Suggestion application record captures when an accepted suggestion was actually applied to a target record (a document, a control, a finding). The application is the link from “AI proposed” to “workspace state changed.”

Evidence link captures the connection from a run or suggestion to an evidence claim. When AI work supports a governance claim, the link makes the relationship explicit.

AI oversight rollup is a periodic projection summarizing AI activity across systems: run count, suggestion count, acceptance rate, rejection rate, applied-change count, by system and by period.

A worked example: an aerospace contractor reviews AI activity for a regulator audit

An aerospace contractor uses Document Intelligence to process supplier quality documents and uses Copilot to assist in regulator correspondence drafting. A regulator audit examining the contractor’s AI governance asks for evidence: what AI was used, when, what it produced, what humans did with the output. The director of quality, Ezekiel, walks the audit through Runs & Evidence like this.

Step 1: scope the period. The audit covers the previous twelve months. Ezekiel filters Runs by date range and by AI system: Document Intelligence runs against supplier quality documents, Copilot runs in the regulator correspondence drafting profile.

Step 2: walk the Document Intelligence runs. For each run:

  • The document the run processed.
  • The profile version that was in effect.
  • The extracted fields proposed (the suggestions).
  • The reviewer who handled each suggestion.
  • The disposition: accepted (and the value applied), modified (with the reviewer’s correction shown beside the model’s proposal), rejected (with reviewer note), pending (none expected in a closed period).

The audit can see that every extracted field reached the document record only after a human accepted or modified the proposal. The audit can also see the rejection rate: a small percentage of model proposals were corrected, demonstrating that reviewers exercise judgment.

Step 3: walk the Copilot runs. For each Copilot run in the regulator correspondence profile:

  • The drafting brief that prompted the invocation.
  • The profile version (and its prompts, retrieval, action policy).
  • The draft Copilot produced.
  • The reviewer who edited the draft.
  • The edited version that became the final.

The audit sees that every Copilot output was a draft; finals always had human editing; the action policy on the profile prevented any autonomous action.

Step 4: review the oversight rollup. For the twelve months:

  • 2,841 Document Intelligence runs on supplier quality documents.
  • 14,205 suggestions produced.
  • 13,610 (96%) accepted.
  • 481 (3.4%) modified.
  • 114 (0.8%) rejected.
  • 0 autonomous applications (per policy).

For Copilot:

  • 312 invocations across the regulator correspondence profile.
  • 312 drafts produced; all reviewed and edited before finalization.
  • 0 autonomous actions.

Step 5: evidence linking. The audit also examines how AI work fed evidence claims. Ezekiel shows the audit several evidence claims (e.g., the supplier quality data extraction process produces evidence supporting an obligation around supplier conformance documentation) and the explicit links from runs/suggestions to those claims.

After the audit, the regulator’s response is favorable: the AI activity is documented, reviewable, and proportionate to the risk.

What you’ll see in the product

Runs & Evidence lives under Governance → AI Governance → Runs with the oversight rollup as a top-level dashboard.

The Runs list shows every run with filters by AI system, profile, date, actor, status.

Inside a run, you see:

  • The system, profile version, authorization, actor, timestamp.
  • The inputs (with privacy controls — sensitive inputs may be redacted in the UI).
  • The outputs.
  • The suggestions (linked).
  • The activity history.

The Suggestions list is a cross-run view of all suggestions, useful for reviewer workflows.

Inside a suggestion, you see the source run, the proposal, the reviewer disposition, the applied target (if applied).

The Oversight Rollup is the leadership view: periodic projections of AI activity across systems.

Every record is captured in the workspace Audit Log.

Common ways AI work becomes evidence:

  • A document classification run on a regulated document → evidence claim that the document is classified per policy.
  • A Copilot draft of an attestation → evidence claim that the attestation was prepared (with human sign-off captured separately).
  • A Document Intelligence extraction of contract obligations → evidence claim that the obligation register was populated from the contract.

The link is created explicitly when the evidence is recorded; not every AI run is evidence, only those that support a defined claim.

What you’ll not find here

  • The model output values themselves are stored in the run record, but for sensitive workloads the in-product UI may redact them for general view. Audit users see them in full.
  • The provider’s own logs are separate. Novantra captures what was sent to the provider, what was received, and what was done with it. The provider’s internal infrastructure logs are theirs.

Common workflows

Auditing AI activity

  1. Filter Runs by date range, system, profile, actor.
  2. Walk the runs and their suggestions.
  3. Cross-reference applied changes to the affected records.
  4. Pull the oversight rollup for the same period for the headline view.

Investigating an unexpected change

  1. From the affected record, find the linked AI suggestion application.
  2. From the suggestion, navigate to the run.
  3. From the run, see the inputs, the profile version, the authorization, the reviewer.
  4. Determine whether the change was per-policy (suggest+human-accept) or anomalous.

Linking evidence

  1. From an evidence claim, link to the relevant AI run or suggestion.
  2. The link appears on both sides for traceability.
Last updated on
Action PoliciesOverview