CHECKLIST · 40 QUESTIONS · 10 CONTROL AREAS

The CIO AI control checklist.

Forty questions before agents go live — inventory, discovery, risk, identity, gateway policy, observability, evaluation, FinOps, containment and audit evidence. A yes without evidence counts as a no. Built for Gartner IT Symposium 2026 conversations; useful long after.

Score each question 0 (no / don’t know), 1 (partially, or yes without evidence) or 2 (yes, with evidence we could show an auditor). Under 55 means agents should not be acting on systems of record yet. The groups are ordered the way control is built: you cannot classify what you have not inventoried, and you cannot contain what you cannot see. Print the page — it is the PDF.

01AI inventory & ownership

  1. 1.Do we have one list of every AI system, model, agent and MCP server in production — or several partial ones?
  2. 2.Does every entry have a named business owner who knows they own it?
  3. 3.When did we last reconcile the inventory against what is actually running?
  4. 4.Can we produce the inventory as evidence — exportable, timestamped, with lifecycle states?

02Agent & MCP discovery

  1. 5.How would we learn about an agent a business unit shipped without telling us?
  2. 6.Which agent platforms — Microsoft, AWS, Google, Salesforce, ServiceNow — can we harvest agents from automatically?
  3. 7.Do we know which MCP servers our agents can reach, and who approved each one?
  4. 8.What is our process when a discovered agent matches no approved use case?

03Risk classification & approvals

  1. 9.Is every AI use case classified against the EU AI Act — or our internal risk tiers — before it ships?
  2. 10.Who approves a high-risk use case, and where is that decision recorded?
  3. 11.Do approvals expire or get re-reviewed when the model, prompt or tools change?
  4. 12.Can we show the classification reasoning, not just the label?

04Identity & tool permissions

  1. 13.Does every agent run under its own identity rather than a shared service account?
  2. 14.Are agent credentials scoped to least privilege for the specific task?
  3. 15.Are credentials issued short-lived per run, or do agents hold standing secrets?
  4. 16.Which agent actions require human approval before they execute?

05Gateway & runtime policy

  1. 17.Does our AI traffic cross one policy point, or does each app call providers directly?
  2. 18.Can we enforce one guardrail set — PII, injection, topic — across every provider and cloud?
  3. 19.Can we change a policy once and have it apply everywhere the same day?
  4. 20.What percentage of our AI traffic bypasses the gateway today — and do we actually know?

06Observability & incident diagnosis

  1. 21.Can we replay a full trace of any AI decision — retrieval, tool calls and policy decisions included?
  2. 22.When quality drops, can we tell whether the model, the retrieval or the prompt changed?
  3. 23.Do our traces stay inside our infrastructure, or do they transit a vendor's cloud?
  4. 24.Who gets paged when an AI system misbehaves, and with what evidence in hand?

07Evaluation & red teaming

  1. 25.Do releases gate on evaluation runs, or on demos?
  2. 26.Are production interactions sampled into online evaluations?
  3. 27.When did we last red-team our highest-risk assistant — and what changed as a result?
  4. 28.Does a failed test become an enforced control, or a ticket that ages?

08AI FinOps & budget controls

  1. 29.Can we attribute AI spend to team, application, use case and agent — including failed and retried calls?
  2. 30.Do budgets alert before the limit, and block past it?
  3. 31.Who is authorized to raise a cap, and is that decision logged?
  4. 32.Would a looping agent burn budget for hours, or for minutes?

09Containment, suspension & kill switch

  1. 33.Is there a runtime boundary our highest-autonomy agents cannot escape?
  2. 34.Can we stop a specific agent mid-run without taking down the platform around it?
  3. 35.When an agent is suspended, what happens to its credentials and in-flight actions?
  4. 36.Have we ever rehearsed the kill switch — or only documented it?

10Audit evidence, sovereignty & procurement

  1. 37.Can we produce EU AI Act, ISO/IEC 42001 or NIST AI RMF evidence from operations, or do we rebuild it for every audit?
  2. 38.Where does the AI control plane run — our infrastructure or the vendor's cloud?
  3. 39.Could we run our AI governance stack air-gapped if a regulator or contract required it?
  4. 40.In procurement, do we test enforcement claims live, or accept slideware?

Working the inventory questions first? The agent registry template (xlsx) and the 30-day inventory plan cover questions 1–8 in practice. The AI capability map shows which product categories answer which groups — and which two groups no point category sells.

Bring your scores to Barcelona.

Thirty minutes at Gartner IT Symposium 2026: your checklist, your architecture, and where the gaps close. Or email sales@kosmoy.com.

Book a demo

Or email sales@kosmoy.com.

Gartner IT Symposium/XpoGartner®ITSYMPOSIUMXpo™

Gartner and Gartner IT Symposium/Xpo are trademarks of Gartner, Inc. and/or its affiliates. Kosmoy is an exhibitor at the 2026 Barcelona conference. Gartner does not endorse Kosmoy or its products.