AI OBSERVABILITY · LAYER 2
See how every model and agent behaves in production.
Full traces across providers — retrieval, tool calls, agent hops, guardrail decisions, latency, cost, evaluation results and feedback — recorded in your own infrastructure, correlated in one place.
Or email sales@kosmoy.com.
An AI observability platform records what models and agents actually do in production, with enough context to explain why. Monitoring and observability are not synonyms — one watches thresholds, the other keeps evidence. Kosmoy provides both, from the same governed traffic: the AI Monitoring layer is the operational surface, and the trace substrate below it is what this page is about.
| Question | Monitoring | Observability |
|---|---|---|
| What it answers | Is something wrong? | Why is it wrong? |
| Unit of record | Metric against a threshold | Trace with full context |
| Knows in advance | What to watch | Nothing — evidence first, questions later |
| Typical output | Dashboard, alert | Root cause, reproducible case |
| Fails when | The failure is novel | The evidence was never captured |
Anatomy of a governed trace.
One request, end to end. Because the traffic crosses the gateway, the trace carries the policy decisions — not just the timings.
- Request received 40 msapp → gateway
- Input guardrails 80 msidentity · PII · injection
- Retrieval 260 msvector store
- Model call 770 msreasoning pass · 1,410 tok
- Tool call 270 msMCP server
- Model call 330 msfinal answer · 930 tok
- Output guardrails 80 mspolicy check
- Response logged 70 msledger row written
One trace: 2,340 tokens · 1.9 s total · guardrails: pass · cost attributed: app · team · user — and the finished trace is sampled into online evals.
Orange spans are policy decisions, blue spans are model and data work, red is the tool action. Illustrative timings.
Agents need trajectories, not logs.
A multi-step agent makes decisions between model calls: which tool, which arguments, whether to retry, when to stop. Kosmoy records the trajectory — every step with its model invocation, tool call, failure and cost — so a fifteen-step run reads as one story. Agent-to-agent (A2A) hops that cross the gateway keep identity and policy context attached across the chain.
Evaluation closes the loop.
Traces are the raw material of quality. Production traffic is sampled into online evaluations, scores accrue on real behaviour, and a failed eval links back to the exact trace that produced it. Offline evals gate releases against datasets; online evals watch what ships. Same evidence, both directions.
Four incidents, one evidence trail.
Quality incident
A user flags a wrong answer. The trace shows what was retrieved, which model ran, what the guardrails saw — and the eval score that should have caught it. Fix the retrieval, the prompt or the eval, with evidence.
Latency incident
P95 doubles overnight. The waterfall shows which span grew: a slow provider, a retry storm, a tool that stopped answering. You escalate to the right owner instead of guessing.
Policy incident
A guardrail blocked something it should not have — or passed something it should not have. The decision is on the trace with its inputs, so the rule is tuned against the actual case, not a reconstruction.
Cost incident
A team burns its monthly budget in a week. Attribution shows which app, which prompt change, which agent loop. The fix might be routing, a cap or a code change — the ledger says which.
Every provider, your boundary.
Because traffic crosses one gateway, the view is provider-neutral: OpenAI, Anthropic, Azure, Bedrock, Vertex and self-hosted models land in the same trace model, comparable side by side. And because traces embed prompts, retrieved documents and identities, Kosmoy keeps them where they belong — single-tenant in your own Kubernetes, private cloud, on-premises or air-gapped.
What Kosmoy does not replace.
General APM — Datadog, New Relic, Grafana and peers — owns infrastructure and application health, and should keep owning it. Kosmoy owns the semantic layer APM cannot see: what the model was asked, what the agent did, what the guardrail decided, what it cost. Alerts from Kosmoy hand off to the incident stack you already run.
Module questions, answered straight.
What is an AI observability platform?
An AI observability platform records what models and agents actually do in production — full traces with retrieval steps, tool calls, agent hops, guardrail decisions, latency and cost — and correlates them with evaluation results and user feedback. Where monitoring tells you something is wrong, observability gives you the evidence to explain why.
What is the difference between AI monitoring and AI observability?
Monitoring watches known metrics and thresholds: error rates, latency, spend against budget. It tells you that something is wrong. Observability keeps the underlying evidence — traces, context, evaluations — so you can work out why it is wrong and what to change. Kosmoy's AI Monitoring module is the operational surface (dashboards, alerts); observability is the trace and evaluation substrate underneath it.
What should an LLM trace contain?
Enough to replay the decision: the request and response, the model and parameters, retrieval steps with what was fetched, every tool call and agent hop, guardrail decisions on input and output, latency per span, token counts and cost, and any evaluation or feedback attached afterwards. A trace missing the policy decisions or the cost is a debugging tool; with them, it is audit evidence.
How do you observe a multi-step AI agent?
Trace the trajectory, not just the calls: each step's model invocation, the tool it chose, the arguments it passed, retries and failures, and the cost of every hop. Kosmoy records agent runs step by step — and because agent traffic crosses the gateway, the trace includes the policy decisions applied along the way. For agents in an Action Capsule, the runtime boundary is on the record too.
Can one platform combine LLM observability and evaluation?
Yes — and they are better together. Kosmoy samples production traces into online evaluations, so quality scores accrue on real traffic, and failed evals link back to the exact traces that produced them. Offline evals run pre-release against datasets; online evals watch production. One loop, shared evidence.
What is the best deployment model for regulated AI observability?
Traces contain prompts, retrieved documents and user context — often the most sensitive data an AI system touches. For regulated enterprises the safest model is observability that runs inside your own boundary. Kosmoy deploys single-tenant in your Kubernetes — private cloud, on-premises or air-gapped — so traces never leave your infrastructure.
Does AI observability replace Datadog, New Relic or another APM platform?
No. General APM owns infrastructure and application health: hosts, services, databases, uptime. AI observability owns the semantic layer APM cannot see — what the model was asked, what it retrieved, what the agent did, what the guardrail decided, what it cost. Run both: APM for the platform your apps run on, AI observability for the AI decisions inside them.
See a governed trace end to end.
Bring a production incident you never fully explained. We will show what the trace would have recorded — policy decisions included.
Or email sales@kosmoy.com.