Best LLM Observability & AI FinOps Platforms in 2026
Ten platforms that watch — and in some cases control — what your LLMs and agents do and spend. Organized by buyer type: evals-first teams, APM estates, OSS self-hosters, and the regulated buyers for whom cost tracking must double as audit evidence.
LLM observability grew up in 2026. What began as request logging is now three overlapping jobs: tracing and debugging agents, evaluating quality systematically, and controlling token spend — the FinOps half that CFOs discovered when the first seven-figure model bill arrived. No single product does all three best, and the vendors know it: this year's consolidation wave (ClickHouse acquiring Langfuse, CoreWeave absorbing Weights & Biases, Mintlify taking Helicone into maintenance mode, Palo Alto Networks buying Portkey) redrew the category mid-flight.
This guide compares ten platforms by buyer situation rather than pretending a single 1-to-N ranking exists. One disclosure up front: Kosmoy publishes this page. We score it on the same rubric as everyone else, it loses the tracing and evals axes to the specialists — plainly and by design — and it wins exactly one buyer segment: the regulated enterprise whose cost tracking must double as audit evidence.
What counts as LLM observability & AI FinOps platforms in 2026
What counts as LLM observability and AI FinOps in 2026? At minimum: tracing of LLM and agent activity (prompts, tool calls, retrieval steps), token-level cost tracking with attribution to teams and applications, and some way to act on quality — evaluators, monitors, alerts. The market splits into two architectural families. SDK-instrumented platforms (LangSmith, Langfuse, Arize, W&B Weave, Braintrust, Opik, Datadog, New Relic) watch from beside the application: rich traces, no enforcement. Data-path platforms (Portkey, Kosmoy) sit in front of the model: their cost numbers are also budgets that can block, and their logs are also policy records.
Both APM giants reframed the category this year — Datadog increasingly presents its product as agent observability, and New Relic began rebranding AI Monitoring toward AI Observability in June 2026 — a signal that agents, not chat apps, now drive the requirements. Adjacent tools worth knowing but not profiled here: MLflow's Apache-2.0 tracing and evaluation, Galileo's evaluation platform, and Helicone — a fine logging proxy now in maintenance mode since the March 2026 Mintlify acquisition, reasonable to keep running, hard to recommend adopting fresh.
How we scored the field
Every product is scored 0–10 on the same ten capability axes. A 10 is reserved for categorical architectural facts; specialists are expected to outscore platforms on their own spoke, and the scores show it.
Observability & FinOps
Trace depth and framework coverage, OpenTelemetry support, token/cost tracking granularity, dashboards, monitors and alerts on production traffic.
Testing, Evals & Red-teaming
Datasets and experiments, LLM-as-judge and code evaluators, human annotation, online evaluation of production, CI/CD integration and red-teaming where documented.
Gateway & Policy Control
Whether the product sits in the request path with real controls — routing, budgets that block rather than alert, key management, traffic-level RBAC.
Guardrails & Runtime Safety
Runtime detection and blocking of PII, toxicity and prompt injection — scored lower when enforcement must be wired per-application in code.
Compliance & Audit
Certifications, audit logs and retention controls; top scores reserved for AI-regulation evidence tooling (EU AI Act, ISO/IEC 42001, NIST AI RMF), which most of this category does not attempt.
Deployment Sovereignty
Self-hosting terms and license gates, VPC and dedicated options, air-gap feasibility, and whether a vendor control plane remains in the loop.
The field, scored
| Capability (0–10) | LangSmith (LangChain) | Langfuse | Arize AI | Datadog LLM Observability | Weights & Biases Weave | Braintrust | Opik (Comet) | Portkey | Kosmoy | New Relic AI Monitoring |
|---|---|---|---|---|---|---|---|---|---|---|
| AI Inventory & Discovery | 2 | 0 | 1 | 4 | 1 | 1 | 2 | 5 | 9 | 3 |
| Security & Shadow AI | 3 | 1 | 3 | 7 | 2 | 1 | 2 | 4 | 8 | 2 |
| Observability & FinOps | 9 | 9 | 9 | 9 | 8 | 8 | 8 | 9 | 7 | 8 |
| Gateway & Policy Control | 5 | 0 | 1 | 3 | 1 | 3 | 1 | 9 | 8 | 0 |
| Guardrails & Runtime Safety | 4 | 1 | 6 | 7 | 5 | 1 | 5 | 8 | 8 | 1 |
| Agent Containment | 7 | 0 | 1 | 3 | 1 | 0 | 0 | 4 | 9 | 0 |
| Compliance & Audit | 4 | 3 | 3 | 3 | 3 | 3 | 2 | 5 | 9 | 3 |
| Testing, Evals & Red-teaming | 9 | 8 | 9 | 6 | 8 | 9 | 8 | 3 | 4 | 2 |
| Agent Building | 9 | 1 | 2 | 4 | 1 | 1 | 2 | 2 | 6 | 3 |
| Deployment Sovereignty | 9 | 9 | 8 | 2 | 7 | 7 | 8 | 9 | 10 | 2 |
Bold marks the highest score on each row. 10 is reserved for categorical architectural facts; specialists are expected to outscore platforms on their own spoke.
Capability shape, vendor by vendor
Each panel shows one vendor across the same ten axes. Read it as area: a specialist climbs on its own spoke and falls away on the rest; a platform holds the frontier. The dashed outline is Kosmoy for reference.
The vendors, by buyer type
No single 1-to-N ranking survives contact with a real shortlist — the right pick depends on who is buying. Each vendor below is labeled with the buyer it fits best.
LangSmith (LangChain)
LLM observability, evals & agent engineering platformBest for evals-first teams (and the broadest platform)
LangSmith is LangChain's commercial platform for agent engineering — tracing, evaluation, prompt management, agent deployment, sandboxes and a no-code agent builder, plus an LLM gateway in private beta — layered on the MIT-licensed LangChain and LangGraph frameworks.
LangSmith is the category's reference point: framework-agnostic tracing with native OpenTelemetry ingestion, automatic cost tracking, and an evaluation suite — datasets with splits, pairwise experiments, multi-turn evaluators, annotation queues — that no rival fully matches. Its 2026 releases (Deployment, Fleet, Sandboxes GA, Engine, an LLM gateway in private beta) make it an agent-engineering platform rather than a tool (Interrupt 2026).
The caveats are procurement-shaped: the platform is proprietary, self-hosting is Enterprise-gated with license-beacon egress outside air-gapped licenses, and there is no AI-regulation compliance tooling documented as of July 15, 2026.
Strengths
- The deepest ecosystem gravity in the category: LangChain (~141.8k stars) and LangGraph (~37.3k stars) are MIT frameworks feeding the commercial platform, backed by a $125M Series B at a $1.25B valuation (October 2025).
- Category-leading evaluation tooling: datasets with splits, experiments and pairwise comparison, LLM-as-judge, code and composite evaluators, online and multi-turn thread evaluators, and annotation queues with rubrics (evaluation docs).
- Framework-agnostic observability with native OpenTelemetry ingestion, automatic token/cost tracking with per-model pricing, dashboards and alerts (observability docs).
Limits
- The LLM Gateway is private beta (waitlist) with a narrow policy surface — spend limits plus PII/secrets redaction across 7 providers; no routing, failover or fine-grained content policies documented as of July 15, 2026.
- No org-wide AI inventory or shadow-AI discovery — visibility covers applications instrumented with LangSmith or routed through its gateway.
- No EU AI Act, ISO/IEC 42001 or NIST AI RMF governance tooling documented as of July 15, 2026; the compliance story is security certifications (SOC 2 Type II, ISO 27001, HIPAA, GDPR) plus audit logs.
Langfuse
Open-source LLM engineering platform (tracing, evals, prompts)Best open-source self-hosted observability
Langfuse is an open-source (MIT-core) LLM engineering platform for tracing, evaluation and prompt management, self-hostable or on Langfuse Cloud, acquired by ClickHouse in January 2026 with public commitments to keep the license, roadmap and self-hosting unchanged.
Langfuse pairs an MIT core (~31k GitHub stars) with the rare promise that the self-hosted build is the exact same codebase as its cloud, unlimited and documented to run without internet access. Evals cover judges, code evaluators, Experiments with CI/CD gates and annotation queues; Monitors & Alerts arrived in June 2026. ClickHouse acquired the company in January 2026 with public continuity commitments (announcement).
Know the open-core boundary: audit logs, retention policies and project RBAC need an Enterprise key when self-hosting, and EE telemetry cannot be disabled. It observes; it never enforces.
Strengths
- One of the most widely adopted open-source LLM engineering platforms: MIT core with ~31.2k GitHub stars and daily active development as of July 2026.
- Self-hosting without asterisks on the core: the self-hosted build runs the exact same codebase as Langfuse Cloud with all core features and APIs unlimited, and the networking docs state it does not require internet access.
- Deep evaluation tooling: LLM-as-a-judge, code evaluators, datasets, first-class Experiments with CI/CD quality gates in GitHub Actions (May 2026), and human annotation queues (changelog).
Limits
- Observe-only: no gateway, runtime policy enforcement or guardrail blocking — Langfuse's own docs delegate runtime security to third-party libraries and position the product as ex-post evaluation.
- No org-wide AI inventory, shadow-AI discovery or agent containment documented as of July 15, 2026.
- Key governance features are Enterprise-licensed when self-hosting (audit logs, retention policies, project-level RBAC, SCIM, server-side masking), and EE license telemetry cannot be disabled.
Arize AI
AI observability & evaluation platform (Arize AX + Phoenix OSS)Best for enterprise eval rigor with an OSS on-ramp
Arize AI pairs the enterprise Arize AX platform (agent observability, evaluation and runtime guards, SaaS or self-hosted) with Phoenix, one of the most active open-source AI observability projects.
Arize AX treats evaluation as a governed discipline — the Evaluator Hub versions LLM-as-a-judge evaluators at commit level — and adds Signal continuous trace review, voice-agent replay and runtime Guards that can actually block or regenerate responses. Phoenix (ELv2, ~10.6k stars) is the open-source on-ramp, and AX Enterprise self-hosts Kubernetes-first — unusual among SaaS observability peers.
Gaps: no gateway, no inventory, no compliance tooling, and Phoenix's Elastic License is not OSI-approved open source.
Strengths
- Dual offering few rivals match: enterprise Arize AX plus the open-source Phoenix project (ELv2, ~10.6k stars, 749 releases), still shipping weekly as of July 2026.
- Deep evaluation stack: an Evaluator Hub with commit-level versioning of LLM-as-a-judge evaluators, datasets and experiments, and online evals on production traffic (Observe 2026 launches).
- Documented runtime Guards — embedding-based and RAG LLM guards on inputs and outputs with block, default-response or regeneration actions (guardrails docs) — rare among observability specialists.
Limits
- No org-wide AI inventory or shadow-AI discovery documented as of July 15, 2026 — projects and spaces exist only inside the platform.
- No LLM gateway or central runtime policy point: guards intercept calls at SDK level inside the application, not at a traffic chokepoint with routing and RBAC.
- No agent containment — sandboxing, kill switch or scoped credentials are not documented as of July 15, 2026.
Datadog LLM Observability
LLM/agent observability within a general-purpose APM platformBest for teams already on Datadog
Datadog LLM Observability — increasingly framed by Datadog as agent observability — adds LLM/agent tracing, token-cost tracking and evaluations to the Datadog platform, with AI Guard supplying runtime AI security.
If Datadog already runs your monitoring, its LLM Observability is the lowest-friction answer in this guide: LLM and agent traces correlated with APM, infrastructure and RUM, cost dashboards, LLM Experiments for evals, and — uniquely among the SDK-instrumented family — AI Guard blocking prompt injection, tool misuse and data exfiltration at runtime. The AI Agents Console (Preview) even tracks coding-agent spend across the organization.
It is SaaS-only — telemetry must flow to Datadog — and standalone list pricing is opaque: the official page publishes no per-span rates as of July 15, 2026.
Strengths
- LLM and agent telemetry correlated with full-stack APM, infrastructure, RUM and security data in one platform (product page) — a correlation story no observability specialist can match.
- AI Guard adds genuine runtime blocking — prompt-injection and jailbreak protection, tool-call protection and sensitive-data exfiltration prevention — rare among observability vendors (AI Guard docs).
- The AI Agents Console (Preview) tracks org-wide usage, spend and impact of third-party coding agents such as Claude Code, Cursor and GitHub Copilot (docs).
Limits
- No self-hosted or customer-VPC deployment: telemetry must flow to Datadog's SaaS, which rules it out for strict sovereignty or air-gapped requirements.
- No org-wide AI system registry or shadow-AI discovery beyond coding-agent usage monitoring, and the AI Agents Console is still in Preview as of July 15, 2026.
- Standalone LLM Observability list pricing is opaque: the official page publishes no per-span rates, and third-party sources report conflicting figures and a May 2026 repricing.
Weights & Biases Weave
LLM/agent tracing & evaluation within the W&B platformBest for ML platform teams on W&B
W&B Weave is Weights & Biases' toolkit for tracing, evaluating and monitoring LLM and agent applications — agent-first trace semantics on top of the W&B ML platform, owned by CoreWeave since May 2025.
W&B Weave extends the experiment-tracking platform ML teams already trust into LLM territory, with agent-first trace semantics (sessions, turns, steps, tools), datasets-plus-scorers evaluation, inline safety scorers and Online Evaluations for production agents. For organizations training models and shipping agents, one platform covering both is a real consolidation.
Weave observes rather than enforces, guardrails are wired per-application in code, and CoreWeave's ownership (May 2025) orients the roadmap toward its AI cloud — self-managed parity and air gap remain undocumented as of July 15, 2026.
Strengths
- An agent-first trace model — sessions, turns, steps, tools and sub-agents as first-class concepts — rather than generic code tracing (Weave production-agent announcement).
- Evaluation lineage from the W&B experiment-tracking platform widely used in ML research: datasets plus scorers, side-by-side comparisons and human feedback (evaluations page).
- Pre-built safety and quality scorers — toxicity, bias, PII, hallucination, prompt injection — usable inline as guardrails (guardrails docs).
Limits
- Owned by CoreWeave since May 2025: the roadmap increasingly orients toward CoreWeave's AI cloud (W&B Inference, Mission Control), a consideration for buyers standardized on other clouds.
- Guardrail enforcement is developer-wired in application code via the SDK — there is no central policy point independent of each application.
- No org-wide AI inventory, gateway or shadow-AI discovery documented as of July 15, 2026.
Braintrust
Evals-first LLM engineering & observability platformBest for product teams where evals are the dev loop
Braintrust is an evaluation-centric platform for building AI products — eval suites, production trace logging on the purpose-built Brainstore, playgrounds and the Loop AI assistant — with a hybrid self-hosted data plane for enterprises.
Braintrust builds everything around the eval: an Eval() SDK, autoevals, human review, Brainstore for querying million-token agent traces, Topics for clustering production failure patterns, and Loop turning those failures into new datasets and scorers from natural language. An $80M Series B (February 2026) signals staying power, and the Enterprise hybrid keeps all trace data in your VPC.
The control plane always runs at Braintrust — no full self-host or air gap — and there are no guardrails, inventory or compliance tooling; scorers observe rather than block.
Strengths
- A deep evals-first workflow — Eval() SDK, autoevals, LLM-as-a-judge and code scorers, datasets, experiments and human review — adopted widely among AI-native product teams (Series B blog).
- Brainstore, a store purpose-built for querying millions of large nested agent traces, with online scoring and Topics pattern clustering across production runs (June 2026).
- A hybrid architecture that keeps all trace, eval and dataset content in the customer's environment while retaining a managed UI (architecture docs).
Limits
- No runtime guardrails or policy gateway: the AI Proxy unifies access but does not enforce safety or policy on traffic, and scorers run asynchronously rather than blocking.
- No org-wide AI inventory, shadow-AI discovery or compliance-framework tooling documented as of July 15, 2026.
- Self-hosting is data-plane only and gated to Enterprise; the control plane always runs at Braintrust, so there is no full on-prem or air-gapped deployment.
Opik (Comet)
Open-source LLM evaluation & observability platformBest budget open-source option
Opik is Comet's Apache-2.0 open-source platform for tracing, evaluating, monitoring and optimizing LLM applications and agents — self-hosted with no feature restrictions, or as Comet-managed cloud.
Opik is the value pick: Apache-2.0 with ~20.6k stars, unlimited spans and no feature restrictions when self-hosted, plus judges, datasets, online evaluation rules, an Agent Optimizer and beta guardrails (PII filtering, topic moderation) in one toolkit. The managed Pro tier at $39/mo with unlimited team members undercuts nearly every commercial rival, and June 2026's Cost Intelligence adds coding-agent spend visibility.
Guardrails remain beta with no prompt-injection blocking documented, and Comet's ~$70M funding is modest against the category's 2025-26 raises.
Strengths
- One of the largest open-source LLM-eval projects: Apache-2.0, ~20.6k stars, 524 releases with daily-cadence development as of July 2026.
- Self-hosted OSS carries unlimited spans and no feature restrictions — a genuinely low-cost sovereign option (Opik FAQ).
- Evals, tracing, guardrails and agent optimization in a single toolkit: LLM-as-a-judge metrics, datasets and experiments, online evaluation rules, and the Opik Agent Optimizer.
Limits
- No org-wide AI inventory, gateway or shadow-AI discovery documented as of July 15, 2026.
- Guardrails are still in beta and cover PII and topic moderation only — no prompt-injection blocking documented as of July 15, 2026.
- No agent containment or AI compliance-framework tooling documented as of July 15, 2026.
Portkey
AI gateway & LLM-ops control planeBest gateway-native FinOps
Portkey is an AI gateway and control plane for production AI — one API to 1,600+ models across 45+ providers, with observability, guardrails, prompt management and MCP/agent access control — acquired by Palo Alto Networks in May 2026.
Portkey earns its slot in an observability guide from inside the data path: because it is the gateway, its 21+ request metrics, per-key budgets and rate limits do not just report spend — they stop it, with hard enforcement per key and workspace and OpenTelemetry export for the rest of your stack (observability docs). For platform teams, that is FinOps with teeth.
Evals are thin (a cookbook, not a product), and the Palo Alto Networks acquisition (May 2026) leaves the public roadmap quiet — the docs changelog stops at April 2026.
Strengths
- Category-leading gateway breadth: one OpenAI-compatible API to 250+ LLMs and 1,600+ models across 45+ providers, with retries, fallbacks, load balancing and caching (gateway repo, MIT, ~12.4k stars).
- Deep LLM observability and FinOps: full request logging, 21+ metrics, per-key budgets and rate limits, OpenTelemetry and data-lake export (observability docs).
- A sovereignty ladder rare among gateways: open-source self-host, hybrid VPC data plane, and a documented fully air-gapped enterprise deployment (self-hosting docs).
Limits
- Does not document EU AI Act, ISO/IEC 42001 or NIST AI RMF evidence generation or AI risk classification as of July 15, 2026 (checked the enterprise-offering and feature-comparison docs).
- Inventory and governance cover only assets routed through the gateway — no discovery of AI systems or agents outside it.
- No first-class evals or red-teaming product, and no agent sandboxing or kill switch documented.
Kosmoy
AI management platformBest for regulated FinOps + audit buyers
A self-hosted control plane for enterprise AI: one inventory, one policy gateway, one audit trail and a containment sandbox for every model, agent and MCP server a company runs.
Kosmoy is the governance entry on this list, and the trade is explicit: it concedes tracing and evaluation depth to LangSmith, Langfuse, Arize and Braintrust — its evals score is a 4, and there is no dataset/judge/experiment tooling at all. What it does that none of them do: cost tracking and budgets enforced at a self-hosted gateway across LLM, MCP and A2A traffic, attribution per team, project, app and model in the Insights Dashboard, and the same event log feeding EU AI Act, ISO/IEC 42001 (aligned) and NIST AI RMF evidence bundles.
It runs single-tenant in your own Kubernetes, air-gapped if needed, and it is in production at Italy's central bank and banking regulator and at Europe's largest defence and aerospace group. If your FinOps numbers must survive an audit, this is the segment Kosmoy wins; every other segment on this page belongs to a specialist.
Strengths
- Four registries — AI systems, models, MCP servers and a master agent registry that pulls agents from Azure AI Foundry, Bedrock, Vertex, Salesforce and ServiceNow into one list.
- One OpenAI-compatible gateway enforcing guardrails, RBAC, budgets and logging on every LLM, MCP and A2A call.
- Action Capsule: kernel-enforced sandboxing for agents, MCP servers and private models, with per-task credentials and a kill switch.
Limits
- No dedicated evaluation or red-teaming suite — teams pair Kosmoy with a specialist evals tool.
- The agent builder covers governed internal use cases; dedicated agent-development platforms go deeper.
- No free or self-service tier — procurement runs through an enterprise sales process.
New Relic AI Monitoring
APM-integrated AI/LLM monitoring platformBest for New Relic APM estates
New Relic AI Monitoring — rebranding toward AI Observability during 2026 — extends New Relic's APM agents to trace LLM calls and AI agents, with token/cost tracking, response-quality feedback and model comparison inside the New Relic platform.
New Relic's answer mirrors Datadog's logic one estate over: LLM traces through the APM agents you already deploy, token cost allocation, model comparison, multi-agent visualization in preview, and a 2026 roadmap (SRE Agent, Agentic Platform with MCP support, the June 2026 shift toward 'AI Observability' branding) that keeps pace. The free ingest allowance makes trying it effectively free for existing customers.
It is the thinnest on evaluation in this guide — no datasets, experiments or judge evaluators documented as of July 15, 2026 — SaaS-only, and its agentic features remain largely in preview.
Strengths
- AI telemetry lands in the same platform as APM, infrastructure and logs — zero extra vendors for existing New Relic estates (AI monitoring docs).
- Instrumentation reuses existing APM language agents, plus OpenTelemetry and OpenLIT ingestion paths for LLM telemetry.
- A visible 2026 agentic roadmap: SRE Agent, AI Agent Monitoring in preview for multi-agent systems, and a no-code Agentic Platform with MCP support, announced February 2026 (TechCrunch).
Limits
- No evaluation framework — datasets, experiments, LLM-as-judge — or red-teaming documented as of July 15, 2026; thin next to eval-first competitors.
- No runtime guardrails or gateway: monitoring only, and drop filters affect telemetry rather than the application's LLM traffic.
- SaaS-only — no self-hosted or customer-VPC deployment documented as of July 15, 2026.
Questions buyers ask
What is the best LLM observability platform in 2026?
There is no single answer, which is why this guide is organized by buyer type. For an AI engineering team building its evaluation discipline, LangSmith is the most complete platform, with Braintrust and Arize close behind on eval depth. For an enterprise already on Datadog or New Relic, extending the APM estate usually beats adding a vendor. For OSS self-hosters, Langfuse. The 'best' tool is the one whose architecture matches where you need the visibility — beside the app, or in the data path.
What is the difference between LLM observability and AI FinOps?
Observability watches behavior — traces, latency, quality. AI FinOps manages spend: token-level cost attribution, budgets, forecasts and enforcement. SDK-instrumented tools (LangSmith, Langfuse, Datadog) report cost accurately but cannot stop it; a request that blows the budget still ran. Gateway-based platforms (Portkey, Kosmoy) enforce budgets in the request path — the overspending call is refused, not just recorded. Mature stacks often need one of each.
Are open-source LLM observability tools good enough for production?
Yes, for the observability job itself. Langfuse (MIT core) and Opik (Apache-2.0) run large production workloads self-hosted, with real evaluation tooling and active development — Langfuse now with ClickHouse's backing. The honest limits: enterprise governance features often sit behind commercial licenses (Langfuse's EE tier), certifications vary, and none of the OSS tools provides runtime enforcement or compliance evidence. Good enough for engineering; not sufficient, alone, for a regulator.
Does Kosmoy replace LangSmith or Langfuse?
No — and this guide says so plainly. Kosmoy has no evaluation suite: no datasets, no LLM-as-judge, no experiments. Teams that need those tools should buy them. Kosmoy replaces the layer none of the eval tools provide: an org-wide [AI inventory](/platform/ai-inventory/), a self-hosted gateway that enforces guardrails and budgets, agent containment with a kill switch, and audit evidence for EU AI Act, ISO/IEC 42001 (aligned) and NIST AI RMF. The common enterprise pattern is one evals tool for builders plus Kosmoy as the control plane.
What happened to Helicone?
Mintlify acquired Helicone on March 3, 2026 and the product entered maintenance mode: security patches, bug fixes and new-model support continue, but feature development ended. Existing deployments keep working, and the open-source core remains available. We left it off this roster because a buyer's guide should recommend forward-looking picks; current users should plan a migration path to Langfuse, Opik or a gateway-native option on their own schedule.
Which platform is best for controlling LLM costs, not just tracking them?
A gateway. Portkey enforces budgets and rate limits per key and workspace with the deepest request analytics in the category. Kosmoy enforces budgets at a self-hosted gateway spanning LLM, MCP and A2A traffic, with attribution per team, project, app and model — and, unlike a pure gateway, ties every cost record to a registered AI system with an owner and a risk tier, so the same data answers the auditor. SDK-instrumented tools tell you what you spent; gateways decide what you may spend.
Methodology
Every vendor was scored on the same ten capability axes used across Kosmoy's comparison pages, from primary sources — vendor documentation, pricing pages, engineering blogs and repositories — checked on July 15, 2026 and cited inline or in each vendor's profile. Scores of 10 are reserved for categorical architectural facts; a specialist always outscores Kosmoy on its own spoke, which is why LangSmith, Langfuse, Arize and Braintrust all beat Kosmoy on observability and evals in this guide.
Numbers from vendors appear as attributed claims with citations; where a fact could not be verified against a primary source — several vendors' exact usage rates, for instance — we link to the pricing page rather than repeat third-party figures. Gaps are phrased as 'does not document X as of July 15, 2026', which is a claim about documentation, not a claim the capability cannot exist.
Disclosure: Kosmoy publishes this guide. The mitigation is structural — the rubric concedes Kosmoy's weak axes explicitly, recommendations are split by buyer type with Kosmoy winning exactly one of the four, and rosters include vendors (Datadog, New Relic, Portkey) that displace Kosmoy entirely for their segments. Ownership changes that affect buying decisions — ClickHouse/Langfuse, CoreWeave/W&B, Mintlify/Helicone, Palo Alto Networks/Portkey — are flagged wherever they bear on a recommendation.
Sources
Every factual claim about another vendor on this page traces to that vendor's own published material or a named third-party source below.
- Interrupt 2026 launches (LangChain blog) — accessed July 15, 2026
- Langfuse joins ClickHouse (January 2026) — accessed July 15, 2026
- Mintlify acquires Helicone (March 3, 2026) — accessed July 15, 2026
- CoreWeave completes acquisition of Weights & Biases (May 2025) — accessed July 15, 2026
- Datadog AI Guard docs — accessed July 15, 2026
- Kosmoy Insights Dashboard — accessed July 15, 2026
- Kosmoy AI Compliance — accessed July 15, 2026
- LangSmith self-hosted overview (docs) — accessed July 15, 2026
- LangSmith self-hosted egress & air-gapped licensing (docs) — accessed July 15, 2026
- LangSmith LLM Gateway (docs, private beta) — accessed July 15, 2026
- LangSmith Sandboxes (docs) — accessed July 15, 2026
- LangSmith Fleet overview (docs) — accessed July 15, 2026
- LangSmith Deployment overview (docs) — accessed July 15, 2026
- LangSmith pricing — accessed July 15, 2026
- Fortune — LangChain raises $125M at $1.25B valuation — accessed July 15, 2026
- Langfuse GitHub repository (MIT core, stars) — accessed July 15, 2026
- Self-hosting overview (same codebase as Cloud) — accessed July 15, 2026
- Enterprise license key (EE feature list, MIT core unlimited) — accessed July 15, 2026
- Networking (no internet access required) — accessed July 15, 2026
- Telemetry (EE license telemetry cannot be disabled) — accessed July 15, 2026
- Security & guardrails doc (runtime blocking delegated to third parties) — accessed July 15, 2026
- Langfuse pricing — accessed July 15, 2026
- Changelog (Experiments, CI/CD gates, Monitors & Alerts) — accessed July 15, 2026
- Phoenix GitHub repository — accessed July 15, 2026
- Arize AX self-hosting docs — accessed July 15, 2026
- Arize AX guardrails docs — accessed July 15, 2026
- Observe 2026 / Arize AX launches (blog) — accessed July 15, 2026
- Arize AX release notes — accessed July 15, 2026
- Arize pricing — accessed July 15, 2026
- Series C press release ($70M) — accessed July 15, 2026
- Datadog LLM Observability product page — accessed July 15, 2026
- AI Agents Console docs — accessed July 15, 2026
- Datadog press release — agentic AI monitoring, LLM Experiments, AI Agents Console — accessed July 15, 2026
- Datadog LLM Observability pricing — accessed July 15, 2026
- AI Guard launch blog — accessed July 15, 2026
- Weave GitHub repository (Apache-2.0 SDK) — accessed July 15, 2026
- W&B pricing — accessed July 15, 2026
- Weave guardrails and monitors docs — accessed July 15, 2026
- CoreWeave + W&B new products press release — accessed July 15, 2026
- Weave production-agent observability announcement — accessed July 15, 2026
- Braintrust Series B announcement — accessed July 15, 2026
- Platform architecture docs (hybrid data plane) — accessed July 15, 2026
- Plans and limits — accessed July 15, 2026
- AI Proxy repository (MIT) — accessed July 15, 2026
- SiliconANGLE — Braintrust $80M Series B (Feb 2026) — accessed July 15, 2026
- Braintrust pricing — accessed July 15, 2026
- Opik GitHub repository (Apache-2.0) — accessed July 15, 2026
- Opik FAQ (deployment, no feature restrictions) — accessed July 15, 2026
- Comet pricing — accessed July 15, 2026
- Opik Guardrails beta announcement — accessed July 15, 2026
- Comet company profile (funding) — accessed July 15, 2026
- Portkey open-source gateway repository — accessed July 15, 2026
- Portkey docs — what is Portkey — accessed July 15, 2026
- Portkey docs — plan & feature comparison (SaaS / hybrid / air-gapped) — accessed July 15, 2026
- Portkey docs — observability — accessed July 15, 2026
- Portkey docs — guardrails — accessed July 15, 2026
- Portkey docs — MCP gateway — accessed July 15, 2026
- Portkey pricing — accessed July 15, 2026
- Palo Alto Networks press release — Portkey acquisition completed (May 29, 2026) — accessed July 15, 2026
- Kosmoy Platform — accessed July 15, 2026
- Kosmoy AI Gateway — accessed July 15, 2026
- Kosmoy Action Capsule — accessed July 15, 2026
- Intro to AI monitoring (docs) — accessed July 15, 2026
- AI agent monitoring (docs) — accessed July 15, 2026
- TechCrunch — New Relic agent platform and OpenTelemetry tools (Feb 2026) — accessed July 15, 2026
- AI Coding Observability coverage (June 2026) — accessed July 15, 2026
- New Relic pricing — accessed July 15, 2026
Shortlisting for a regulated environment?
Kosmoy puts an inventory, a policy gateway and a containment sandbox around every AI your teams run — in your own Kubernetes.
Or email sales@kosmoy.com.