Observability AI apps

Evaluation, observability, and optimization platform for AI agents and LLM apps.

Future AGI is an end-to-end platform for testing, evaluating, observing, and improving generative-AI applications. It spans simulations, evaluation suites, real-time tracing and dashboards, runtime guardrails, and a model gateway, with multimodal evaluation across text, image, and audio. The core stack is open-source under Apache 2.0 and can be self-hosted or used as a managed cloud.

AI reliability platform for monitoring and fixing AI and IT systems.

InsightFinder is an AI-driven reliability platform that detects anomalies, pinpoints root causes, and remediates issues across both AI systems and traditional IT infrastructure. It ingests metrics, logs, traces, and events, applying 'composite AI' — streaming anomaly detection plus deterministic causal analysis — and ships fine-tuned, domain-specific small language models rather than relying on generative AI alone. Its ARI agent runs end-to-end incident management: detection, diagnosis, evidence collection, and real-time remediation. The platform runs as SaaS or in fully air-gapped environments and integrates with OpenTelemetry, Datadog, and Prometheus.

The continuous-improvement stack for AI agents.

An evaluation and monitoring platform for AI agents, built around the open-source judgeval framework. Judgment traces an agent's full trajectory — tool calls, memory, search queries, and long reasoning chains — then uses trajectory-level judges to surface failure modes, validate fixes before deploy, and catch behavioral regressions in production. The captured environment data and evals feed back into agent post-training (RL and SFT), not just pass/fail scoring. judgeval is Apache-2.0 and free; the hosted platform adds the dashboard, AutoRubrics, and enterprise features.

An autonomous AI SRE that investigates production alerts and finds root cause.

Cleric is an AI agent for site reliability engineering that automates incident response. When an alert fires, it investigates across your observability dashboards, performs root-cause analysis in minutes, recommends and verifies fixes against the live environment, and retains what it learns as institutional knowledge for the team. It works inside Slack and existing tooling with read-only access by default, adding write access only when teams are ready.

MCP-native eval and observability server for AI agents.

Iris scores AI agent output, catches safety failures, and enforces cost budgets — exposed as an MCP server rather than an SDK. Any MCP-compatible agent discovers its tools (trace logging, output evaluation, rule management, LLM-as-judge) and uses them automatically, with no code changes, so every output flowing through the protocol gets evaluated. It detects PII leaks, prompt injection, hallucinations, and budget anomalies. The core is MIT-licensed and free to self-host; a managed cloud adds dashboards and alerting.

An autonomous AI SRE that triages alerts and finds root causes in minutes.

Traversal is an AI site reliability engineer (SRE) that autonomously triages noisy alerts, traces incidents across services and dependencies to isolate root causes, and feeds production context back to prevent recurrence. Its causal search engine and 'production world model' narrow failures down at petabyte scale. At American Express it reached 82% root-cause accuracy and cut potential MTTR by ~32%, with PepsiCo also among its enterprise users.

Simulate real users to ship reliable voice and text agents.

Okareo is an evaluation and testing platform for AI agents. It drives an agent with synthetic users ("Drivers") that hold personality-rich, multi-turn conversations across voice, text, and headless channels, surfacing edge cases before release. Teams gate releases on conversation quality in CI/CD, turn production failures into automated test scenarios, and evaluate models, RAG pipelines, and agents from one workspace.

Open-source observability for logs, metrics, traces and LLMs in one platform.

OpenObserve (O2) is a unified, open-source observability platform that ingests logs, metrics, traces and real-user monitoring with dashboards, alerts and session replay. It adds LLM observability, a natural-language query assistant and an SRE agent for automated incident correlation and root-cause analysis. It runs as a single self-hosted binary or Helm chart, or as fully managed OpenObserve Cloud.

Automated testing and monitoring for voice and chat agents.

Hamming is an enterprise platform for testing and monitoring conversational AI agents. It auto-generates test scenarios from an agent's prompt, load-tests with tens of thousands of concurrent calls, replays production calls for regression testing, and scores 50+ audio-native metrics like latency, hallucinations, sentiment, and compliance. It integrates natively with Vapi, Retell, ElevenLabs, LiveKit, and Pipecat.

Test, monitor, and self-improve voice and chat AI agents.

Cekura is an automated QA and observability platform for conversational AI agents. Before launch it runs simulations across thousands of diverse personas and edge cases, with red-teaming for bias, toxicity, and jailbreaks; in production it monitors live conversations for instruction-following, hallucinations, and voice-specific quality regressions with real-time alerting. It targets regulated sectors like healthcare and finance where reliability and compliance matter.

Simulation and evaluation platform for voice and chat AI agents.

Coval is an evaluation and monitoring platform for conversational AI agents, applying the simulation-driven testing rigor developed in self-driving to voice and chat. From a handful of test cases it generates thousands of realistic scenarios, runs them against an agent over text or live phone calls, and scores the results on built-in or custom metrics. In production it monitors and scores real calls so teams can catch regressions across millions of conversations.

Evaluation layer that finds and fixes AI agent failures.

Atla is an evaluation platform that automatically discovers, clusters, and ranks failures in AI agents, then suggests fixes. Rather than prompting a general model to grade outputs, it runs on Atla's own Selene LLM-judge models, purpose-trained to score and critique generative-AI responses. It offers Python and TypeScript SDKs and integrates with stacks like OpenAI and LangChain.

The AI quality platform from the team behind DeepEval.

Confident AI is the hosted platform built on top of DeepEval, the open-source LLM evaluation framework. It adds dataset and test management, research-backed metrics, production tracing and monitoring, adversarial red teaming, and governance dashboards so teams can benchmark, observe, and safeguard LLM apps across the dev-to-prod loop. Python and TypeScript SDKs plug into CI and OpenTelemetry, with managed cloud and enterprise self-hosting.

Observability and tracing built for AI agents.

AgentOps is a developer platform for monitoring, debugging, and evaluating AI agents. It records every LLM call, tool use, and decision in a replayable session trace, with time-travel debugging, token and cost tracking, and agent benchmarking. Its open-source SDK drops into Python or TypeScript agents in two lines and integrates with frameworks like CrewAI, LangChain, Autogen, and the OpenAI Agents SDK.

Evaluation and observability for ML and LLM systems.

Evidently is an open-source Python framework for evaluating, testing, and monitoring AI systems — from tabular ML models to LLM apps, RAG pipelines, and agents. It ships 100+ built-in metrics covering data drift, quality, hallucinations, PII leaks, and jailbreaks, plus LLM-as-judge scoring and monitoring dashboards. Evidently Cloud adds a hosted service with no-code evals, alerting, and team management on top of the open library.

Observability and evals for AI agents in production.

Latitude is an open-source monitoring platform for AI agents in production. It captures agent-native traces across multi-turn sessions, tool calls, and full execution paths, then clusters failures into tracked issues and alerts via Slack, email, or webhooks. It is OpenTelemetry-native, supports semantic search over sessions, and auto-builds evals from your team's judgments — run it as managed cloud or self-host the whole stack.

Self-driving observability, evals, and an AI gateway for LLM agents.

Respan is a unified LLM-engineering platform that routes, observes, and evaluates every model call from one control plane. Its AI gateway fronts 500+ models with fallbacks, load balancing, and spend caps; trace trees, dashboards, and alerts cover observability; and rule checks, AI judges, and human review run as one evaluation workflow. An automated eval agent surfaces issues from production traffic so teams can fix what breaks without stitching tools together.

Open-source platform for the ML and GenAI lifecycle.

MLflow is an open-source platform for managing the full machine-learning and GenAI lifecycle — experiment tracking, model registry, deployment, and, more recently, LLM/agent observability. Its GenAI stack adds OpenTelemetry-based tracing, systematic evaluation with built-in metrics and LLM judges, and prompt versioning. Framework- and provider-agnostic, it runs on your own infrastructure with no vendor lock-in.

Open-source evaluation and tracing for LLM and agent apps.

TruLens is an open-source Python library for evaluating and tracing LLM, RAG, and agent applications. You wrap your app with feedback functions that score outputs on metrics like groundedness, context relevance, and answer relevance, then trace runs and compare versions on a metrics leaderboard. It integrates OpenTelemetry tracing and runs locally with a built-in dashboard.

Open-source observability built for AI agents.

An observability and evals platform purpose-built for AI agents: one-line OpenTelemetry-native tracing across SDKs like the Vercel AI SDK, OpenAI, Anthropic, LangChain and browser agents, plus an evals SDK/CLI for local and CI runs. Self-host the Apache-2.0 stack or use Laminar Cloud to debug long-running agent failures.

Open-source product analytics with LLM observability built in.

A product analytics platform that folds session replay, feature flags, experiments, and a data warehouse into one stack. Its LLM analytics suite tracks traces, token costs, latency, and model errors for AI products, with automatic clustering that groups traces by behavior. Usage-based pricing with a generous free tier on every product.

Testing-first evaluation and monitoring for LLM and ML systems.

Deepchecks brings a testing-first approach to AI quality. Its open-source Python library validates ML models and data from research through production, and the LLM Evaluation product extends that into continuous validation of LLM applications — measuring quality, performance, and pitfalls across experimentation and production with CI/CD hooks. Enterprise deployment runs in VPC, on-prem, or bare metal for teams that can't use cloud-hosted eval.

LLM observability, evaluation, and agent testing.

An open-source platform for monitoring, evaluating, and testing LLM and agent applications. LangWatch captures traces, runs evaluations and simulations, and surfaces quality and cost metrics in production. Offered as managed cloud or fully self-hosted for teams with strict data-residency needs.

Observability for LLM and agent apps, from the Pydantic team.

An observability platform that traces your whole application stack — LLM calls, agents, databases, and HTTP — not just the model layer. The Python/JS/Rust SDKs are open source and built on OpenTelemetry, while the hosted backend handles storage, querying, and dashboards. Free tier covers 10M spans per month.

Open-source LLMOps: prompt management, evaluation, and observability.

An open-source platform for building and improving LLM apps. Agenta combines a prompt playground, prompt versioning, evaluation (human and LLM-as-judge), and tracing/observability in one tool. Available as managed cloud or self-hosted, so teams can keep the whole eval-and-trace loop on their own infra.

Build, test, and monitor LLM apps with evals and observability.

Athina AI is a collaborative platform for building, evaluating, and monitoring LLM features. It bundles prompt management, datasets, experiments, production tracing, and a library of 50+ preset and custom evaluations, with human annotation tools on top. The platform pairs with an open-source eval SDK and works with OpenAI, Azure, Bedrock, Vertex, and custom models hosted anywhere.

The observability and evaluation layer for production AI agents.

A platform that unifies monitoring and testing for LLM apps and agents into one improvement loop: distributed tracing, online evaluations and alerts, offline experiments, annotation queues for expert feedback, and CI/CD-integrated regression testing. Built OpenTelemetry-native with support for 100+ models and agent frameworks. The free Developer tier covers small teams; Enterprise adds scale, self-host, and compliance.

LLM observability built on OpenTelemetry.

A reliability platform for LLM apps: its open-source OpenLLMetry SDK instruments LLM, vector-DB, and framework calls as standard OpenTelemetry spans, which Traceloop's hosted dashboard turns into traces, cost/latency analytics, and quality monitoring. Because the data is plain OTel, you can pipe it to existing observability stacks instead of a proprietary one.

AI gateway with observability, guardrails, and governance.

A production AI gateway that gives apps and agents unified access to 1,600+ LLMs across providers behind a single API, with built-in observability, prompt management, guardrails, and governance. Portkey adds routing, caching, fallbacks, cost limits, PII redaction, RBAC, and an MCP gateway. Its core gateway is open-source; run it self-hosted/hybrid or use the managed cloud, which offers a free tier.

Open-source observability and prompt management for LLM apps.

An open-source platform for monitoring, debugging, and improving LLM applications and chatbots. Lunary combines request tracing, cost and user analytics, versioned prompt management with A/B testing, plus human-in-the-loop review and automated scoring. Self-host the Apache-2.0 community edition or use the managed cloud, which starts free with a 10k-events monthly tier.

Tracing and evaluation for LLM apps, from Weights & Biases.

An observability and evaluation toolkit for generative-AI applications. A single @weave.op decorator traces every model call — capturing inputs, outputs, latency, token cost, and errors — and the same SDK builds rigorous evaluations using LLM-as-judge and custom scorers. Traces and experiments are organized in the Weights & Biases web platform for side-by-side comparison across prompts and models.

Evaluation and observability for GenAI apps and agents, with inline guardrails.

A platform for testing, monitoring, and guardrailing LLM and agent applications. It ships 20+ out-of-the-box evals for RAG, agents, and safety, lets teams author custom evaluators, and turns those offline evals into real-time production guardrails powered by its own Luna eval models.

AI observability and security platform for LLM apps, agents, and ML models.

An enterprise platform to monitor, analyze, and safeguard generative AI and ML in production. The Fiddler Trust Service scores prompts and responses for hallucination, toxicity, PII leakage, and prompt-injection, with low-latency guardrails plus real-time alerting and root-cause analysis. Originally an explainable-AI and model-monitoring pioneer, now spanning LLM and agent observability.

OpenTelemetry-based observability for LLM apps and agents.

Langtrace is an open-source observability and evaluation platform for LLM applications, capturing traces, token usage, latency, and cost across popular models, frameworks, and vector databases. Because it emits standard OpenTelemetry spans, traces flow to any OTel-compatible backend, and instrumentation is a two-line SDK install in Python or TypeScript. It ships as a hosted cloud with a free tier plus a self-hostable / on-prem option for data-sensitive teams.

Simulate, evaluate, and observe AI agents end-to-end.

An end-to-end platform for testing and monitoring AI agents across their lifecycle. It combines a prompt experimentation IDE, agent simulation across scenarios and personas, offline and online evaluations with custom metrics, and production observability with tracing and alerts. Aimed at teams shipping reliable agentic and RAG systems.

Open-source LLM evaluation, tracing, and monitoring.

Open-source platform from Comet for debugging and evaluating LLM and agent apps: full tracing of calls, tools, and agent steps, LLM-as-a-judge and heuristic evals, prompt management, and production dashboards. Self-host via Docker or Kubernetes, or use Comet's hosted cloud.

LLM tracing and evaluation with retrieval debugging.

Phoenix is Arize's observability platform — run locally in a notebook or as a hosted service. Especially strong for inspecting RAG pipelines, finding bad chunks, and tracking retrieval quality over time.

LangChain's hosted observability + eval platform.

Tracing, dataset management, eval orchestration, and prompt playground from the LangChain team. Pairs naturally if LangChain or LangGraph already runs in your stack, but works standalone via SDKs.

Drop-in LLM proxy with logging, caching, and cost tracking.

One-line integration — change your OpenAI/Anthropic base URL and get a dashboard with every prompt, response, latency, and dollar tracked. Adds caching and rate-limit handling without code changes.

Open-source LLM observability. Self-hostable, OpenTelemetry-native.

Tracing, evals, prompt management, and dataset tooling for LLM apps — self-host on your own infra or use Langfuse Cloud. The open-source default when you want full ownership of your observability stack.