Skip to content

EvalIris

Iris

MCP-native eval and observability server for AI agents.

Categories
EvalObservabilityMCP
Pricing
FREEMIUM
Source
Open core
Hosting
Hybrid
Platforms
API
Models
BYO key / model
Verified
Jun 20, 2026

Iris scores AI agent output, catches safety failures, and enforces cost budgets — exposed as an MCP server rather than an SDK. Any MCP-compatible agent discovers its tools (trace logging, output evaluation, rule management, LLM-as-judge) and uses them automatically, with no code changes, so every output flowing through the protocol gets evaluated. It detects PII leaks, prompt injection, hallucinations, and budget anomalies. The core is MIT-licensed and free to self-host; a managed cloud adds dashboards and alerting.

Capabilities 5

What it actually does — grouped by capability family.

  • MCP server (secondary capability)
  • LLM evaluation (primary capability)
  • LLM observability (primary capability)
  • Guardrails (secondary capability)
  • Cited answers (secondary capability)

Pros & cons

  • No SDK or instrumentation to add
  • Free self-host, free cloud tier
  • Trace logging and LLM-as-judge scoring
  • PII, injection, and cost checks
  • Newer, niche MCP-focused tool
  • Best fit for MCP-based agents
  • Smaller ecosystem than SDK evals

Tags

Alternatives to Iris (3)

View all Eval →
  • View Braintrust details
    EvalFREEMIUM

    Braintrust

    Braintrust

    Hosted eval + tracing platform for LLM apps.

    Production-grade eval orchestration with a dashboard, dataset versioning, and OpenTelemetry tracing. Useful once eval volume outgrows a CI YAML file.

    Eval workflow as the primary interface
    Closed-source SaaS
    • eval
    • tracing
    • datasets
    • production
    Open
  • View DeepEval details
    EvalFREEMIUMOpen core

    DeepEval

    Confident AI

    Pytest-style framework for evaluating LLM apps in CI.

    Open-source (Apache 2.0) framework for evaluating LLM apps the way Pytest tests code — assertions backed by 50+ ready metrics spanning LLM-as-judge, RAG, agents, conversation, and safety. Plugs into LangChain, CrewAI, OpenAI Agents and more. Confident AI is the paid cloud platform that adds test management, dashboards, and observability on top.

    Assertions run in your CI pipeline
    LLM-as-judge adds cost
    • eval
    • open-source
    • llm-as-judge
    • rag
    • +1
    Open
  • View Promptfoo details
    EvalFREEOSS

    Promptfoo

    Promptfoo

    LLM eval CLI with rubric scoring and golden sets.

    YAML-driven eval harness. Pair a prompt with a goldset, define rubrics, run across multiple models in CI. Strong for catching prompt regressions before they hit production.

    YAML-driven, version-controllable evals
    CLI-first, less of a hosted UI
    • eval
    • ci
    • rubric
    • open-source
    Open
Open Iris