Skip to content

EvalUK AI Security Institute

Inspect AI

Open-source Python framework for large language model evaluations.

Category
Eval
Pricing
FREE
Source
Open source
Platforms
CLIAPI
Models
BYO key / model
Verified
Jun 8, 2026

A framework for building and running reproducible LLM and agent evaluations, structured around datasets, solvers, and scorers. Ships sandboxed tool execution, multi-turn agent workflows, and a log viewer, plus a companion library of 200+ prebuilt evals. Run any eval against any model via the inspect CLI or the Python API.

Capabilities 5

What it actually does — grouped by capability family.

  • Sandboxed code execution (secondary capability)
  • Agent framework (secondary capability)
  • Tool / function calling (secondary capability)
  • LLM evaluation (primary capability)
  • LLM observability (secondary capability)

Pros & cons

  • Adopted across major safety labs
  • Composable datasets/solvers/scorers
  • 200+ prebuilt evals (inspect_evals)
  • Sandboxed tool + multi-turn agent runs
  • MIT-licensed, provider-agnostic
  • Python/code framework, not a UI product
  • Steeper than no-code eval tools
  • You wire up your own model keys

Tags

Further reading

Alternatives to Inspect AI (4)

View all Eval →
  • View Promptfoo details
    EvalFREEOSS

    Promptfoo

    Promptfoo

    LLM eval CLI with rubric scoring and golden sets.

    YAML-driven eval harness. Pair a prompt with a goldset, define rubrics, run across multiple models in CI. Strong for catching prompt regressions before they hit production.

    YAML-driven, version-controllable evals
    CLI-first, less of a hosted UI
    • eval
    • ci
    • rubric
    • open-source
    Open
  • View DeepEval details
    EvalFREEMIUMOpen core

    DeepEval

    Confident AI

    Pytest-style framework for evaluating LLM apps in CI.

    Open-source (Apache 2.0) framework for evaluating LLM apps the way Pytest tests code — assertions backed by 50+ ready metrics spanning LLM-as-judge, RAG, agents, conversation, and safety. Plugs into LangChain, CrewAI, OpenAI Agents and more. Confident AI is the paid cloud platform that adds test management, dashboards, and observability on top.

    Assertions run in your CI pipeline
    LLM-as-judge adds cost
    • eval
    • open-source
    • llm-as-judge
    • rag
    • +1
    Open
  • View Ragas details
    EvalFREEOSS

    Ragas

    Exploding Gradients

    Evaluation toolkit for RAG and LLM applications.

    Open-source (Apache-2.0) Python framework for evaluating retrieval-augmented generation and LLM apps. Provides reference-free metrics — faithfulness, answer relevancy, context precision/recall — plus knowledge-graph-based synthetic test generation. Integrates with LangChain, LlamaIndex, and CI pipelines.

    Faithfulness & relevancy metrics
    LLM-judge scores add cost/variance
    • eval
    • rag
    • llm-as-judge
    • open-source
    • +1
    Open
  • View Braintrust details
    EvalFREEMIUM

    Braintrust

    Braintrust

    Hosted eval + tracing platform for LLM apps.

    Production-grade eval orchestration with a dashboard, dataset versioning, and OpenTelemetry tracing. Useful once eval volume outgrows a CI YAML file.

    Eval workflow as the primary interface
    Closed-source SaaS
    • eval
    • tracing
    • datasets
    • production
    Open
Open Inspect AI