RAG evaluation tools 2026: Ragas vs DeepEval vs Braintrust vs Giskard

Summary. For most teams shipping retrieval-augmented generation in late 2026, the practical default is DeepEval (Apache-2.0, v4.3.1 released 16 October 2026) for CI gating, with Braintrust as the hosted choice when engineers and product managers review experiments together. Ragas still defines the vocabulary (faithfulness, context precision, context recall, noise sensitivity) under Apache-2.0, but it looks dormant: the last release was v0.4.3 on 13 January 2026, nothing has been committed to the main branch since 24 February, its founders now build agent simulation environments as Vibrant Labs, and CVE-2026-6587 in its multimodal faithfulness code has no patch. Giskard released a full v3 rewrite on 26 August 2026 that replaces RAGET with quality_scan and needs Python 3.12. Confident AI (DeepEval's hosted platform) charges per organisation at $0, $200 or $2,000 a month. Braintrust Pro costs $249 a month with 50,000 scores included, then $1.50 per 1,000. Giskard publishes no price for its enterprise platform. OpenAI's hosted Evals platform turns read-only on 31 October and shuts on 30 November 2026, pointing users to Promptfoo, which joined OpenAI in March. In our worked example the judge model drives cost far more than the platform: 500 questions, 4 metrics and 5 runs a week cost about $867 a month on gpt-6.1-sol and $43 on gpt-6-luna.

About this article. Versions, licences, prices, CVE records and research claims were checked against vendor pages, documentation, GitHub, PyPI, NVD, arXiv and the Gazette of India on 22 October 2026. Vendors change pricing often, so re-check before you commit. No vendor paid for inclusion or placement. Researched with AI assistance and reviewed by our editorial team.

What changed in 2026

Five changes make this a different decision from a year ago.

Ragas went quiet. The Ragas creators came out of stealth on 19 November 2025 as Vibrant Labs, with a $2.5 million pre-seed round to build simulation environments for long-horizon agents. The repository, now at vibrantlabsai/ragas, has had no release since v0.4.3 and no commit to main since 24 February 2026, and outside pull requests that tried to fix the CVE below were closed without merging. CVE-2026-6587, published on 20 April 2026, is a server-side request forgery in the multimodal faithfulness module affecting versions up to 0.4.3. VulDB scores it 6.3 under CVSS 3.1 (2.1 under CVSS 4.0) and Red Hat 8.1, and the record says the vendor did not respond. GitHub's advisory lists no patched version.

Giskard rewrote its library. Giskard v3.0.0 shipped on 26 August 2026 and was announced on 2 September as "a full rewrite rather than an in-place API upgrade". It is split into five packages, requires Python 3.12, and v2 stays installable but is no longer actively maintained. The migration guide confirms that RAGET becomes quality_scan.

Platform pricing was reset. Braintrust customers who signed up before 16 March 2026 and have not changed plans stay on legacy plans; downgrading ends legacy status. Braintrust raised an $80 million Series B led by ICONIQ in February. Confident AI bills its Starter and Team plans per organisation.

OpenAI is leaving hosted evals. OpenAI's deprecation notice of 3 June 2026 says existing evals become read-only on 31 October 2026 and the Evals dashboard and API shut down on 30 November, with a Promptfoo migration guide as the suggested path. OpenAI agreed to acquire the MIT-licensed Promptfoo on 9 March 2026, and Promptfoo says it joined OpenAI on 16 March and remains MIT.

Non-generative judges arrived. DeepEval added JevEval, built on Jev, a System One model from TypeSafe AI that answers each rubric question with a calibrated probability rather than generated text. Braintrust serves Jev for scorers at $0.042 per million input tokens and $0 for output once monthly credits are used. Evidence is thin: a September 2026 radiology study found Jev a practical, low-cost judge, but a local tool, RadMatch, agreed better with experts on clinically significant errors.

Ragas: the reference metrics, with a maintenance question

Ragas began as a 2023 paper proposing reference-free RAG metrics, so you can score retrieval and generation without hand-labelled answers. It is an Apache-2.0 Python library (about 16.1k GitHub stars) that runs wherever your code runs, with any judge model.

Metrics. The documented set covers context precision, context recall, context entities recall, noise sensitivity, response relevancy, faithfulness and multimodal variants, plus agent, SQL, rubric, factual-correctness and BLEU, ROUGE or CHRF metrics. Faithfulness is supported claims divided by total claims: an LLM extracts the claims, then checks each against the context. A documented variant swaps the checking step for Vectara's open HHEM-2.1-Open classifier, which runs on CPU by default.

Test-set generation. Ragas builds a knowledge graph from your documents, then generates single-hop and multi-hop questions, with documented persona generation. It is still one of the better free generators for a first golden set.

Pricing. Free. Ragas publishes no hosted plan or price; enterprise enquiries go to the founders by email.

Limits. No CI runner, so you wrap metrics in pytest. The API has churned: there is a v0.3 to v0.4 migration guide, and metric pages say the legacy API will be removed in version 1.0. With no commits since February, assume you will patch issues yourself.

Best for: teams that want the canonical metric definitions, can pin a version, and will patch or fork. Avoid the multimodal faithfulness and relevance metrics, or never pass untrusted URLs or file paths into retrieved contexts, until the CVE is fixed.

DeepEval and Confident AI: pytest-style gating

DeepEval is Confident AI's open-source framework, Apache-2.0, with about 19k GitHub stars and v4.3.1 on PyPI as of 16 October 2026. Its README describes it as Pytest-like but built for unit-testing LLM apps, and deepeval test run is its CI entry point.

Metrics. RAG coverage includes answer relevancy, faithfulness, contextual recall, contextual precision, contextual relevancy and a RAGAS composite that averages four of them. It also ships agent, multi-turn and MCP metrics, and custom judges through G-Eval, the graph-based DAG builder and JevEval; Confident AI counts 30+ single-turn and 15+ multi-turn metrics. A golden synthesizer generates test data, and component-level evals score the retriever and generator as separate spans inside one trace.

Pricing (Confident AI). The library is free and runs locally with your own judge key. The hosted platform is optional:

• Free: $0, 2 seats, 1 project, 5 test runs a week, 1 GB-month of trace spans.

• Starter: $200 a month per organisation, unlimited seats, 5 projects, 5 GB-months, then $1 per GB-month.

• Team: $2,000 a month, unlimited projects, 75 GB-months, SSO, custom RBAC and SOC 2.

• Enterprise: custom, with on-premises deployment, custom data residency and HIPAA.

Strengths. assert_test() fails the build when a metric drops below its threshold (tests marked flaky only warn). Component-level evals show whether the retriever or the generator caused a regression.

Limits. Metric breadth invites over-measuring; pick four or five and calibrate them. Marking a run as the official baseline and sharing reports need a Confident AI key; without one, everything runs locally.

Best for: engineering teams that want RAG and agent evals as code that fails pull requests, with an optional flat-priced dashboard.

Braintrust: hosted experiments and scorers

Braintrust is a commercial platform for evals, datasets, experiments and production tracing, with a built-in Loop agent that runs evals, generates test cases and iterates on prompts. Its MIT-licensed scorer library, autoevals (about 1,100 stars), works without the platform and includes eight RAG scorers in Python and TypeScript: context precision, relevancy, recall and entity recall, faithfulness, answer relevancy, answer similarity and answer correctness.

Pricing. Braintrust bills processed data (bytes ingested across logs, experiments and datasets, measured at ingestion, so deleting data does not reduce it) and scores (every offline, online, human-review or custom scorer result), plus model tokens beyond monthly credits.

• Starter: $0, 1 GB a month then $4 per GB, 10,000 scores then $2.50 per 1,000, 14-day log retention, $10 of model credits, unlimited users.

• Pro: $249 a month, 5 GB then $3 per GB, 50,000 scores then $1.50 per 1,000, 30-day retention (extendable to 180 days at $0.50 per GB-month), $100 of model credits.

• Enterprise: custom, with BYOC and self-hosted data planes, SAML or OIDC SSO, the SOC 2 attestation report and a BAA.

Braintrust lists SOC 2 Type II compliance on every plan, and experiments keep up to 365 days of history on every plan, which matters more for eval work than log retention.

CI. The MIT-licensed eval-action posts a live summary of score changes, improvements and regressions as a pull-request comment. Its terminate_on_failure input defaults to false, so set it explicitly if you want the job to fail.

Limits. Self-hosting and BYOC are Enterprise-only. Ten metrics on a 500-row dataset is 5,000 billable scores per run.

Best for: teams where product, QA and engineering compare prompt, retriever and model changes in one place. For how Braintrust's tracing compares with Langfuse, LangSmith and Phoenix, see our LLM observability comparison.

Giskard: RAG quality plus red-teaming, freshly rewritten

Giskard is a Paris-based company whose open-source library lives at Giskard-AI/giskard-oss (about 5.9k GitHub stars). The repository is Apache-2.0, though several v3 sub-packages, including giskard-checks, declare the MIT licence. Version 3 centres on two stable packages:

• giskard-checks: single-turn and multi-turn scenario evals with deterministic checks and LLM judges such as Groundedness, exporting JUnit reports for CI.

• giskard-scan: vulnerability_scan for prompt injection, jailbreaks and harmful content, and quality_scan with a KnowledgeBase for RAG quality, succeeding RAGET.

Pricing. The open-source tier lists best-effort maintenance and community support with no fee. The Professional platform shows no price: 50+ adversarial probes including multi-turn attacks, domain-specific generation of evaluation datasets, human review, SSO and RBAC, and on-premises, private cloud or SaaS deployment. Giskard lists SOC 2, HIPAA and GDPR compliance on its pricing page.

Limits. v3 is weeks old (the giskard 3.0.1 meta-package shipped on 2 October 2026), so expect API movement and fewer examples. RAGET code needs porting, and the tabular ML scan stays on v2.

Best for: teams that want RAG quality checks and security probing from one library, and European buyers who prefer an EU vendor.

Alternatives readers ask about

TruLens is maintained in the open by Snowflake. Arize calls Phoenix open source, but its Elastic License 2.0 is not OSI-approved, so it is source-available. LangSmith includes 5,000 base traces on Developer and 10,000 on Plus, with further usage billed in LangChain Standard Units.

Head-to-head comparison

Which one to pick

Mixing is common: Ragas or Giskard to generate the first golden set, DeepEval or autoevals in CI, and a hosted platform only if non-engineers need the results.

Method: running RAG evals you can trust

Four practices matter more than the tool you choose.

Build a golden set before you pick metrics

Start with 200 to 500 questions from redacted user queries, subject-matter experts and synthetic generation, tagged by source. Include unanswerable, multi-hop and distractor-heavy cases (what Ragas's noise sensitivity measures), and freeze the set per release. Score retrieval separately: with labelled relevant chunks, hit rate and MRR cost nothing and catch indexing regressions before any judge runs. Parsing and index choices drive most retrieval failures, as our guides to RAG document parsing and vector databases for RAG explain.

Treat the judge as an instrument with known biases

The MT-Bench study found strong judges such as GPT-4 agreed with human preferences over 80% of the time, the same level as human-to-human agreement, but documented position, verbosity and self-enhancement biases. Later work showed judges recognise and favour their own generations. Use a judge from a different model family than your generator, prefer binary or short ordinal scales, randomise pairwise order, and pin the judge model and prompt.

Calibrate against humans

Have a domain expert label 50 to 100 items per metric and measure judge agreement. Rewrite or drop metrics that disagree, and re-check whenever the judge model changes. Ragas, Braintrust and Confident AI all support this; a spreadsheet also works.

Gate regressions, not absolute scores

Compare each run with a baseline from your main branch and fail on drops beyond measured run-to-run noise. Gate per slice, because an average can rise while a critical slice falls. Our guide to agent evals in CI/CD covers pipeline wiring, and our tool-use reliability evaluation guide covers agentic RAG that calls tools.

Worked cost example: 500 questions, 4 metrics, 5 runs a week

Assumptions. A 500-question golden set scored on faithfulness, response relevancy, context precision and context recall. We assume two judge calls per metric per question on average (faithfulness is a two-step metric; context precision grows with chunk count), so 8 calls per question and 4,000 per run. Each call carries about 3,000 input tokens (question, answer, roughly 2,000 tokens of context, rubric) and 400 output tokens; tokenisers differ by vendor, so count your own prompts with our LLM token counter.

Per run: 4,000 × 3,000 = 12 million input tokens and 4,000 × 400 = 1.6 million output tokens. At 5 runs a week, a month is 5 × 52 ÷ 12 ≈ 21.7 runs.

For gpt-6.1-sol, that is 12 × $2 + 1.6 × $10 = $40 per run. List prices are from OpenAI and Anthropic (Haiku 5.5's rate applies to prompts up to 100,000 tokens); the figures are arithmetic on our assumptions, not measurements.

Platform layer. Four metrics on 500 rows is 2,000 scores per run, about 43,300 a month.

• Ragas or DeepEval run locally: $0.

• Braintrust Starter: 10,000 scores included, so about 33,300 × $2.50 per 1,000 ≈ $83. At an assumed 50 KB per logged row, a month is about 0.54 GB, inside the free 1 GB. Pro at $249 covers both.

• Confident AI Free allows exactly 5 test runs a week (2 seats, 1 project); Starter is $200.

Totals with gpt-6.1-sol: about $867 (library only), $950 (Braintrust Starter), $1,116 (Braintrust Pro) or $1,067 (Confident AI Starter). With gpt-6-luna: about $43, $126, $292 or $243.

The judge choice moves the bill twentyfold between gpt-6.1-sol and gpt-6-luna; the platform moves it by at most $249. Calibrate a cheap judge first and pay for a frontier judge only where the cheap one disagrees with your experts. Batch pricing suits nightly runs, not a blocking pull-request gate.

India, EU and US data residency

All four open-source libraries (Ragas, DeepEval, Giskard and autoevals) run inside your own VPC. Data leaves only through judge model calls and any platform you send results to.

Judge models. OpenAI charges a 10% uplift on regional processing endpoints for models released on or after 5 March 2026, and Anthropic charges 1.1 times standard prices for US-only inference on Claude 4.6 and later models. Check your provider's supported regions for your judge, or self-host an open-weights judge if payloads cannot leave India.

Platforms. Confident AI stores data in the US by default, with the region chosen at sign-up. Its data-residency docs list an EU region on all plans and a HIPAA BAA from the Team plan, while the same page's header and the pricing page place data residency and HIPAA under Enterprise, so confirm in writing. Braintrust runs SaaS on every plan; BYOC and self-hosted data planes in your own cloud require Enterprise. LangSmith offers US or EU cloud on Plus and Enterprise. Giskard Professional offers on-premises and private cloud.

DPDP. India's DPDP Rules were notified on 13 November 2025, and most data fiduciary obligations, including notice, consent, security safeguards and the transfer rules, apply from 13 May 2027. Golden sets built from production queries contain personal data: redact them before they reach a judge and store them under production-log controls. EU teams should sign the vendor's DPA (Braintrust offers a click-through DPA on Pro).

Implementation checklist

1. Map one metric to each of your three costliest failure modes (wrong answer from good context, missed retrieval, unsafe refusal).

1. Build a versioned, redacted golden set of 200 to 500 questions, tagged by slice.

1. Add free retrieval metrics (hit rate, MRR) before LLM-judged ones.

1. Pin one judge from a different model family and calibrate it on 50 to 100 human labels per metric.

1. Wire the gate: deepeval test run, Braintrust eval-action with terminate_on_failure: true, Giskard JUnit output, or pytest around Ragas.

1. If you use OpenAI's Evals platform, export definitions before 31 October and finish migrating before 30 November 2026.

1. If you depend on Ragas, pin the version, keep a fallback implementation of critical metrics, and keep untrusted URLs out of multimodal metrics.

If your RAG system routes between retrieval strategies, evaluate each route separately; our agentic RAG vs classic RAG guide explains why.

How eCorpIT can help

eCorpIT is a Gurugram-based technology organisation, founded in 2021, assessed at CMMI Level 5 and MSME certified, with senior-led engineering teams working across AWS, Microsoft and Google platforms. Our AI evals and observability service builds evaluation pipelines for RAG assistants and agents, with golden sets from your real queries, judges calibrated against your domain experts and regression gates in GitHub Actions, GitLab CI or Jenkins, alongside our RAG knowledge assistant service. Talk to us at /contact-us/.

Last updated: 22 October 2026.

References

1. deepeval 4.3.1 — PyPI

1. Ragas repository — GitHub

1. Ragas team emerges from stealth as Vibrant Labs — Vibrant Labs

1. CVE-2026-6587 detail — NIST National Vulnerability Database

1. Giskard v3 is here, discussion 2814 — GitHub

1. Migrate from v2 — Giskard documentation

1. Billing FAQ — Braintrust documentation

1. Announcing our Series B — Braintrust

1. Deprecations — OpenAI API documentation

1. Promptfoo is joining OpenAI — Promptfoo

1. JevEval metric — DeepEval documentation

1. Plans and limits — Braintrust documentation

1. Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality — arXiv

1. Ragas: Automated Evaluation of Retrieval Augmented Generation — arXiv

1. List of available metrics — Ragas documentation

1. Faithfulness — Ragas documentation

1. Migrate from v0.3 to v0.4 — Ragas documentation

1. Pricing — Confident AI

1. Component-level LLM evaluation — DeepEval documentation

1. Unit testing in CI/CD — DeepEval documentation

1. autoevals repository — GitHub

1. Pricing — Braintrust

1. eval-action repository — GitHub

1. Self-hosting — Braintrust documentation

1. giskard-oss repository — GitHub

1. Pricing — Giskard

1. TruLens repository — GitHub

1. Phoenix LICENSE (Elastic License 2.0) — GitHub

1. LangSmith plans and pricing — LangChain

1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — arXiv

1. LLM Evaluators Recognize and Favor Their Own Generations — arXiv

1. API pricing — OpenAI

1. Pricing — Claude Platform documentation

1. Data residency — Confident AI documentation

1. Digital Personal Data Protection Rules, 2025, G.S.R. 846(E) — Gazette of India