Agent evals in CI/CD: 4 gates that catch silent failures before customers do (2026)

Summary. LangChain's State of Agent Engineering, published 12 June 2026 from 1,340 responses, found that 89% of organisations have some form of agent observability but only 52.4% run offline evaluations on test sets. Among teams already in production the split is starker: 94% have observability and 71.5% have full tracing. Observability tells you what your agent did after it did it. An eval gate stops the build. That asymmetry is why agent regressions reach customers quietly: nothing crashes, no alert fires, and the failure is a plausible-sounding wrong answer. Quality is the top barrier to production for one third of respondents, ahead of latency at 20%. This guide builds the gate in four stages, with real costs: a 150-case judge run costs about $0.27 on gpt-5.4-mini versus $1.80 on gpt-5.5 at OpenAI's July 2026 list prices, and the tools to wire it (DeepEval, Braintrust, GitHub Actions) ship defaults that will silently let a failing suite pass.

The gap: almost everyone watches, half of them test

The numbers from LangChain's survey are worth sitting with, because they describe a specific engineering failure and not a general lack of maturity. The survey ran from 18 November to 2 December 2025 and collected 1,340 responses, 63% of them from technology companies.

57.3% of respondents have agents running in production, up from 51% the prior year, with another 30.4% actively building toward deployment. So agents are shipping. Of those already in production, 94% have observability in place and 71.5% have detailed tracing that shows individual steps and tool calls.

Evals lag badly. Just 52.4% run offline evaluations against test sets. Online evals sit at 37.3%, rising to 44.8% among teams with production agents. Even among production teams, 22.8% run no evals at all.

The tooling split matters too. Of teams running any evals, human review leads at 59.8% and LLM-as-judge follows at 53.3%. Traditional ML metrics like ROUGE and BLEU see limited use, which makes sense for open-ended agent output where several answers are equally valid.

Read those two lines together. 94% of production teams can reconstruct what an agent did. Roughly half can tell you, before merge, whether a change made it worse. That is the gap where silent failures live.

Why agent failures are silent

A conventional web service fails loudly. It throws a 500, the error rate spikes, the pager goes off. An agent that regresses does none of this. It returns HTTP 200 with a fluent, well-formed, confidently wrong answer.

Three properties make this worse than ordinary flakiness.

Non-determinism compounds. Every model call inside an agent has its own variance. Once a tool result or a retrieval changes, the downstream state diverges, so the same input can produce a different trajectory on the next run. A single passing test proves very little.

The failure surface is semantic, not structural. Your JSON schema validator is happy. The tool call is well-formed. The refund amount is wrong. Exact-match assertions, the backbone of normal CI, cannot see this, because the model returns varying text that is all technically valid.

Regressions arrive without a deploy. A prompt tweak, a retrieval index rebuild, or a provider silently updating a model checkpoint can move behaviour without a single line of your code changing. LangChain's respondents at 10,000-plus employee organisations named hallucinations and output consistency as their biggest quality challenge in write-in answers.

This is the case for a gate rather than a dashboard. A dashboard tells you about the regression on Tuesday. A gate refuses the merge on Monday.

The four gates

Run them in this order. Cost and speed increase down the list, so fail fast and cheap.

Gate 1: deterministic assertions run first

The strongest advice in this area comes from Hamel Husain, founder of the ML consultancy Parlance Labs and co-creator, with Shreya Shankar, of the AI Evals for Engineers and PMs course. Writing on the difference between CI and production evaluation, he is blunt about what belongs in CI: "Favor assertions or other deterministic checks over LLM-as-judge evaluators."

That instruction cuts against most vendor marketing, and it is correct. A deterministic assertion is free, instant, and never disagrees with itself. Before you pay a judge model to reason about quality, check the things that have exactly one right answer:

            def test_refund_agent_calls_the_right_tool(golden):
    trace = run_agent(golden.input)

    # Structural properties: no model needed
    assert trace.tool_calls[0].name == "lookup_order"
    assert trace.output_json["currency"] in {"INR", "USD"}
    assert trace.output_json["refund_amount"] <= golden.order_total
    assert " language model" not in trace.output_text
    assert trace.total_tokens < 8000          # cost budget
    assert trace.wall_clock_ms < 12000        # latency budget
          

Every one of those catches a real production bug class, and none of them costs a token. Refund amounts above the order total, a router calling the wrong tool, a prompt leak, a runaway loop burning 40,000 tokens on one request: assertions catch all of it deterministically.

Gate 2: a small, curated golden set

Husain's guidance on CI datasets is specific: "Test datasets for CI are small (in many cases 100+ examples) and purpose-built. Examples cover core features, regression tests for past bugs, and known edge cases. Since CI tests are run frequently, the cost of each test has to be carefully considered (that's why you carefully curate the dataset)."

The instinct to build a 5,000-case eval set is wrong for CI. It is slow, it is expensive on every pull request, and it dilutes signal. A hundred well-chosen cases that each encode a real failure you have actually seen will catch more regressions than thousands of synthetic ones.

The rule that keeps the set honest: every production incident becomes a test case. Husain describes CI and production monitoring as complementary systems, where new failure patterns found in production get added to the CI dataset to prevent recurrence. That is the loop. Without it your golden set decays into a museum of the bugs you had in month one.

DeepEval models this as an EvaluationDataset of Golden objects, loaded from Confident AI, a CSV, a JSON file, or straight from code:

            from deepeval.dataset import Golden, EvaluationDataset

dataset = EvaluationDataset(goldens=[
    Golden(input="What is the return window for product X1000?"),
    Golden(input="Refund my order 88213 to a different card"),  # ticket #4471
])
          

Gate 3: LLM-as-judge, only where assertions cannot reach

Some qualities have no deterministic check. Whether an answer is factually supported by the retrieved context, whether it actually completed the user's task, whether it held the right tone: these need a model to grade them.

DeepEval, which is Apache 2.0 licensed, exposes these as metrics you attach to a test case, including TaskCompletionMetric, AnswerRelevancyMetric, and FaithfulnessMetric (which requires retrieval_context to be populated). Multi-turn agents get TurnRelevancyMetric, KnowledgeRetentionMetric, RoleAdherenceMetric, and ConversationCompletenessMetric.

The distinction that matters for a gate is between an evaluator and a guardrail. Husain draws the line clearly: guardrails run inline, before the response reaches the user, and can redact, refuse, or regenerate, so their false positives are production bugs. Evaluators run after a response is produced, feed dashboards and regression tests, and do not block the original answer. Judges are usually run asynchronously or in batch precisely because they are heavy. Inline judging is possible only when the latency budget allows it.

Keep the judge scoped. It should grade what assertions cannot, on the subset of cases where structural checks already passed.

Gate 4: compare against a marked baseline

A score in isolation means nothing. 0.83 is only information if you know the last good run scored 0.80.

Both major tools implement this. DeepEval lets you designate a run as the official baseline that future runs are compared against for regressions:

            deepeval test run test_llm_app.py --official
          

The docs are explicit that this is "typically done on your main branch so every baseline reflects merged, production-ready code". Marking a run official requires a CONFIDENT_API_KEY, since official runs live on Confident AI.

Braintrust's GitHub Action does the equivalent and posts the delta straight into the pull request. Its own README shows the report shape:

That table is the whole point of gate 4. An average that moved up three percentage points while four individual cases regressed is exactly the signal a single aggregate score hides.

What it costs to judge every pull request

Judge cost is the reason teams skip gate 3, so price it properly rather than guessing.

Take a realistic CI suite: 150 curated cases, roughly 1,500 input tokens per judge call (rubric, input, agent output, retrieved context) and about 150 output tokens for the verdict and its reasoning. That is 225,000 input and 22,500 output tokens per run. Against OpenAI's published list prices as of July 2026, and assuming 40 pipeline runs a day on a busy repository:

Prices are OpenAI list as of July 2026; the per-run and per-month figures are calculated from the token assumptions above, not measured from a specific repository.

Three things fall out of that table.

The frontier judge is 6.7 times the cost of the mini judge for the same suite, and $2,160 a month is real money for a check that runs on every push. Most CI grading does not need frontier reasoning, because you have already spent gates 1 and 2 removing the cases a small model would get wrong.

The Batch API halves the bill but cannot gate a pull request, because batch jobs are asynchronous and a merge gate has to answer now. Batch belongs in the nightly full-suite run, not the blocking one.

Prompt caching is the real lever, and it is underused. The rubric is identical across all 150 cases, so it is the same prefix every time. Cached input on gpt-5.4-mini is $0.075 per 1M against $0.75 standard, a 10x reduction on the largest part of the payload. Braintrust builds this into its action: setting use_proxy (which defaults to true) points OPENAI_BASE_URL at https://braintrustproxy.com/v1, which its README says "will automatically cache repetitive LLM calls and run your evals faster".

The real cost of an eval gate is rarely the tokens. It is the afternoon someone spends curating the hundred cases that matter.

The defaults that quietly disable your gate

This is the part that turns a green pipeline into a false sense of safety, and both popular tools ship a version of it.

Braintrust's eval-action has a terminate_on_failure option. Its README states the default plainly: "Either true or false. If set to true, the evaluation process will stop when an error occurs. Defaults to false." A team that copies the quickstart YAML gets a workflow that runs evals, posts a nice comment with scores, and does not stop on error. The check is decorative until you change it.

DeepEval has the same trap at the API level, and its FAQ is refreshingly direct about it: "assert_test() is built for CI/CD and raises an assertion error when a metric falls below its threshold, while evaluate() is for scripts and notebooks and collects results without failing. Use assert_test() for gating."

If your CI calls evaluate(), your build cannot fail. Ever. You have bought a dashboard and called it a gate.

The same document warns against reaching for plain pytest: "The plain pytest command works but is highly not recommended. deepeval test run adds a range of functionalities on top of Pytest for unit-testing LLMs, enabled by 8+ optional flags", covering async behaviour, error handling, repeats, and identifiers.

Check three things in your own pipeline this week. Does a deliberately broken agent fail the build? Are you calling the asserting function rather than the collecting one? Does an error in the eval process stop the run, or get swallowed? If you have never watched your gate go red on purpose, you do not have a gate.

A working GitHub Actions setup

DeepEval's own CI/CD documentation gives the shape. This is their pattern, with the ordering discipline from gates 1 through 4 applied:

            name: Agent evals

on:
  pull_request:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Install dependencies
        run: |
          curl -sSL https://install.python-poetry.org | python3 -
          echo "$HOME/.local/bin" >> $GITHUB_PATH
          poetry install --no-root

      # Gate 1: deterministic assertions. Cheap, fast, no tokens.
      - name: Assertions
        run: poetry run pytest tests/test_assertions.py

      # Gates 2 and 3: golden set plus judged metrics. Fails on threshold.
      - name: Agent evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}
        run: poetry run deepeval test run test_llm_app.py
          

The test file itself, following DeepEval's documented pattern for an instrumented agent:

            import pytest
from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe, update_current_trace

dataset = EvaluationDataset(goldens=[
    Golden(input="What is the return window for product X1000?"),
])

@observe()
def my_ai_agent(query: str) -> str:
    answer = support_agent.run(query)
    update_current_trace(input=query, output=answer)
    return answer

@pytest.mark.parametrize("golden", dataset.goldens)
def test_llm_app(golden: Golden):
    my_ai_agent(golden.input)
    assert_test(golden=golden, metrics=[TaskCompletionMetric()])
          

Two details worth noting. You do not need a Confident AI account for the gate itself: DeepEval's FAQ confirms that an LLM judge key such as OPENAI_API_KEY runs the tests entirely locally, and CONFIDENT_API_KEY is optional and only sends results to the cloud. And it is not GitHub-specific. Any provider that runs a shell step works, including GitLab CI, CircleCI, and Jenkins.

Once the app is instrumented with @observe, you can attach metrics to individual spans rather than the whole trace, grading retrievers, tool calls, and sub-agents separately. That is where a failing gate stops being a mystery: instead of "the answer was bad", you get "the retriever returned nothing for this query".

India-specific considerations

For teams building agents from India, or serving Indian users, two things change the calculation.

Data residency has a price now. OpenAI's pricing page states that regional processing endpoints for data residency "are charged a 10% uplift for models released on or after March 5, 2026, that are eligible for data residency". If your eval harness ships customer conversations to a judge model, those payloads carry the same personal data your production path handles. Treat them under the same Digital Personal Data Protection Act 2023 (DPDP) controls you apply in production, rather than assuming a test suite sits outside them. Applied to the table above, a gpt-5.4-mini gate at $324 a month becomes roughly $356 with regional processing. That is a rounding error against a compliance finding.

The cheaper answer is often to not send the data at all. Gate 1 assertions run entirely inside your own CI runner. A golden set built from synthesised or redacted inputs, rather than raw production transcripts, keeps most of your eval suite outside the judge model's context. This is where the ordering of the four gates pays off twice: the deterministic checks that cost no tokens also move no personal data.

For teams designing agent systems against Indian regulatory requirements, our guides on AI agent security and prompt injection guardrails and enterprise AI agent governance layers cover the surrounding controls. If judge spend is your constraint, free tools to measure LLM costs and our LLM token counter help you size the bill before you commit to a suite.

What to do this week

Start with the smallest thing that can go red. Pick the three failures that most recently reached your users, write them as deterministic assertions, and put them in the pipeline. Confirm the build fails. Then add a golden set from the next ten incidents, and only then attach a judge to the qualities the assertions cannot see.

The 52.4% number from LangChain's survey is not a story about tooling gaps. Every tool described here exists, is documented, and is largely free to start. The gap is that observability can be bought and switched on, while evals have to be curated by someone who understands the product. That work does not have a vendor.

FAQ

How eCorpIT can help

eCorpIT builds and hardens production AI agent systems for teams that have moved past the prototype and need the deployment path to be safe. Our senior engineering teams set up the eval harness described here: deterministic assertion suites, curated golden sets drawn from your real incident history, judged metrics scoped to what assertions cannot check, and a baseline comparison wired into your existing pipeline on GitHub Actions, GitLab CI, or Jenkins. We design these systems aligned with DPDP 2023 requirements, keeping personal data out of judge payloads where the check can be made deterministically instead. If your agents are in production without a gate, talk to our team.

References

1. LangChain, State of Agent Engineering, 12 June 2026

1. Hamel Husain, LLM Evals: Everything You Need to Know

1. Hamel Husain, Your AI Product Needs Evals

1. DeepEval documentation, Unit Testing in CI/CD

1. Braintrust, eval-action GitHub Action repository

1. OpenAI API pricing

1. Parlance Labs, team

1. DeepEval documentation, Flags and Configs

1. DeepEval documentation, Component-Level Evals

1. DeepEval documentation, Metrics introduction

1. DeepEval, LLM-as-a-Judge in 2026: evaluation techniques and best practices

1. Braintrust eval on GitHub Marketplace

1. Maven, AI Evals for Engineers and PMs by Hamel Husain and Shreya Shankar

1. DeepEval guides, Regression Testing LLM Systems in CI/CD

Last updated: 16 July 2026.