BigQuery Data Engineering Agent vs dbt and Dataform in 2026: should you let AI write your pipelines?

Summary. Google moved the BigQuery Data Engineering Agent to general availability on 22 April 2026, eight months after it first appeared in preview on 4 November 2025. The agent is powered by Gemini, and it writes SQL pipelines directly into BigQuery pipelines and Dataform repositories from plain-language prompts. Early customer Vodafone reported a 90% cut in the time a manual ETL migration usually takes. That number is the headline, but it is not the decision. The agent cannot execute a pipeline, cannot test one for you, and only runs against BigQuery. dbt and Dataform are still the frameworks the code lands in. This guide compares the three on capability, cost, and control, so you can decide what an agent should touch and what your engineers must still own. The short answer: use the agent for boilerplate, migration, and troubleshooting, keep review, testing, and orchestration in human hands, and pick dbt over Dataform only if you need to run on more than one warehouse.

What the BigQuery Data Engineering Agent actually is

The Data Engineering Agent is a first-party agent from Google Cloud, announced in preview on 4 November 2025 and made generally available on 22 April 2026. It runs on Gemini and sits inside BigQuery pipelines, reachable through an "Ask Agent" button in BigQuery Studio or the Dataform user interface.

It is not a new transformation framework. It is an authoring and maintenance layer on top of the framework you already use on Google Cloud. When you describe a pipeline in words, the agent generates the SQL and organises it inside a BigQuery pipeline or a Dataform repository. A prompt such as "load data from the customer_orders bucket, standardise the date formats, remove duplicate entries, and load it into a table named clean_orders" produces working pipeline code you then review.

Four capabilities matter for a real team. The agent creates pipelines from natural language. It modifies existing pipelines, analysing current code and proposing changes you approve before anything runs. It troubleshoots failures through Gemini Cloud Assist, reading execution logs to identify a root cause. And it migrates legacy jobs onto BigQuery, translating proprietary formats from older data-processing tools into native pipelines. It also pulls context from Google Cloud's Knowledge Catalog and Dataplex, so it can read business glossaries, generate data-quality assertions, and automatically encrypt columns flagged as containing personally identifiable information.

The one benchmark Google put its name to

Vendors quote productivity numbers loosely, so it is worth being precise about the one figure with a named engineer behind it. Chris Benfield, Head of Engineering at Vodafone, described the telecom's move to a Dataform environment: "the Data Engineer Agent successfully replicated all existing data and transformations scripts with 100% automation and zero manual intervention. This achievement resulted in a 90% reduction in the time typically required for manual ETL migration."

Read that carefully. The 90% gain is on migration, which is repetitive translation work where an agent is strong. It is not a claim about designing new business logic, tuning cost, or catching a silent data error. Google's other named reference, Spanish-language media group PRISA, was more measured, saying the agent "already delivers value in automating maintenance and small optimizations" while calling its broader promise a work in progress. Treat migration and maintenance as the proven use, and everything else as something you pilot.

What it cannot do

The honest limits are published in the Data Engineering Agent documentation, and they shape the whole decision:

• It cannot execute pipelines. You review the generated code, then run or schedule it yourself. The agent authors; it does not operate.

• It does not accept natural-language commands for notebooks or for data-preparation file types.

• It cannot search web links or URLs passed through prompts or instructions.

• Over Apache Iceberg external tables managed by the Lakehouse runtime catalog, it cannot generate write mutations such as INSERT, UPDATE, DELETE, or MERGE, nor DDL such as CREATE TABLE or DROP TABLE.

• It is bound to BigQuery. There is no Snowflake, Databricks, or Redshift target.

None of these are defects. They define a tool that accelerates authoring inside one warehouse while leaving execution, orchestration, and cross-engine portability to you. The IDE integration that brings the agent into Antigravity and VS Code through the Data Cloud extension is still in preview as of mid-2026, so local-editor workflows are not yet a stable footing for production.

Where Dataform and dbt sit

A common confusion is treating the agent as an alternative to dbt or Dataform. It is not. The code the agent writes lands in Dataform, which uses SQLX, a superset of SQL that adds table references, assertions for testing, and reusable logic. So the real comparison for a Google Cloud team is between two authoring styles on the same Dataform substrate, and between Dataform and dbt as competing frameworks.

Dataform is free. There is no charge for the service itself; you pay only for the BigQuery compute your transformations consume. It is BigQuery-native, version-controlled through Git, and now h author sitting on top of it in the form of the Data Engineering Agent. Its limit is also its design: it runs on BigQuery and nowhere else.

dbt is the portable alternative. dbt Core is open source under the Apache 2.0 licence and runs against many warehouses, including Snowflake, Databricks, Redshift, and BigQuery itself. In 2026 dbt Labs shipped a significant change: dbt Core v2 is a Rust rewrite of the engine, and dbt Labs moved the runtime that powers it back under Apache 2.0 rather than the more restrictive Elastic Version 2 licence it had earlier proposed. The dbt Fusion engine extends that base with column-level lineage and full SQL comprehension, some free and some gated behind a paid plan.

The decision is about reach, not intelligence. If BigQuery is your only warehouse and will stay that way, Dataform plus the Data Engineering Agent gives you native authoring, AI assistance, and zero framework licence cost. If you run, or expect to run, more than one warehouse, dbt keeps your transformation layer portable across all of them.

Feature and control comparison

The pattern is clear. The agent trades portability for a native, no-licence, AI-assisted experience on BigQuery. dbt trades some of that convenience for the freedom to move your transformation code between warehouses.

What you actually pay

Framework choice and running cost are separate line items, and the biggest number is usually neither.

On Google Cloud, the Data Engineering Agent and Dataform carry no framework fee. Your bill is BigQuery compute. Under BigQuery on-demand pricing, the first 1 TiB of data processed each month is free, and you are charged per TiB scanned after that (US multi-region on-demand runs at about $6.25 per TiB in 2026), with active storage at roughly $0.02 per GB per month. The agent needs Gemini in BigQuery enabled on the project, so that entitlement is a prerequisite rather than a separate per-pipeline charge.

dbt Core is free to license, so its cost is the compute and the orchestrator you run it on, such as Airflow, Dagster, or Prefect, plus the engineering time to operate them. dbt Cloud starts free for a single Developer seat and moves to $100 per developer seat per month on the paid tier, with Enterprise pricing negotiated. Independent purchasing data gives a sense of real contracts: procurement marketplace Vendr reports a median dbt Cloud spend of $26,460 per year, with deals ranging from about $14,404 to $87,914.

The honest engineering point: on a BigQuery-only stack, the query bill will usually dwarf any framework fee. That makes the AI authoring layer close to free at the margin, and it makes cost discipline in the SQL the agent generates far more important than the sticker price of the tool. This is where a review step earns its keep, because an agent that writes a wide, unpartitioned scan can quietly raise a monthly bill more than a licence ever would. Teams that track query cost as part of their cloud FinOps practice tend to catch that early.

Should you let AI write your pipelines?

The framing "let AI write your pipelines" is slightly wrong, because the agent already cannot run them. A better framing is: which parts of the pipeline lifecycle can you safely delegate, and which stay with your engineers?

Delegate the mechanical work. Migrations, boilerplate loads, format standardisation, deduplication, schema scaffolding for Medallion, Data Vault, or star schemas, and first-pass documentation are exactly where the agent is strong and where Vodafone's 90% migration gain came from. An agent that drafts a Dataform model in seconds and explains it is a real accelerator for a small team.

Keep three things human. First, review: read the generated SQL before it runs, because the agent optimises for a working answer, not always a cheap or correct one. Second, testing: the agent does not guarantee your data is right, so assertions, freshness checks, and agent evaluations in your CI/CD pipeline are how you catch the silent failures that generated code introduces. Third, orchestration and governance: the agent cannot execute, so scheduling, access control, and change approval still belong to your platform. When an agent can write production data code, the governance layers around it matter more, not less.

This is the same lesson emerging across the wider category of enterprise AI agents in production: the value is real on narrow, verifiable tasks, and the failures come from handing an agent an open-ended job without a check. The Data Engineering Agent's inability to execute is a feature, because it forces the review step that keeps a generated pipeline honest.

The wider data-agent shift

Google is not alone. Every major data platform shipped a natural-language layer over its warehouse in 2025 and 2026, and the choices rhyme with this one. If you work across the lakehouse world, the same build-versus-trust question shows up in the Databricks Genie and Snowflake Cortex Analyst comparison, where per-query AI cost and governance decide as much as raw capability. The common thread is that these agents are strong at translating intent into query code and weak at owning the consequences of running it. The teams getting value are the ones that treat the agent as a fast junior engineer whose work is always reviewed, not as an autonomous operator.

India-specific considerations

For teams in India, two factors sharpen the decision. Cost sensitivity is higher, and a free framework plus pay-as-you-go BigQuery compute is attractive against a per-seat dbt Cloud contract quoted in dollars. A five-engineer team on dbt Cloud's paid tier at $100 per seat is roughly ₹4.9 lakh a year in licence alone before compute, at an assumed rate near ₹83 to the dollar, whereas Dataform adds no licence line. That maths pushes many Google Cloud shops toward Dataform and the agent.

The second factor is data protection. The Digital Personal Data Protection Act 2023 governs how personal data is processed, and the Data Engineering Agent's ability to read Dataplex profiles and automatically encrypt columns flagged as personally identifiable information is genuinely useful for building consent-aware pipelines. It does not make you compliant on its own. You still design the data model, the retention rules, and the access controls. Applications built here should be designed aligned with DPDP requirements, with the agent treated as one control among several rather than a compliance guarantee.

FAQ

How eCorpIT can help

eCorpIT builds and modernises data platforms on Google Cloud for teams that want AI-assisted pipelines without losing control of cost or correctness. Our senior engineering teams set up BigQuery, Dataform, and the Data Engineering Agent with the review, testing, and governance guardrails that keep generated code safe in production, and we help you decide honestly between a BigQuery-native stack and portable dbt. If you are planning a warehouse migration or a data-platform build, see our data platform engineering service and talk to us through our contact page.

References

1. Google Cloud Blog, "Exploring the Data Engineering Agent in BigQuery" (updated 22 April 2026, GA notice): cloud.google.com/blog/products/data-analytics/exploring-the-data-engineering-agent-in-bigquery

1. Google Cloud Documentation, "Use the Data Engineering Agent to build and modify data pipelines" (limitations, requirements): cloud.google.com/bigquery/docs/data-engineering-agent-pipelines

1. Google Cloud Documentation, "Introduction to BigQuery pipelines": cloud.google.com/bigquery/docs/pipelines-introduction

1. Google Cloud, "BigQuery pricing": cloud.google.com/bigquery/pricing

1. Google Cloud Documentation, "Dataplex Knowledge Catalog introduction": cloud.google.com/dataplex/docs/introduction

1. dbt Labs, "dbt pricing": getdbt.com/pricing

1. dbt Developer Blog, "dbt Core v2 is here: still open source, now rebuilt for what's next": docs.getdbt.com/blog/dbt-core-v2-is-here

1. dbt Labs, "New code, new license: understanding the new license for the dbt Fusion Engine": getdbt.com/blog/new-code-new-license-understanding-the-new-license-for-the-dbt-fusion-engine

1. Vendr, "dbt Cloud software pricing and plans": vendr.com/marketplace/dbt-cloud

1. dbt Developer Hub, "About Fusion": docs.getdbt.com/docs/fusion/about-fusion

1. Google Cloud Documentation, "Set up Gemini in BigQuery": cloud.google.com/bigquery/docs/gemini-set-up

_Last updated: 31 July 2026._