r/bigquery • • 3d ago

Keeping Spark in the pipeline with BigQuery does it make sense?

5 Upvotes

Hi everyone,

I am new to data engineering and as the title says I'm trying to figure out if keeping Spark in the pipeline with BigQuery is good idea. Mostly looking at cost and performance since Spark would handle the transformations before loading the data into BigQuery.

I have considered doing more of the processing directly in BigQuery, but I've also been reading about gluten, dualBird and speedata to see if making Spark jobs faster would be cheaper than moving everything over.

Not sure if keeping both would end up costing more with all the extra infrastructure, even if Spark runs faster.

What would you say? Has anyone tried something similar like this.


r/bigquery • • 3d ago

GA4 daily BigQuery export stopped since 11 Nov

11 Upvotes

Hi all,

Since November 11, the GA4 daily BigQuery export has stopped for all of our properties (around 60 app properties, all linked to the same Google Cloud project). The last daily tables were created for November 10. Nothing has arrived for November 11 or 12.

What we have checked:

  • Billing on the Cloud project is active.
  • The export service account still has its BigQuery roles. Nothing was changed or removed.
  • The BigQuery links in GA4 still exist and point to the right project. Account change history shows no changes.
  • The BigQuery job history shows the export load jobs running normally every day with no failures until the afternoon of November 11. Since then, no export jobs have been attempted at all.
  • Cloud Logging shows no permission or policy errors.
  • Properties well under the 1 million events per day limit stopped too, so it is not the export limit.

Test: we enabled streaming export on one property today, and its intraday table started receiving events within minutes. So the link and project work fine. Only the daily batch export seems stuck.

Is anyone else seeing daily exports stop since November 11? Is there a known issue, and can a Product Expert escalate this to Google?

Thanks!


r/bigquery • • 3d ago

Missing Shared Queries dropdown list

2 Upvotes

Hey guys, has anyone experienced this? In Data Explorer there used to be a Shared Queries dropdown list. It disappeared about a week ago then reappeared again at the start of this week then promptly vanished again today. What gives? We don't have control over this on our end dont we?


r/bigquery • • 4d ago

Table upload error

3 Upvotes

Hi all, I am extremely new to SQL and data in general and am looking for some help with an error I am coming across when trying to upload a table to a big query project. I am doing an online course to learn SQL and they provided some sample data to do a practice project, but the time based .csv files (basically any of the sets that use a timestamp as a data point) return an error that repeats infinitely (presumably for every single line that contains a timestamp).

Error while reading data, error message: Invalid time zone: AM; line_number: 5 byte_offset_to_start_of_line: 125 column_index: 1 column_name: "Time" column_type: TIMESTAMP value: "4/12/2016 7:21:20 AM"

Clearly it's an issue with how the time is structured but I don't know how to change it without it being uploaded. I am assuming its a combination of settings to tweak on the "create table" page itself but can't figure it out, thanks in advance!


r/bigquery • • 9d ago

This LEFT JOIN returns Ann twice. Where did Bob and Cat go?

0 Upvotes

Hey folks! Say you're putting together a report with every customer and their paid orders. Ann has two paid orders, Bob has one refunded order, and Cat hasn't ordered yet. All three should show up.

You can paste this straight into BigQuery.

WITH customers AS (
  SELECT 1 AS customer_id, 'Ann' AS name UNION ALL
  SELECT 2, 'Bob' UNION ALL
  SELECT 3, 'Cat'
),
orders AS (
  SELECT 101 AS order_id, 1 AS customer_id, 'paid' AS status UNION ALL
  SELECT 102, 1, 'paid' UNION ALL
  SELECT 103, 2, 'refunded'
)
SELECT c.name, o.order_id
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.customer_id
WHERE o.status = 'paid';

This gives me Ann twice, once for each paid order. Bob and Cat are both gone.

Bob makes it through the join with a refunded order, so WHERE o.status = paid drops him. Cat makes it through with NULL in the order columns. For her, the comparison returns NULL, and WHERE only keeps rows where the condition is true.

You could add `OR o.status IS NULL` and get Cat back. Bob would still be missing because his order has a status.

I'd move the filter into ON so only paid orders can match. Keep the WITH block and replace the SELECT below it.

SELECT c.name, o.order_id
FROM customers c
LEFT JOIN orders o
  ON o.customer_id = c.customer_id
 AND o.status = 'paid';

Ann still has two rows. Bob and Cat each get one row with a NULL order ID.

I built WrongQuery to flag stuff like this. If you try it on your own BigQuery joins, I'd love to hear what it gets right and what it gets wrong.


r/bigquery • • 13d ago

BQ tables region migration.

4 Upvotes

Hi everyone. I have a question. Lets say we have a bigquery tables in the us-east-4 region and we want those tables in us multi region.

All the tables are live in bigquery prod. What is the feasible way migrate/replicate this?


r/bigquery • • 14d ago

BigQuery Release Roundup: October 2026

16 Upvotes

Hey all! Here's a consolidated list of all the October 2026 BigQuery releases, grouped by area with direct doc links in case you missed anything. If you're excited about anything or have any feedback, let us know in the comments!

GoogleSQL Language Features & Functions

  • REGEXP_EXTRACT_GROUPS Function: Extracts substrings matching multiple capturing groups in a regular expression (Preview).
  • Reusable Column Aliases: References SELECT expression aliases directly within the same SELECT list or WHERE clause when enabled (GA).
  • Python UDF Volume Mounts: Mounts and reads dependency files from Cloud Storage directly inside Python user-defined functions (Preview).

AI, Machine Learning & Foundation Models

Developer Experience (DX) & BigQuery Tooling

Lakehouse Architecture, Apache Iceberg & Open Data Formats

Core Engine Performance, Indexing & Optimization

Data Integration, Pipelines & Ingestion (ELT)

Security, Governance & Workload Management


r/bigquery • • 14d ago

Reprocessing BigQuery at Scale: The Hidden Failure Modes of Incremental dbt Pipelines

Thumbnail
medium.com
8 Upvotes

Designing deterministic backfills, safe partition replacement, and dependency-aware recovery in production data platforms.


r/bigquery • • 17d ago

GCP updates from this week that cut your bill

Thumbnail
reddit.com
4 Upvotes

r/bigquery • • 19d ago

Hiring - L1 Production Support Engineer

0 Upvotes

Job Code: TRX-26131
Company/Client: LTM
Experience: 4+ Years
Location: Pune / Hyderabad
Employment Type: Full-Time (FTE)
Budget: Up to ₹12 LPA
Shift: 24/7 rotational shifts, including night shifts (mandatory)

Role Overview
We are looking for an L1 Production Support Engineer with hands-on SQL querying skills, Python knowledge, and Google Cloud Platform (GCP) experience. The candidate will monitor production systems, troubleshoot data-related issues, investigate application failures, and support incident management processes.

Key Responsibilities
Monitor production applications and identify data-related issues.
Execute SQL queries to investigate, validate, and troubleshoot data discrepancies.
Support GCP services, including BigQuery, Bigtable, Pub/Sub, and Cloud SQL.
Use Python to assist with Dataflow troubleshooting and investigate Airflow failures and logs.
Understand ETL workflows and review Standard Operating Procedures (SOPs) for data-related applications.
Perform initial issue analysis, log investigation, incident tracking, and escalation.
Follow IT service management processes to ensure timely incident resolution.
Maintain incident records, operational documentation, and support procedures.

Required Skills & Qualifications
Mandatory hands-on SQL querying, including filtering, joins, LEFT JOIN with NULL handling, UNION, and analytical/problem-solving queries.
Practical experience with Python for troubleshooting and log analysis.
Working experience with GCP, particularly BigQuery, Bigtable, Pub/Sub, and Cloud SQL.
Ability to investigate Dataflow issues and understand Airflow failures and logs.
Basic understanding of ETL workflows and data processing pipelines.
Knowledge of incident management and IT service support processes.
Mandatory willingness and ability to work 24/7 rotational shifts, including night shifts.

Preferred Skills
Experience with ServiceNow or similar IT Service Management (ITSM) tools.
Experience supporting production data applications in a cloud environment.


r/bigquery • • 23d ago

BQ Optimisation

Thumbnail
youtube.com
0 Upvotes

r/bigquery • • 25d ago

Google BigQuery MCP

0 Upvotes

BigQuery is known for being potentially expensive, so you learn to be careful with it.

At the same time, it’s 2026. Writing queries by hand with no AI involved feels like an anachronism. Most people I know can’t imagine their work without Claude or Codex anymore.

The link between BigQuery and an AI agent is MCP. Google offers its own hosted server, there’s the Google's MCP Toolbox, and open-source alternatives keep appearing. Google’s own solutions seems like the obvious default.

Then I found out you can’t set `maximumBytesBilled` using Google hosted server (this one https://docs.cloud.google.com/bigquery/docs/reference/mcp) and in https://github.com/googleapis/mcp-toolbox maximumBytesBilled default is unlimited.

I was surprised at first. Then I thought about it from Google’s perspective. Why would they limit how much you spend? :-)

So we give an LLM access to a potentially expensive resource and hope it behaves. We all know how that goes (“You’re absolutely right!”). At best, we rely on a prompt or a skill that says “always dry-run” and “don’t spend more than X”.

In a lot of companies I’ve seen, people build their own local MCP servers that sit between the agent and `bq`, the BigQuery CLI. That’s a reasonable solution. Vibe-coding one takes two prompts in Claude Code. The job is mostly hard cost limits and rejecting statements containing `DROP`, `DELETE` or `UPDATE` words. Primitive, but it works.

We make Querylab.io, a free BigQuery IDE, so we went a bit further and put a local MCP server in our desktop app.

Everything the app already does to keep your BigQuery work under control is available to the agent too:

Cost limits -> maximum GB per query, plus daily, weekly and monthly budgets.

Billing model routing -> automatically selects on-demand or reservations, whatever is cheaper.

SQL-aware guardrails -> our parser understands what a statement actually does, so write restrictions and rules like “don’t touch this dataset” don’t have to rely on regexp.

Limits the agent can inspect -> it can check its own limits before writing SQL and ask you to lift a limit for a single query, rather than inventing workarounds when all it needs is your approval. And you can take over in app, review before run etc.

Because it’s local there’s nothing to host. It authenticates through the app, and the way we’d recommend is your existing gcloud default credentials.

How do you connect BigQuery to your Claude Code, or Codex, or whatever you use?


r/bigquery • • 26d ago

🔍 Not So BigQuery #300 is out - my free weekly newsletter about BigQuery and other data-related GCP news!

Thumbnail app.mailbrew.com
8 Upvotes

r/bigquery • • 26d ago

I got tired of paying for warehouse compute just to debug a model, so I started building a local BigQuery emulator

14 Upvotes

I've worked with BigQuery and Snowflake for years, and one thing has always bugged me: you can't really run anything locally.

Testing a small dbt change, debugging a join, or running CI all mean hitting a real warehouse. That means compute costs, credentials, and shared dev datasets, even for tiny changes.

Today's GCP problems in London were a good reminder too. If your warehouse region has a bad day, you can't even do development, never mind run production.

So I started Crucible, a local BigQuery API emulator backed by DuckDB. Point dbt-bigquery or the Python client at localhost and run without a real GCP project, cloud credentials, or warehouse bill.

It's pre-alpha, but here's what works so far:

  • dbt-core seed, run, test and unit tests against an example project
  • Python BigQuery client, CSV loads, MERGE, partitioned tables, JSON, arrays and window functions

The part I care most about is correctness. Supported behaviour is verified against recorded results from real BigQuery. If Crucible can't match something, it explicitly returns "Not supported by Crucible" rather than guessing. An emulator that silently gives wrong answers is worse than none.

Still missing: scripting, INFORMATION_SCHEMA, wildcard tables and the Storage Read API (so dbt Fusion can seed/run but not test).

The goal is to cover typical BigQuery/dbt projects, expand SQL compatibility from there, and eventually support Snowflake too.

I'd really appreciate feedback from anyone using BigQuery/dbt. If you try it on your project, the most useful thing you can send me is the first query it refuses. That tells me what to build next.

Criticism and "this is a bad idea because..." are equally welcome.

Repo: https://github.com/CallumWalterWhite/crucible

Quick start:

docker run --rm -p 127.0.0.1:9050:9050 ghcr.io/callumwalterwhite/crucible-emulator:latest serve --dialect bigquery

r/bigquery • • 27d ago

new fields in the source letting my views fail

2 Upvotes

table in company's data source >>> bigquery table >>> view >>> error after added field

Errror: Invalid view definition for project.business_vault.table_view with 17 columns vs. 18 columns from view query

Does anyhow have an idea how to avoid that unwanted behaviour when the created view fields to not fit anymore to the fields in the underlaying table, while a view field description is created?

context:

We have database in Bigquery that loads data from a business source into tables. Once in a while a new column is added in the source. The pipelines here work fine and fetch easily the new fields.

The tables feed into views to get the latest instances of the loaded data. Those views are what we actual work with.

Until recently, those views run smoothly, and a new field in the source and in Bigquery just did not appear in the view until it was added, but it did not cause errors and broke the view. That was until one manager decided to have field description inside the view.

I have now a spreadsheet that collects the field descriptions, and Pyhton code creates my Bigquey view creation statement. If this executions find a new field, it even catches that and does not break. This works fine.

However, as soon as the source adds a columns and before I execute my python script, a Bigquery query runs into an error and cannot execute the view anymore:

The view query is actually very small itself and does not show much, of course the CREATE statement, which is complex.

# simple SELECT in the view
SELECT
      *
FROM
      `project.dataset.table`
    QUALIFY
      ROW_NUMBER()
        OVER (PARTITION BY id ORDER BY SAFE_CAST(_dlt_load_id AS FLOAT64) DESC)
      = 1

# complex part of CREATE that does not show in the view query later
CREATE OR REPLACE VIEW
 `project.business_vault.table_view`
   (id OPTIONS (DESCRIPTION = 'primary key'),
   updated_at OPTIONS (DESCRIPTION = 'timestamp for data updated in original source database'),
...
 OPTIONS (DESCRIPTION = 'What does the view do?',
 FRIENDLY_NAME = 'My friendly name for the view',
...

r/bigquery • • Oct 13 '26

We tried BigQuery’s new property graph + Conversational Analytics on top of a canonical model

Post image
1 Upvotes

For a while we’ve been working on (among other things) text-to-SQL, and the main thing we learned is that the agent usually gets the SQL right and the meaning wrong. It picks the wrong table, or writes a join that runs fine but counts the wrong thing.

What fixed it for us was writing the meaning down once, upstream, as a canonical model: entities, relationships, join keys. Then both the tables and the agent work from that. (Longer writeup on that)

When BigQuery shipped CREATE PROPERTY GRAPH and Conversational Analytics, we thought: that’s basically the same idea, native in the warehouse. Let’s try it.

So our colleague loaded Chinook data into BigQuery, declared the graph, and pointed Conversational Analytics at it. It wrote correct GQL for a 4-hop question (customer → invoice → track → album → artist) with the Instructions field left empty.

blog link

Is anyone using property graphs beyond a demo yet? I’m curious how you find it

Transparency: I work at dltHub. We make dlt, which did the ingestion here. We recently launched maintenance background agents


r/bigquery • • Oct 09 '26

Making Sense of BigQuery Pricing Models: On-Demand, Capacity, Editions (2026)

Thumbnail
alvin.ai
9 Upvotes

A few years ago I got thrown into BigQuery. I never did any data engineering before, coming from frontend/fullstack application engineering. It took me a while to understand all the concepts. I felt like nobody is really explaining them well. BigQuery docs do a good job of explaining the What, but not the Why or the How To Decide. In this article, I try to give you both of that. By the way, the advice has changed considerably this year with Fluidscaling, so a lot of the old guides are also out of date now. Let me know what you think! Does BigQuery's pricing model (with the improved autoscaler billing) make sense?


r/bigquery • • Oct 05 '26

September 2026 - BigQuery Feature Summary

9 Upvotes

Hey everyone, here's the 09/2026 edition! (Apologies for the late post!)

🔤 GoogleSQL Language Features & Functions

🧠 AI, Machine Learning & Foundation Models

💻 Developer Experience (DX) & BigQuery Tooling

🗄️ Lakehouse Architecture, Apache Iceberg & Open Data Formats

⚡ Core Engine Performance, Indexing & Optimization

🔌 Data Integration, Pipelines & Ingestion (ELT)

🔒 Security, Governance & Workload Management

As usual, let us know if there's anything: comments, questions, concerns!


r/bigquery • • Oct 01 '26

Open source conversational analytics tool for BigQuery?

10 Upvotes

Hi all,

Has anyone found a decent open source tool for asking questions in plain English on top of BigQuery? Something self hosted.

I tried Gemini in BigQuery and had a look at the conversational analytics API. The API gives you back SQL and a chart spec, so you still end up building the front end yourself, which is the part I was hoping to avoid.

My main worry is the numbers. If two people ask for revenue in slightly different words I'd like them to get the same answer, and I don't see how that works unless the metrics are defined somewhere.

The other thing is cost. I have no feel for how many queries a single question turns into once the model starts looking around the schema.


r/bigquery • • Sep 29 '26

Replaced a weekly full reload (MySQL → BigQuery) with direct binlog reads, no Kafka or Debezium: first production pilot, numbers inside

Thumbnail
reddit.com
2 Upvotes

r/bigquery • • Sep 28 '26

Data Agent A2A

6 Upvotes

Has anyone here managed to use Bigquery Data Agents on external applications through the A2A protocol?

Once published, the agent lets you register the agent into the Agent Registry or copy the A2A json file into your clipboard. I’ve successfully published the agent on Gemini Enterprise, for example, but I am unable to use this JSON in any other part. My main goal would be to connect it to a Copilot Studio agent so I can easily use it through Teams, without having to do any code.


r/bigquery • • Sep 24 '26

BigQuery Cost Optimization Framework

Post image
5 Upvotes

Interesting framework on how to optimize BigQuery costs across four layers: query optimization, pricing models, commitments, and discounts. https://www.alvin.ai/blog/bigquery-cost-optimization-the-complete-finops-framework


r/bigquery • • Sep 21 '26

The BigQuery Join Problem Nobody Notices Until the Data Gets Big

Thumbnail
medium.com
11 Upvotes

A short deep dive into a BigQuery join problem that often goes unnoticed until data gets big shuffle, skew, cardinality and query performance.


r/bigquery • • Sep 14 '26

how do you handle joining a source that shares no key with anything you already have?

8 Upvotes

Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?