# BigQuery

The target-bigquery loader loads extracted data into a BigQuery data warehouse.

To learn more about target-bigquery, refer to the repository at https://github.com/adswerve/target-bigquery.

# Getting Started

# Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project
  3. Add an extractor to pull data from a source

Then, follow the steps in the "Activate the Google BigQuery API" section of the repository's README.

# Installation and configuration

# Using the Command Line Interface

  1. Add the target-bigquery loader to your project using meltano add:

    meltano add loader target-bigquery
    
  2. Configure the settings below using meltano config.

# Using Meltano UI

  1. Start Meltano UI using meltano ui:

    meltano ui
    
  2. Open the Loaders interface at http://localhost:5000/loaders.

  3. Click the "Add to project" button for "BigQuery".

  4. Configure the settings below in the "Configuration" interface that opens automatically.

# Next steps

Follow the remaining step of the Getting Started guide:

  1. Run a data integration (EL) pipeline

# Settings

target-bigquery requires the configuration of the following settings:

These and other supported settings are documented below. To quickly find the setting you're looking for, use the Table of Contents in the sidebar.

# Minimal configuration

A minimal configuration of target-bigquery in your meltano.yml project file will look like this:






 
 
 
 

plugins:
  loaders:
  - name: target-bigquery
    variant: adswerve
    pip_url: git+https://github.com/adswerve/target-bigquery.git@v0.10.2
    config:
      project_id: my-project-id
      dataset_id: my-dataset-id
      location: EU

# Project ID

BigQuery project

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set project_id <id>

export TARGET_BIGQUERY_PROJECT_ID=<id>

# Dataset ID

BigQuery dataset

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set dataset_id <id>

export TARGET_BIGQUERY_DATASET_ID=<id>

# Location

Dataset location

See https://cloud.google.com/bigquery/docs/locations.

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set location EU

export TARGET_BIGQUERY_LOCATION=EU

# Credentials Path

  • Name: credentials_path
  • Environment variable: TARGET_BIGQUERY_CREDENTIALS_PATH, alias: GOOGLE_APPLICATION_CREDENTIALS
  • Default: $MELTANO_PROJECT_ROOT/client_secrets.json

Fully qualified path to client_secrets.json for your service account.

See the "Activate the Google BigQuery API" section of the repository's README and https://cloud.google.com/docs/authentication/production.

By default, this file is expected to be at the root of your project directory.

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set credentials_path /home/user/Downloads/client_secrets.json

export TARGET_BIGQUERY_CREDENTIALS_PATH=/home/user/Downloads/client_secrets.json

# Validate Records

Validate records

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set validate_records true

export TARGET_BIGQUERY_VALIDATE_RECORDS=true

# Add Metadata Columns

  • Name: add_metadata_columns
  • Environment variable: TARGET_BIGQUERY_ADD_METADATA_COLUMNS
  • Default: false

Add _time_extracted and _time_loaded metadata columns

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set add_metadata_columns true

export TARGET_BIGQUERY_ADD_METADATA_COLUMNS=true

# Replication Method

  • Name: replication_method
  • Environment variable: TARGET_BIGQUERY_REPLICATION_METHOD
  • Options: append, truncate
  • Default: append

Replication method, append or truncate

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set replication_method truncate

export TARGET_BIGQUERY_REPLICATION_METHOD=truncate

# Table Prefix

Add prefix to table name

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set table_prefix <prefix>

export TARGET_BIGQUERY_TABLE_PREFIX=<prefix>

# Table Suffix

Add suffix to table name

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set table_suffix <suffix>

export TARGET_BIGQUERY_TABLE_SUFFIX=<suffix>

# Max Cache

Maximum cache size in MB

# How to use

Manage this setting using Meltano UI, meltano config, or an environment variable:

meltano config target-bigquery set max_cache 100

export TARGET_BIGQUERY_MAX_CACHE=100