# Extractors & Loaders

# Extractors

Meltano Extractors extract data in bulk and are commonly prefixed with tap. Meltano currently supports:

• tap-carbon-intensity
• tap-csv
• tap-fastly
• tap-gitlab
• tap-google-analytics
• tap-marketo
• tap-mongodb
• tap-salesforce
• tap-stripe
• tap-zendesk
• tap-zuora

# Carbon Intensity

tap-carbon-intensity pulls data from the Official Carbon Emissions Intensity API for Great Britain, which was developed by the National Grid. For more information, check out http://carbonintensity.org.uk/.

# Info

• Data Source: Carbon Intensity API
• Repository: https://gitlab.com/meltano/tap-carbon-intensity

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-carbon-intensity

If you are successful, you should see Added and installed extractors 'tap-carbon-intensity' in your terminal.

# CSV

tap-csv is a CSV reader that is optimized for tasks where the file structure is highly predictable.

# Info

• Data Source: Traditionally-delimited CSV files (commas separated columns, newlines indicate new rows, double quoted values) as defined by the defaults of the python csv library.
• Repository: https://gitlab.com/meltano/tap-csv

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-csv

If you are successful, you should see Added and installed extractors 'tap-csv' in your terminal.

# Configuration

.env

export TAP_CSV_FILES_DEFINITION="files_def.json"

Where files_def.json is a json file with all the CSV files to be loaded:

files_def.json

[
  { "entity" : "leads",
    "file" : "/path/to/leads.csv",
    "keys" : ["Id"]
  },
  { "entity" : "opportunities",
    "file" : "/path/to/opportunities.csv",
    "keys" : ["Id"]
  }
]

Description of available options:

• entity: The entity name to be passed to singer (i.e. the table name).
• file: Local path to the file to be ingested. Note that this may be a directory, in which case all files in that directory and any of its subdirectories will be recursively processed.
• keys: The names of the columns that constitute the unique keys for that entity.

# Fastly

tap-fastly pulls raw data from Fastly and produces JSON-formatted data per the Singer spec.

# Info

• Data Source: Fastly
• Repository: https://gitlab.com/meltano/tap-fastly

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-fastly

If you are successful, you should see Added and installed extractors 'tap-fastly' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export FASTLY_API_TOKEN="yourFastlyApiToken"
# The date uses ISO-8601 and supports time if desired
export FASTLY_START_DATE="YYYY-MM-DD"

# GitLab

tap-gitlab pulls raw data from GitLab's REST API and extracts the following resources from GitLab:

• Branches
• Commits
• Issues
• Projects
• Project milestones
• Project Merge Requests
• Users
• Groups
• Group Milestones
• Group and Project members
• Tags
• Releases
• Group Labels
• Project Labels
• Epics (only available for Gitlab Ultimate and Gitlab.com Gold accounts)
• Epic Issues (only available for Gitlab Ultimate and Gitlab.com Gold accounts)

It incrementally pulls data based on the input state and then outputs the schema for each resource and the extracted data. For more information you can check the documentation for tap-gitlab.

# Info

• Data Source: GitLab's REST API
• Repository: https://gitlab.com/meltano/tap-gitlab

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-gitlab

If you are successful, you should see Added and installed extractors 'tap-gitlab' in your terminal.

3. Get your GitLab access token
   • Login to your GitLab account
   • Navigate to your profile page
   • Create an access token

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export GITLAB_API_TOKEN="access token from step 3"
export GITLAB_API_GROUPS="myorg mygroup"
export GITLAB_API_PROJECTS="myorg/repo-a myorg-repo-b"
# The date uses ISO-8601 and supports time if desired
export GITLAB_API_START_DATE="YYYY-MM-DD"
export GITLAB_API_ULTIMATE_LICENSE="true"

If ultimate_license is true (defaults to false), then the GitLab account used has access to the Gitlab Ultimate or Gitlab.com Gold features. It will enable fetching Epics, Epic Issues and other entities available for Gitlab Ultimate and Gitlab.com Gold accounts.

# Google Analytics

tap-google-analytics pulls raw data from the Google Analytics Reporting APIv4.

# Info

• Data Source: Google Analytics Reporting APIv4
• Repository: https://gitlab.com/meltano/tap-google-analytics

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-google-analytics

If you are successful, you should see Added and installed extractors 'tap-google-analytics' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

Required:

export GOOGLE_ANALYTICS_API_CLIENT_SECRETS="client_secrets.json"
export GOOGLE_ANALYTICS_API_VIEW_ID="YOUR VIEW ID"
export GOOGLE_ANALYTICS_API_START_DATE="2019-02-01T00:00:00Z"

Optional:

export GOOGLE_ANALYTICS_API_REPORTS="cli_reports.json"
export GOOGLE_ANALYTICS_API_END_DATE="2019-06-01T00:00:00Z"

Check the README for details.

# Marketo

tap-marketo pulls raw data from Marketo's REST API and extracts activity types, activites, and leads from Marketo.

# Info

• Data Source: Marketo's REST API
• Repository: https://gitlab.com/meltano/tap-marketo

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-marketo

If you are successful, you should see Added and installed extractors 'tap-marketo' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export MARKETO_CLIENT_ID="yourClientId"
export MARKETO_CLIENT_SECRET="yourClientSecret"
export MARKETO_ENDPOINT="yourEndpointUrl"
export MARKETO_IDENTITY="yourIdentity"
export MARKETO_START_TIME="yourStartTime"

# MongoDB

tap-mongodb pulls raw data from a MongoDB source.

# Info

• Data Source: MongoDB source
• Repository: https://github.com/singer-io/tap-mongodb

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-mongodb

If you are successful, you should see Added and installed extractors 'tap-mongodb' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

# MongoDB databse host URI
export MONGODB_HOST=""
# MongoDB database port
export MONGODB_PORT=""
# MongoDB database username
export MONGODB_USERNAME=""
# MongoDB database password
export MONGODB_PASSWORD=""
# MongoDB database name
export MONGODB_DBNAME=""

# Salesforce

tap-salesforce is an extractor that pulls data from a Salesforce database and produced JSON-formatted data following the Singer spec.

# Info

• Data Source: Salesforce
• Repository: https://gitlab.com/meltano/tap-salesforce

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-salesforce

If you are successful, you should see Added and installed extractors 'tap-salesforce' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export SFDC_CLIENT_ID="yourSalesforceClientId"
export SFDC_PASSWORD="yourSalesforcePassword"
export SFDC_SECURITY_TOKEN="yourSalesforceSecurityToken"
export SFDC_START_DATE="yourSalesforceStartDate"
export SFDC_USERNAME="yourSalesforceUsername"

# Stripe

tap-stripe is an extractor that pulls data from Stripe's API and produces JSON-formatted data following the Singer spec.

# Info

• Data Source: Stripe's API
• Repository: https://github.com/meltano/tap-stripe

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-stripe

If you are successful, you should see Added and installed extractors 'tap-stripe' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export STRIPE_API_KEY="yourStripeApiKey"
# The date uses ISO-8601 and supports time if desired
export STRIPE_START_DATE="YYYY-MM-DD"

# Zendesk

tap-zendesk is an extractor that pulls data from a Zendesk REST API and produces JSON-formatted data following the Singer spec.

# Info

• Data Source: Zendesk REST API
• Repository: https://github.com/meltano/tap-zendesk

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-zendesk

If you are successful, you should see Added and installed extractors 'tap-zendesk' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export ZENDESK_EMAIL="yourZendeskEmail"
export ZENDESK_API_TOKEN="yourZendeskApiToken"
export ZENDESK_SUBDOMAIN="yourZendeskSubdomain"
# The date uses ISO-8601 and supports time if desired
export ZENDESK_START_DATE="yourZendeskStartDate"

# Zuora

tap-zuora is an extractor that pulls data from a Zuora REST API and produces JSON-formatted data following the Singer spec.

# Info

• Data Source: Zuora REST API
• Repository: https://github.com/singer-io/tap-zuora

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add extractor tap-zuora

If you are successful, you should see Added and installed extractors 'tap-zuora' in your terminal.

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export ZUORA_USERNAME=""
export ZUORA_PASSWORD=""
export ZUORA_API_TOKEN=""   # preferred to ZUORA_PASSWORD
export ZUORA_API_TYPE=""    # specifically 'REST' or 'AQuA'
export ZUORA_PARTNER_ID=""  # optional, only for the 'AQuA` API type
export ZUORA_START_DATE=""
export ZUORA_SANDBOX=""     # specifically 'true' or 'false'

# Loaders

Meltano Loaders load data in bulk and are commonly prefixed with target. Meltano currently supports:

• target-csv
• target-postgres
• target-snowflake
• target-sqlite

# CSV

target-csv is a loader that works with other extractors in order to move data into CSV-formatted files.

# Info

• Data Warehouse: CSV Files
• Repository: https://gitlab.com/meltano/target-csv

# Install

1. Navigate to your Meltano project in the terminal
2. Run the following command:

meltano add loader target-csv

If you are successful, you should see Added and installed loaders 'target-csv' in your terminal.

# Configuration

If you want to customize your delimited or quote character, open meltano.yml for your desired project and update the configuration there.

  - config:
      delimiter": "\t"
      quotechar": ''''
    name: target-csv
    pip_url: git+https://gitlab.com/meltano/target-csv.git

# Snowflake

target-snowflake is a loader that works with other extractors in order to move data into a Snowflake database.

# Info

• Data Warehouse: Snowflake
• Repository: https://gitlab.com/meltano/target-snowflake

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export SF_ACCOUNT=""
export SF_USER=""
export SF_PASSWORD=""
export SF_ROLE=""       # in UPPERCASE
export SF_DATABASE=""   # in UPPERCASE
export SF_WAREHOUSE=""  # in UPPERCASE

# Postgres

target-postgres is a loader that works with other extractors in order to move data into a Postgres database.

# Info

• Data Warehouse: Postgres
• Repository: https://github.com/meltano/target-postgres

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export PG_ADDRESS=""
export PG_USERNAME=""
export PG_PORT=""
export PG_PASSWORD=""
export PG_DATABASE=""
export PG_SCHEMA=""

# Sqlite

target-sqlite is a loader that works with other extractors in order to move data into a SQLite database.

# Info

• Data Warehouse: SQLite
• Repository: https://gitlab.com/meltano/target-sqlite

# Configuration

1. Open your project's .env file in a text editor
2. Add the following variables to your file:

export SQLITE_DATABASE=""